2024-08-02T23:28:35.852 INFO:root:teuthology version: 0.0.1.dev379+g53ce146 2024-08-02T23:28:35.868 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-08-02T23:28:35.950 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2024-08-02_15:39:04-crimson-rados-squid-release-distro-default-smithi/7833276 branch: squid-release description: crimson-rados/basic/{clusters/fixed-2 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/seastore tasks/rados_api_tests} email: yweinste@redhat.com first_in_suite: false job_id: '7833276' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi name: yuriw-2024-08-02_15:39:04-crimson-rados-squid-release-distro-default-smithi no_nested_subset: false os_type: centos os_version: 9.stream overrides: admin_socket: branch: squid-release ceph: conf: client: debug ms: 1 global: enable experimental unrecoverable data corrupting features: crimson ms client mode: crc ms cluster mode: crc ms mon client mode: crc ms mon cluster mode: crc ms mon service mode: crc ms service mode: crc mgr: debug mgr: 20 debug ms: 1 mgr stats period: 30 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon warn on pool no app: false osd pool default crimson: true osd: crimson alien thread cpu cores: 6-7 crimson osd obc lru size: 10 debug ms: 1 debug osd: 20 osd blocked scrub grace period: 3600 osd class default list: '*' osd class load list: '*' osd objectstore: seastore osd shutdown pgref assert: true osd.0: crimson seastar cpu cores: 0-2 osd.1: crimson seastar cpu cores: 3-5 osd.2: crimson seastar cpu cores: 0-2 osd.3: crimson seastar cpu cores: 3-5 flavor: crimson fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(POOL_APP_NOT_ENABLED\) - reached quota - but it is still running - overall HEALTH_ - \(POOL_FULL\) - \(SMALLER_PGP_NUM\) - \(CACHE_POOL_NO_HIT_SET\) - \(CACHE_POOL_NEAR_FULL\) - \(POOL_APP_NOT_ENABLED\) - \(PG_AVAILABILITY\) - \(PG_DEGRADED\) sha1: 26c3fb8e197dcf7a49a54d1f4c8a7362ee35a8ea ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: crimson sha1: 26c3fb8e197dcf7a49a54d1f4c8a7362ee35a8ea selinux: allowlist: - scontext=system_u:system_r:getty_t:s0 workunit: branch: squid-release env: CRIMSON_COMPAT: '1' sha1: 26c3fb8e197dcf7a49a54d1f4c8a7362ee35a8ea owner: scheduled_yuriw@teuthology priority: 99 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - osd.0 - osd.1 - client.0 - node-exporter.a - - mgr.x - osd.2 - osd.3 - client.1 - prometheus.a - node-exporter.b seed: 5670 sha1: 26c3fb8e197dcf7a49a54d1f4c8a7362ee35a8ea sleep_before_teardown: 0 subset: 111/120000 suite: crimson-rados suite_branch: squid-release suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_26c3fb8e197dcf7a49a54d1f4c8a7362ee35a8ea/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 26c3fb8e197dcf7a49a54d1f4c8a7362ee35a8ea targets: smithi073.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIKvQZFa6VwuEe1OB7leedPBiptAZaVBbyHRtF8y8R1eMRCe1ZAwnuytmeA9MKuyXX5kWXmo/XdEtGVc4T1M8aI= smithi131.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPRu4Go9/I1OWxz4uAF5w3bG6Emuz4/nUMZtNrjl1gfafid+PoEwVI1qZgMZzgaBeDrJp9+b1Cc0cLrqNEDC5q8= tasks: - install: null - ceph: conf: mon: mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 paxos service trim min: 10 osd: debug monc: 20 flavor: crimson - workunit: clients: client.0: - rados/test.sh --crimson - rados/test_pool_quota.sh teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 53ce1462e129f6eb4071986336534c740fdebd31 timestamp: 2024-08-02_15:39:04 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.4074525 2024-08-02T23:28:35.950 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_26c3fb8e197dcf7a49a54d1f4c8a7362ee35a8ea/qa; will attempt to use it 2024-08-02T23:28:35.951 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_26c3fb8e197dcf7a49a54d1f4c8a7362ee35a8ea/qa/tasks 2024-08-02T23:28:35.952 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-08-02T23:28:35.953 INFO:teuthology.task.internal:Checking packages... 2024-08-02T23:28:35.971 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'crimson' and ceph hash '26c3fb8e197dcf7a49a54d1f4c8a7362ee35a8ea' 2024-08-02T23:28:35.972 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-08-02T23:28:35.972 INFO:teuthology.packaging:ref: None 2024-08-02T23:28:35.972 INFO:teuthology.packaging:tag: None 2024-08-02T23:28:35.972 INFO:teuthology.packaging:branch: squid-release 2024-08-02T23:28:35.972 INFO:teuthology.packaging:sha1: 26c3fb8e197dcf7a49a54d1f4c8a7362ee35a8ea 2024-08-02T23:28:35.972 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&ref=squid-release 2024-08-02T23:28:36.113 INFO:teuthology.task.internal:Found packages for ceph version 19.1.0-1260.g26c3fb8e 2024-08-02T23:28:36.115 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-08-02T23:28:36.122 INFO:teuthology.task.internal:no buildpackages task found 2024-08-02T23:28:36.122 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-08-02T23:28:36.170 INFO:teuthology.task.internal:Saving configuration 2024-08-02T23:28:36.182 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-08-02T23:28:36.188 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-08-02T23:28:36.213 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi073.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-08-02_15:39:04-crimson-rados-squid-release-distro-default-smithi/7833276', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-08-02 23:23:25.677884', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIKvQZFa6VwuEe1OB7leedPBiptAZaVBbyHRtF8y8R1eMRCe1ZAwnuytmeA9MKuyXX5kWXmo/XdEtGVc4T1M8aI='} 2024-08-02T23:28:36.236 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi131.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-08-02_15:39:04-crimson-rados-squid-release-distro-default-smithi/7833276', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-08-02 23:23:25.679063', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPRu4Go9/I1OWxz4uAF5w3bG6Emuz4/nUMZtNrjl1gfafid+PoEwVI1qZgMZzgaBeDrJp9+b1Cc0cLrqNEDC5q8='} 2024-08-02T23:28:36.236 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-08-02T23:28:36.243 INFO:teuthology.task.internal:roles: ubuntu@smithi073.front.sepia.ceph.com - ['mon.a', 'osd.0', 'osd.1', 'client.0', 'node-exporter.a'] 2024-08-02T23:28:36.243 INFO:teuthology.task.internal:roles: ubuntu@smithi131.front.sepia.ceph.com - ['mgr.x', 'osd.2', 'osd.3', 'client.1', 'prometheus.a', 'node-exporter.b'] 2024-08-02T23:28:36.243 INFO:teuthology.run_tasks:Running task console_log... 2024-08-02T23:28:36.299 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fee6b56beb0>, signals=[15]) 2024-08-02T23:28:36.299 INFO:teuthology.run_tasks:Running task internal.connect... 2024-08-02T23:28:36.305 INFO:teuthology.task.internal:Opening connections... 2024-08-02T23:28:36.305 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi073.front.sepia.ceph.com 2024-08-02T23:28:36.307 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi073.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-02T23:28:36.380 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi131.front.sepia.ceph.com 2024-08-02T23:28:36.381 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi131.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-02T23:28:36.452 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-08-02T23:28:36.459 DEBUG:teuthology.orchestra.run.smithi073:> uname -m 2024-08-02T23:28:36.477 INFO:teuthology.orchestra.run.smithi073.stdout:x86_64 2024-08-02T23:28:36.477 DEBUG:teuthology.orchestra.run.smithi073:> cat /etc/os-release 2024-08-02T23:28:36.534 INFO:teuthology.orchestra.run.smithi073.stdout:NAME="CentOS Stream" 2024-08-02T23:28:36.534 INFO:teuthology.orchestra.run.smithi073.stdout:VERSION="9" 2024-08-02T23:28:36.534 INFO:teuthology.orchestra.run.smithi073.stdout:ID="centos" 2024-08-02T23:28:36.535 INFO:teuthology.orchestra.run.smithi073.stdout:ID_LIKE="rhel fedora" 2024-08-02T23:28:36.535 INFO:teuthology.orchestra.run.smithi073.stdout:VERSION_ID="9" 2024-08-02T23:28:36.535 INFO:teuthology.orchestra.run.smithi073.stdout:PLATFORM_ID="platform:el9" 2024-08-02T23:28:36.535 INFO:teuthology.orchestra.run.smithi073.stdout:PRETTY_NAME="CentOS Stream 9" 2024-08-02T23:28:36.535 INFO:teuthology.orchestra.run.smithi073.stdout:ANSI_COLOR="0;31" 2024-08-02T23:28:36.535 INFO:teuthology.orchestra.run.smithi073.stdout:LOGO="fedora-logo-icon" 2024-08-02T23:28:36.535 INFO:teuthology.orchestra.run.smithi073.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-08-02T23:28:36.535 INFO:teuthology.orchestra.run.smithi073.stdout:HOME_URL="https://centos.org/" 2024-08-02T23:28:36.535 INFO:teuthology.orchestra.run.smithi073.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-08-02T23:28:36.535 INFO:teuthology.orchestra.run.smithi073.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-08-02T23:28:36.535 INFO:teuthology.orchestra.run.smithi073.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-08-02T23:28:36.536 INFO:teuthology.lock.ops:Updating smithi073.front.sepia.ceph.com on lock server 2024-08-02T23:28:36.553 DEBUG:teuthology.orchestra.run.smithi131:> uname -m 2024-08-02T23:28:36.571 INFO:teuthology.orchestra.run.smithi131.stdout:x86_64 2024-08-02T23:28:36.571 DEBUG:teuthology.orchestra.run.smithi131:> cat /etc/os-release 2024-08-02T23:28:36.628 INFO:teuthology.orchestra.run.smithi131.stdout:NAME="CentOS Stream" 2024-08-02T23:28:36.628 INFO:teuthology.orchestra.run.smithi131.stdout:VERSION="9" 2024-08-02T23:28:36.628 INFO:teuthology.orchestra.run.smithi131.stdout:ID="centos" 2024-08-02T23:28:36.628 INFO:teuthology.orchestra.run.smithi131.stdout:ID_LIKE="rhel fedora" 2024-08-02T23:28:36.628 INFO:teuthology.orchestra.run.smithi131.stdout:VERSION_ID="9" 2024-08-02T23:28:36.628 INFO:teuthology.orchestra.run.smithi131.stdout:PLATFORM_ID="platform:el9" 2024-08-02T23:28:36.628 INFO:teuthology.orchestra.run.smithi131.stdout:PRETTY_NAME="CentOS Stream 9" 2024-08-02T23:28:36.628 INFO:teuthology.orchestra.run.smithi131.stdout:ANSI_COLOR="0;31" 2024-08-02T23:28:36.628 INFO:teuthology.orchestra.run.smithi131.stdout:LOGO="fedora-logo-icon" 2024-08-02T23:28:36.628 INFO:teuthology.orchestra.run.smithi131.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-08-02T23:28:36.629 INFO:teuthology.orchestra.run.smithi131.stdout:HOME_URL="https://centos.org/" 2024-08-02T23:28:36.629 INFO:teuthology.orchestra.run.smithi131.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-08-02T23:28:36.629 INFO:teuthology.orchestra.run.smithi131.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-08-02T23:28:36.629 INFO:teuthology.orchestra.run.smithi131.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-08-02T23:28:36.629 INFO:teuthology.lock.ops:Updating smithi131.front.sepia.ceph.com on lock server 2024-08-02T23:28:36.645 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-08-02T23:28:36.653 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-08-02T23:28:36.660 INFO:teuthology.task.internal:Checking for old test directory... 2024-08-02T23:28:36.660 DEBUG:teuthology.orchestra.run.smithi073:> test '!' -e /home/ubuntu/cephtest 2024-08-02T23:28:36.662 DEBUG:teuthology.orchestra.run.smithi131:> test '!' -e /home/ubuntu/cephtest 2024-08-02T23:28:36.684 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-08-02T23:28:36.690 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-08-02T23:28:36.690 DEBUG:teuthology.orchestra.run.smithi073:> test -z $(ls -A /var/lib/ceph) 2024-08-02T23:28:36.722 DEBUG:teuthology.orchestra.run.smithi131:> test -z $(ls -A /var/lib/ceph) 2024-08-02T23:28:37.003 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-08-02T23:28:37.057 INFO:teuthology.run_tasks:Running task kernel... 2024-08-02T23:28:37.071 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-08-02T23:28:37.071 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mgr.x': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-08-02T23:28:37.071 DEBUG:teuthology.orchestra.run.smithi073:> test -f /run/.containerenv -o -f /.dockerenv 2024-08-02T23:28:37.072 DEBUG:teuthology.orchestra.run.smithi131:> test -f /run/.containerenv -o -f /.dockerenv 2024-08-02T23:28:37.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-02T23:28:37.088 DEBUG:teuthology.orchestra.run.smithi073:> uname -r 2024-08-02T23:28:37.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-02T23:28:37.089 DEBUG:teuthology.orchestra.run.smithi131:> uname -r 2024-08-02T23:28:37.145 INFO:teuthology.orchestra.run.smithi073.stdout:5.14.0-437.el9.x86_64 2024-08-02T23:28:37.145 INFO:teuthology.task.kernel:Running kernel on smithi073: 5.14.0-437.el9.x86_64 2024-08-02T23:28:37.146 DEBUG:teuthology.orchestra.run.smithi073:> sudo yum install -y kernel 2024-08-02T23:28:37.147 INFO:teuthology.orchestra.run.smithi131.stdout:5.14.0-437.el9.x86_64 2024-08-02T23:28:37.147 INFO:teuthology.task.kernel:Running kernel on smithi131: 5.14.0-437.el9.x86_64 2024-08-02T23:28:37.147 DEBUG:teuthology.orchestra.run.smithi131:> sudo yum install -y kernel 2024-08-02T23:28:39.613 INFO:teuthology.orchestra.run.smithi073.stdout:CentOS Stream 9 - BaseOS 10 MB/s | 8.2 MB 00:00 2024-08-02T23:28:40.960 INFO:teuthology.orchestra.run.smithi131.stdout:CentOS Stream 9 - BaseOS 3.6 MB/s | 8.2 MB 00:02 2024-08-02T23:28:41.714 INFO:teuthology.orchestra.run.smithi073.stdout:CentOS Stream 9 - AppStream 20 MB/s | 20 MB 00:01 2024-08-02T23:28:45.162 INFO:teuthology.orchestra.run.smithi131.stdout:CentOS Stream 9 - AppStream 6.4 MB/s | 20 MB 00:03 2024-08-02T23:28:47.239 INFO:teuthology.orchestra.run.smithi073.stdout:CentOS Stream 9 - CRB 3.6 MB/s | 6.6 MB 00:01 2024-08-02T23:28:49.590 INFO:teuthology.orchestra.run.smithi073.stdout:CentOS Stream 9 - Extras packages 16 kB/s | 18 kB 00:01 2024-08-02T23:28:50.552 INFO:teuthology.orchestra.run.smithi073.stdout:Extra Packages for Enterprise Linux 27 MB/s | 22 MB 00:00 2024-08-02T23:28:50.979 INFO:teuthology.orchestra.run.smithi131.stdout:CentOS Stream 9 - CRB 3.3 MB/s | 6.6 MB 00:01 2024-08-02T23:28:52.521 INFO:teuthology.orchestra.run.smithi131.stdout:CentOS Stream 9 - Extras packages 87 kB/s | 18 kB 00:00 2024-08-02T23:28:53.525 INFO:teuthology.orchestra.run.smithi131.stdout:Extra Packages for Enterprise Linux 25 MB/s | 22 MB 00:00 2024-08-02T23:28:55.883 INFO:teuthology.orchestra.run.smithi073.stdout:lab-extras 39 kB/s | 1.7 kB 00:00 2024-08-02T23:28:57.831 INFO:teuthology.orchestra.run.smithi073.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-08-02T23:28:57.836 INFO:teuthology.orchestra.run.smithi073.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-08-02T23:28:57.930 INFO:teuthology.orchestra.run.smithi073.stdout:Dependencies resolved. 2024-08-02T23:28:57.937 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2024-08-02T23:28:57.937 INFO:teuthology.orchestra.run.smithi073.stdout: Package Architecture Version Repository Size 2024-08-02T23:28:57.938 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2024-08-02T23:28:57.938 INFO:teuthology.orchestra.run.smithi073.stdout:Installing: 2024-08-02T23:28:57.938 INFO:teuthology.orchestra.run.smithi073.stdout: kernel x86_64 5.14.0-480.el9 baseos 1.5 M 2024-08-02T23:28:57.938 INFO:teuthology.orchestra.run.smithi073.stdout:Installing dependencies: 2024-08-02T23:28:57.938 INFO:teuthology.orchestra.run.smithi073.stdout: kernel-core x86_64 5.14.0-480.el9 baseos 17 M 2024-08-02T23:28:57.939 INFO:teuthology.orchestra.run.smithi073.stdout: kernel-modules x86_64 5.14.0-480.el9 baseos 36 M 2024-08-02T23:28:57.939 INFO:teuthology.orchestra.run.smithi073.stdout: kernel-modules-core x86_64 5.14.0-480.el9 baseos 30 M 2024-08-02T23:28:57.939 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-08-02T23:28:57.939 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction Summary 2024-08-02T23:28:57.939 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2024-08-02T23:28:57.939 INFO:teuthology.orchestra.run.smithi073.stdout:Install 4 Packages 2024-08-02T23:28:57.939 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-08-02T23:28:57.940 INFO:teuthology.orchestra.run.smithi073.stdout:Total download size: 84 M 2024-08-02T23:28:57.940 INFO:teuthology.orchestra.run.smithi073.stdout:Installed size: 126 M 2024-08-02T23:28:57.940 INFO:teuthology.orchestra.run.smithi073.stdout:Downloading Packages: 2024-08-02T23:28:58.492 INFO:teuthology.orchestra.run.smithi073.stdout:(1/4): kernel-5.14.0-480.el9.x86_64.rpm 5.4 MB/s | 1.5 MB 00:00 2024-08-02T23:28:58.826 INFO:teuthology.orchestra.run.smithi073.stdout:(2/4): kernel-core-5.14.0-480.el9.x86_64.rpm 28 MB/s | 17 MB 00:00 2024-08-02T23:28:58.839 INFO:teuthology.orchestra.run.smithi131.stdout:lab-extras 20 kB/s | 1.7 kB 00:00 2024-08-02T23:28:59.310 INFO:teuthology.orchestra.run.smithi073.stdout:(3/4): kernel-modules-5.14.0-480.el9.x86_64.rpm 33 MB/s | 36 MB 00:01 2024-08-02T23:28:59.785 INFO:teuthology.orchestra.run.smithi073.stdout:(4/4): kernel-modules-core-5.14.0-480.el9.x86_6 23 MB/s | 30 MB 00:01 2024-08-02T23:28:59.785 INFO:teuthology.orchestra.run.smithi073.stdout:-------------------------------------------------------------------------------- 2024-08-02T23:28:59.785 INFO:teuthology.orchestra.run.smithi073.stdout:Total 46 MB/s | 84 MB 00:01 2024-08-02T23:29:00.518 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction check 2024-08-02T23:29:00.804 INFO:teuthology.orchestra.run.smithi131.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-08-02T23:29:00.804 INFO:teuthology.orchestra.run.smithi131.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-08-02T23:29:00.848 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction check succeeded. 2024-08-02T23:29:00.849 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction test 2024-08-02T23:29:00.891 INFO:teuthology.orchestra.run.smithi131.stdout:Dependencies resolved. 2024-08-02T23:29:00.898 INFO:teuthology.orchestra.run.smithi131.stdout:================================================================================ 2024-08-02T23:29:00.899 INFO:teuthology.orchestra.run.smithi131.stdout: Package Architecture Version Repository Size 2024-08-02T23:29:00.899 INFO:teuthology.orchestra.run.smithi131.stdout:================================================================================ 2024-08-02T23:29:00.899 INFO:teuthology.orchestra.run.smithi131.stdout:Installing: 2024-08-02T23:29:00.899 INFO:teuthology.orchestra.run.smithi131.stdout: kernel x86_64 5.14.0-480.el9 baseos 1.5 M 2024-08-02T23:29:00.899 INFO:teuthology.orchestra.run.smithi131.stdout:Installing dependencies: 2024-08-02T23:29:00.899 INFO:teuthology.orchestra.run.smithi131.stdout: kernel-core x86_64 5.14.0-480.el9 baseos 17 M 2024-08-02T23:29:00.899 INFO:teuthology.orchestra.run.smithi131.stdout: kernel-modules x86_64 5.14.0-480.el9 baseos 36 M 2024-08-02T23:29:00.899 INFO:teuthology.orchestra.run.smithi131.stdout: kernel-modules-core x86_64 5.14.0-480.el9 baseos 30 M 2024-08-02T23:29:00.899 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-08-02T23:29:00.899 INFO:teuthology.orchestra.run.smithi131.stdout:Transaction Summary 2024-08-02T23:29:00.899 INFO:teuthology.orchestra.run.smithi131.stdout:================================================================================ 2024-08-02T23:29:00.899 INFO:teuthology.orchestra.run.smithi131.stdout:Install 4 Packages 2024-08-02T23:29:00.899 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-08-02T23:29:00.900 INFO:teuthology.orchestra.run.smithi131.stdout:Total download size: 84 M 2024-08-02T23:29:00.900 INFO:teuthology.orchestra.run.smithi131.stdout:Installed size: 126 M 2024-08-02T23:29:00.900 INFO:teuthology.orchestra.run.smithi131.stdout:Downloading Packages: 2024-08-02T23:29:01.832 INFO:teuthology.orchestra.run.smithi131.stdout:(1/4): kernel-5.14.0-480.el9.x86_64.rpm 1.7 MB/s | 1.5 MB 00:00 2024-08-02T23:29:01.842 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction test succeeded. 2024-08-02T23:29:01.842 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction 2024-08-02T23:29:03.549 INFO:teuthology.orchestra.run.smithi131.stdout:(2/4): kernel-modules-core-5.14.0-480.el9.x86_6 17 MB/s | 30 MB 00:01 2024-08-02T23:29:04.283 INFO:teuthology.orchestra.run.smithi073.stdout: Preparing : 1/1 2024-08-02T23:29:04.948 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : kernel-modules-core-5.14.0-480.el9.x86_64 1/4 2024-08-02T23:29:05.043 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : kernel-core-5.14.0-480.el9.x86_64 2/4 2024-08-02T23:29:07.696 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: kernel-core-5.14.0-480.el9.x86_64 2/4 2024-08-02T23:29:07.845 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : kernel-modules-5.14.0-480.el9.x86_64 3/4 2024-08-02T23:29:09.532 INFO:teuthology.orchestra.run.smithi131.stdout:(3/4): kernel-core-5.14.0-480.el9.x86_64.rpm 2.0 MB/s | 17 MB 00:08 2024-08-02T23:29:12.704 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: kernel-modules-5.14.0-480.el9.x86_64 3/4 2024-08-02T23:29:12.805 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : kernel-5.14.0-480.el9.x86_64 4/4 2024-08-02T23:29:15.332 INFO:teuthology.orchestra.run.smithi131.stdout:(4/4): kernel-modules-5.14.0-480.el9.x86_64.rpm 2.5 MB/s | 36 MB 00:14 2024-08-02T23:29:15.332 INFO:teuthology.orchestra.run.smithi131.stdout:-------------------------------------------------------------------------------- 2024-08-02T23:29:15.332 INFO:teuthology.orchestra.run.smithi131.stdout:Total 5.9 MB/s | 84 MB 00:14 2024-08-02T23:29:16.070 INFO:teuthology.orchestra.run.smithi131.stdout:Running transaction check 2024-08-02T23:29:16.428 INFO:teuthology.orchestra.run.smithi131.stdout:Transaction check succeeded. 2024-08-02T23:29:16.428 INFO:teuthology.orchestra.run.smithi131.stdout:Running transaction test 2024-08-02T23:29:17.485 INFO:teuthology.orchestra.run.smithi131.stdout:Transaction test succeeded. 2024-08-02T23:29:17.485 INFO:teuthology.orchestra.run.smithi131.stdout:Running transaction 2024-08-02T23:29:17.624 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: kernel-modules-core-5.14.0-480.el9.x86_64 4/4 2024-08-02T23:29:19.930 INFO:teuthology.orchestra.run.smithi131.stdout: Preparing : 1/1 2024-08-02T23:29:20.666 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : kernel-modules-core-5.14.0-480.el9.x86_64 1/4 2024-08-02T23:29:20.834 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : kernel-core-5.14.0-480.el9.x86_64 2/4 2024-08-02T23:29:23.476 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: kernel-core-5.14.0-480.el9.x86_64 2/4 2024-08-02T23:29:23.653 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : kernel-modules-5.14.0-480.el9.x86_64 3/4 2024-08-02T23:29:28.480 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: kernel-modules-5.14.0-480.el9.x86_64 3/4 2024-08-02T23:29:28.671 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : kernel-5.14.0-480.el9.x86_64 4/4 2024-08-02T23:29:33.518 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: kernel-modules-core-5.14.0-480.el9.x86_64 4/4 2024-08-02T23:29:47.515 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: kernel-core-5.14.0-480.el9.x86_64 4/4 2024-08-02T23:29:47.555 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: kernel-modules-5.14.0-480.el9.x86_64 4/4 2024-08-02T23:29:47.930 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: kernel-5.14.0-480.el9.x86_64 4/4 2024-08-02T23:29:47.930 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : kernel-5.14.0-480.el9.x86_64 1/4 2024-08-02T23:29:47.930 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : kernel-core-5.14.0-480.el9.x86_64 2/4 2024-08-02T23:29:47.930 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : kernel-modules-5.14.0-480.el9.x86_64 3/4 2024-08-02T23:29:48.716 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : kernel-modules-core-5.14.0-480.el9.x86_64 4/4 2024-08-02T23:29:48.716 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-08-02T23:29:48.716 INFO:teuthology.orchestra.run.smithi073.stdout:Installed: 2024-08-02T23:29:48.716 INFO:teuthology.orchestra.run.smithi073.stdout: kernel-5.14.0-480.el9.x86_64 2024-08-02T23:29:48.716 INFO:teuthology.orchestra.run.smithi073.stdout: kernel-core-5.14.0-480.el9.x86_64 2024-08-02T23:29:48.716 INFO:teuthology.orchestra.run.smithi073.stdout: kernel-modules-5.14.0-480.el9.x86_64 2024-08-02T23:29:48.716 INFO:teuthology.orchestra.run.smithi073.stdout: kernel-modules-core-5.14.0-480.el9.x86_64 2024-08-02T23:29:48.716 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-08-02T23:29:48.716 INFO:teuthology.orchestra.run.smithi073.stdout:Complete! 2024-08-02T23:29:49.106 DEBUG:teuthology.orchestra.run.smithi073:> rpm -q kernel | sort -rV | head -n 1 2024-08-02T23:29:49.179 INFO:teuthology.orchestra.run.smithi073.stdout:kernel-5.14.0-480.el9.x86_64 2024-08-02T23:29:49.179 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-480.el9.x86_64 2024-08-02T23:29:49.179 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-437.el9.x86_64 Expected: 5.14.0-480.el9.x86_64 2024-08-02T23:29:49.179 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-08-02T23:29:49.179 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-08-02T23:29:49.180 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-08-02T23:29:49.180 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi073.front.sepia.ceph.com, path=None, version=distro) 2024-08-02T23:29:49.180 DEBUG:teuthology.orchestra.run.smithi073:> rpm -q kernel | sort -rV | head -n 1 2024-08-02T23:29:49.235 INFO:teuthology.orchestra.run.smithi073.stdout:kernel-5.14.0-480.el9.x86_64 2024-08-02T23:29:49.236 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-480.el9.x86_64 2024-08-02T23:29:49.236 DEBUG:teuthology.orchestra.run.smithi073:> sudo rpm -qi grub2-tools 2024-08-02T23:29:49.269 INFO:teuthology.orchestra.run.smithi073.stdout:Name : grub2-tools 2024-08-02T23:29:49.269 INFO:teuthology.orchestra.run.smithi073.stdout:Epoch : 1 2024-08-02T23:29:49.269 INFO:teuthology.orchestra.run.smithi073.stdout:Version : 2.06 2024-08-02T23:29:49.269 INFO:teuthology.orchestra.run.smithi073.stdout:Release : 68.el9 2024-08-02T23:29:49.269 INFO:teuthology.orchestra.run.smithi073.stdout:Architecture: x86_64 2024-08-02T23:29:49.269 INFO:teuthology.orchestra.run.smithi073.stdout:Install Date: Fri 19 Apr 2024 09:32:39 PM UTC 2024-08-02T23:29:49.269 INFO:teuthology.orchestra.run.smithi073.stdout:Group : Unspecified 2024-08-02T23:29:49.269 INFO:teuthology.orchestra.run.smithi073.stdout:Size : 8264254 2024-08-02T23:29:49.270 INFO:teuthology.orchestra.run.smithi073.stdout:License : GPLv3+ 2024-08-02T23:29:49.270 INFO:teuthology.orchestra.run.smithi073.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-08-02T23:29:49.270 INFO:teuthology.orchestra.run.smithi073.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-08-02T23:29:49.270 INFO:teuthology.orchestra.run.smithi073.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-08-02T23:29:49.270 INFO:teuthology.orchestra.run.smithi073.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-08-02T23:29:49.270 INFO:teuthology.orchestra.run.smithi073.stdout:Packager : builder@centos.org 2024-08-02T23:29:49.270 INFO:teuthology.orchestra.run.smithi073.stdout:Vendor : CentOS 2024-08-02T23:29:49.270 INFO:teuthology.orchestra.run.smithi073.stdout:URL : http://www.gnu.org/software/grub/ 2024-08-02T23:29:49.270 INFO:teuthology.orchestra.run.smithi073.stdout:Summary : Support tools for GRUB. 2024-08-02T23:29:49.270 INFO:teuthology.orchestra.run.smithi073.stdout:Description : 2024-08-02T23:29:49.270 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-08-02T23:29:49.270 INFO:teuthology.orchestra.run.smithi073.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-08-02T23:29:49.270 INFO:teuthology.orchestra.run.smithi073.stdout:customizable bootloader with modular architecture. It supports a rich 2024-08-02T23:29:49.271 INFO:teuthology.orchestra.run.smithi073.stdout:variety of kernel formats, file systems, computer architectures and 2024-08-02T23:29:49.271 INFO:teuthology.orchestra.run.smithi073.stdout:hardware devices. 2024-08-02T23:29:49.271 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-08-02T23:29:49.271 INFO:teuthology.orchestra.run.smithi073.stdout:This subpackage provides tools for support of all platforms. 2024-08-02T23:29:49.272 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-08-02T23:29:49.272 INFO:teuthology.task.kernel:Updating grub on smithi073 to boot 5.14.0-480.el9.x86_64 2024-08-02T23:29:49.272 DEBUG:teuthology.orchestra.run.smithi073:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-08-02T23:29:49.877 INFO:teuthology.orchestra.run.smithi073.stderr:Generating grub configuration file ... 2024-08-02T23:29:51.216 INFO:teuthology.orchestra.run.smithi073.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-08-02T23:29:51.258 INFO:teuthology.orchestra.run.smithi073.stderr:done 2024-08-02T23:29:51.261 DEBUG:teuthology.orchestra.run.smithi073:> mktemp 2024-08-02T23:29:51.276 INFO:teuthology.orchestra.run.smithi073.stdout:/tmp/tmp.7wNlDDq8L2 2024-08-02T23:29:51.276 DEBUG:teuthology.orchestra.run.smithi073:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.7wNlDDq8L2 2024-08-02T23:29:51.349 DEBUG:teuthology.orchestra.run.smithi073:> sudo chmod 0666 /tmp/tmp.7wNlDDq8L2 2024-08-02T23:29:51.487 DEBUG:teuthology.orchestra.remote:smithi073:/tmp/tmp.7wNlDDq8L2 is 6KB 2024-08-02T23:29:51.536 DEBUG:teuthology.orchestra.run.smithi073:> rm -fr /tmp/tmp.7wNlDDq8L2 2024-08-02T23:29:51.550 DEBUG:teuthology.orchestra.run.smithi073:> sudo /bin/ls /boot/loader/entries || true 2024-08-02T23:29:51.613 INFO:teuthology.orchestra.run.smithi073.stdout:7fc8828e06d0482694bcf4a447d13cb6-0-rescue.conf 2024-08-02T23:29:51.613 INFO:teuthology.orchestra.run.smithi073.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-239.el9.x86_64.conf 2024-08-02T23:29:51.613 INFO:teuthology.orchestra.run.smithi073.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-437.el9.x86_64.conf 2024-08-02T23:29:51.613 INFO:teuthology.orchestra.run.smithi073.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-480.el9.x86_64.conf 2024-08-02T23:29:51.614 DEBUG:teuthology.orchestra.run.smithi073:> sudo grub2-set-default 7fc8828e06d0482694bcf4a447d13cb6-5.14.0-480.el9.x86_64 2024-08-02T23:29:51.759 DEBUG:teuthology.orchestra.run.smithi073:> sudo shutdown -r now 2024-08-02T23:30:03.142 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: kernel-core-5.14.0-480.el9.x86_64 4/4 2024-08-02T23:30:03.183 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: kernel-modules-5.14.0-480.el9.x86_64 4/4 2024-08-02T23:30:03.582 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: kernel-5.14.0-480.el9.x86_64 4/4 2024-08-02T23:30:03.582 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : kernel-5.14.0-480.el9.x86_64 1/4 2024-08-02T23:30:03.582 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : kernel-core-5.14.0-480.el9.x86_64 2/4 2024-08-02T23:30:03.582 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : kernel-modules-5.14.0-480.el9.x86_64 3/4 2024-08-02T23:30:04.351 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : kernel-modules-core-5.14.0-480.el9.x86_64 4/4 2024-08-02T23:30:04.351 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-08-02T23:30:04.351 INFO:teuthology.orchestra.run.smithi131.stdout:Installed: 2024-08-02T23:30:04.351 INFO:teuthology.orchestra.run.smithi131.stdout: kernel-5.14.0-480.el9.x86_64 2024-08-02T23:30:04.351 INFO:teuthology.orchestra.run.smithi131.stdout: kernel-core-5.14.0-480.el9.x86_64 2024-08-02T23:30:04.351 INFO:teuthology.orchestra.run.smithi131.stdout: kernel-modules-5.14.0-480.el9.x86_64 2024-08-02T23:30:04.351 INFO:teuthology.orchestra.run.smithi131.stdout: kernel-modules-core-5.14.0-480.el9.x86_64 2024-08-02T23:30:04.352 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-08-02T23:30:04.352 INFO:teuthology.orchestra.run.smithi131.stdout:Complete! 2024-08-02T23:30:04.731 DEBUG:teuthology.orchestra.run.smithi131:> rpm -q kernel | sort -rV | head -n 1 2024-08-02T23:30:04.809 INFO:teuthology.orchestra.run.smithi131.stdout:kernel-5.14.0-480.el9.x86_64 2024-08-02T23:30:04.809 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-480.el9.x86_64 2024-08-02T23:30:04.809 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-437.el9.x86_64 Expected: 5.14.0-480.el9.x86_64 2024-08-02T23:30:04.809 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-08-02T23:30:04.809 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-08-02T23:30:04.809 INFO:teuthology.task.kernel:Installing distro kernel on mgr.x... 2024-08-02T23:30:04.809 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi131.front.sepia.ceph.com, path=None, version=distro) 2024-08-02T23:30:04.810 DEBUG:teuthology.orchestra.run.smithi131:> rpm -q kernel | sort -rV | head -n 1 2024-08-02T23:30:04.866 INFO:teuthology.orchestra.run.smithi131.stdout:kernel-5.14.0-480.el9.x86_64 2024-08-02T23:30:04.866 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-480.el9.x86_64 2024-08-02T23:30:04.867 DEBUG:teuthology.orchestra.run.smithi131:> sudo rpm -qi grub2-tools 2024-08-02T23:30:04.906 INFO:teuthology.orchestra.run.smithi131.stdout:Name : grub2-tools 2024-08-02T23:30:04.906 INFO:teuthology.orchestra.run.smithi131.stdout:Epoch : 1 2024-08-02T23:30:04.906 INFO:teuthology.orchestra.run.smithi131.stdout:Version : 2.06 2024-08-02T23:30:04.906 INFO:teuthology.orchestra.run.smithi131.stdout:Release : 68.el9 2024-08-02T23:30:04.906 INFO:teuthology.orchestra.run.smithi131.stdout:Architecture: x86_64 2024-08-02T23:30:04.906 INFO:teuthology.orchestra.run.smithi131.stdout:Install Date: Fri 19 Apr 2024 09:32:39 PM UTC 2024-08-02T23:30:04.906 INFO:teuthology.orchestra.run.smithi131.stdout:Group : Unspecified 2024-08-02T23:30:04.906 INFO:teuthology.orchestra.run.smithi131.stdout:Size : 8264254 2024-08-02T23:30:04.907 INFO:teuthology.orchestra.run.smithi131.stdout:License : GPLv3+ 2024-08-02T23:30:04.907 INFO:teuthology.orchestra.run.smithi131.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-08-02T23:30:04.907 INFO:teuthology.orchestra.run.smithi131.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-08-02T23:30:04.907 INFO:teuthology.orchestra.run.smithi131.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-08-02T23:30:04.907 INFO:teuthology.orchestra.run.smithi131.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-08-02T23:30:04.907 INFO:teuthology.orchestra.run.smithi131.stdout:Packager : builder@centos.org 2024-08-02T23:30:04.907 INFO:teuthology.orchestra.run.smithi131.stdout:Vendor : CentOS 2024-08-02T23:30:04.907 INFO:teuthology.orchestra.run.smithi131.stdout:URL : http://www.gnu.org/software/grub/ 2024-08-02T23:30:04.907 INFO:teuthology.orchestra.run.smithi131.stdout:Summary : Support tools for GRUB. 2024-08-02T23:30:04.907 INFO:teuthology.orchestra.run.smithi131.stdout:Description : 2024-08-02T23:30:04.907 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-08-02T23:30:04.907 INFO:teuthology.orchestra.run.smithi131.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-08-02T23:30:04.907 INFO:teuthology.orchestra.run.smithi131.stdout:customizable bootloader with modular architecture. It supports a rich 2024-08-02T23:30:04.908 INFO:teuthology.orchestra.run.smithi131.stdout:variety of kernel formats, file systems, computer architectures and 2024-08-02T23:30:04.908 INFO:teuthology.orchestra.run.smithi131.stdout:hardware devices. 2024-08-02T23:30:04.908 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-08-02T23:30:04.908 INFO:teuthology.orchestra.run.smithi131.stdout:This subpackage provides tools for support of all platforms. 2024-08-02T23:30:04.909 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-08-02T23:30:04.909 INFO:teuthology.task.kernel:Updating grub on smithi131 to boot 5.14.0-480.el9.x86_64 2024-08-02T23:30:04.909 DEBUG:teuthology.orchestra.run.smithi131:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-08-02T23:30:05.556 INFO:teuthology.orchestra.run.smithi131.stderr:Generating grub configuration file ... 2024-08-02T23:30:06.896 INFO:teuthology.orchestra.run.smithi131.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-08-02T23:30:06.937 INFO:teuthology.orchestra.run.smithi131.stderr:done 2024-08-02T23:30:06.939 DEBUG:teuthology.orchestra.run.smithi131:> mktemp 2024-08-02T23:30:06.955 INFO:teuthology.orchestra.run.smithi131.stdout:/tmp/tmp.12nQdG25jF 2024-08-02T23:30:06.955 DEBUG:teuthology.orchestra.run.smithi131:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.12nQdG25jF 2024-08-02T23:30:07.019 DEBUG:teuthology.orchestra.run.smithi131:> sudo chmod 0666 /tmp/tmp.12nQdG25jF 2024-08-02T23:30:07.157 DEBUG:teuthology.orchestra.remote:smithi131:/tmp/tmp.12nQdG25jF is 6KB 2024-08-02T23:30:07.205 DEBUG:teuthology.orchestra.run.smithi131:> rm -fr /tmp/tmp.12nQdG25jF 2024-08-02T23:30:07.219 DEBUG:teuthology.orchestra.run.smithi131:> sudo /bin/ls /boot/loader/entries || true 2024-08-02T23:30:07.280 INFO:teuthology.orchestra.run.smithi131.stdout:7fc8828e06d0482694bcf4a447d13cb6-0-rescue.conf 2024-08-02T23:30:07.280 INFO:teuthology.orchestra.run.smithi131.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-239.el9.x86_64.conf 2024-08-02T23:30:07.281 INFO:teuthology.orchestra.run.smithi131.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-437.el9.x86_64.conf 2024-08-02T23:30:07.281 INFO:teuthology.orchestra.run.smithi131.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-480.el9.x86_64.conf 2024-08-02T23:30:07.282 DEBUG:teuthology.orchestra.run.smithi131:> sudo grub2-set-default 7fc8828e06d0482694bcf4a447d13cb6-5.14.0-480.el9.x86_64 2024-08-02T23:30:07.463 DEBUG:teuthology.orchestra.run.smithi131:> sudo shutdown -r now 2024-08-02T23:30:21.772 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-08-02T23:30:21.772 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi073.front.sepia.ceph.com' 2024-08-02T23:30:21.773 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi073.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-02T23:30:37.478 INFO:teuthology.task.kernel:Checking client mgr.x for new kernel version... 2024-08-02T23:30:37.479 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi131.front.sepia.ceph.com' 2024-08-02T23:30:37.479 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi131.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-02T23:31:21.818 DEBUG:teuthology.orchestra.remote:timed out 2024-08-02T23:31:30.826 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi073.front.sepia.ceph.com' 2024-08-02T23:31:30.827 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi073.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-02T23:31:37.490 DEBUG:teuthology.orchestra.remote:timed out 2024-08-02T23:31:38.392 DEBUG:teuthology.orchestra.run.smithi073:> true 2024-08-02T23:31:38.770 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi073.front.sepia.ceph.com' 2024-08-02T23:31:38.771 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-480.el9.x86_64"... 2024-08-02T23:31:38.771 DEBUG:teuthology.orchestra.run.smithi073:> uname -r 2024-08-02T23:31:38.785 INFO:teuthology.orchestra.run.smithi073.stdout:5.14.0-480.el9.x86_64 2024-08-02T23:31:38.785 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-480.el9.x86_64 vs 5.14.0-480.el9.x86_64 2024-08-02T23:31:38.785 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-08-02T23:31:38.785 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-08-02T23:31:39.786 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-08-02T23:31:39.786 DEBUG:teuthology.orchestra.run.smithi073:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-08-02T23:31:39.872 INFO:teuthology.orchestra.run.smithi073.stdout:ttyS1 2024-08-02T23:31:39.928 DEBUG:teuthology.parallel:result is None 2024-08-02T23:31:46.491 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi131.front.sepia.ceph.com' 2024-08-02T23:31:46.492 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi131.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-02T23:31:46.827 DEBUG:teuthology.orchestra.run.smithi131:> true 2024-08-02T23:31:47.220 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi131.front.sepia.ceph.com' 2024-08-02T23:31:47.220 INFO:teuthology.task.kernel:Checking kernel version of mgr.x, want "5.14.0-480.el9.x86_64"... 2024-08-02T23:31:47.221 DEBUG:teuthology.orchestra.run.smithi131:> uname -r 2024-08-02T23:31:47.235 INFO:teuthology.orchestra.run.smithi131.stdout:5.14.0-480.el9.x86_64 2024-08-02T23:31:47.235 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-480.el9.x86_64 vs 5.14.0-480.el9.x86_64 2024-08-02T23:31:47.235 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-08-02T23:31:47.236 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-08-02T23:31:48.236 INFO:teuthology.task.kernel:Enabling kdb on mgr.x... 2024-08-02T23:31:48.237 DEBUG:teuthology.orchestra.run.smithi131:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-08-02T23:31:48.317 INFO:teuthology.orchestra.run.smithi131.stdout:ttyS1 2024-08-02T23:31:48.360 DEBUG:teuthology.parallel:result is None 2024-08-02T23:31:48.360 INFO:teuthology.run_tasks:Running task internal.base... 2024-08-02T23:31:48.368 INFO:teuthology.task.internal:Creating test directory... 2024-08-02T23:31:48.368 DEBUG:teuthology.orchestra.run.smithi073:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-08-02T23:31:48.371 DEBUG:teuthology.orchestra.run.smithi131:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-08-02T23:31:48.388 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-08-02T23:31:48.444 INFO:teuthology.run_tasks:Running task internal.archive... 2024-08-02T23:31:48.451 INFO:teuthology.task.internal:Creating archive directory... 2024-08-02T23:31:48.452 DEBUG:teuthology.orchestra.run.smithi073:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-08-02T23:31:48.454 DEBUG:teuthology.orchestra.run.smithi131:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-08-02T23:31:48.504 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-08-02T23:31:48.510 INFO:teuthology.task.internal:Enabling coredump saving... 2024-08-02T23:31:48.511 DEBUG:teuthology.orchestra.run.smithi073:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-08-02T23:31:48.513 DEBUG:teuthology.orchestra.run.smithi131:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-08-02T23:31:48.542 INFO:teuthology.orchestra.run.smithi073.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-02T23:31:48.554 INFO:teuthology.orchestra.run.smithi073.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-02T23:31:48.555 INFO:teuthology.orchestra.run.smithi131.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-02T23:31:48.565 INFO:teuthology.orchestra.run.smithi131.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-02T23:31:48.566 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-08-02T23:31:48.573 INFO:teuthology.task.internal:Configuring sudo... 2024-08-02T23:31:48.574 DEBUG:teuthology.orchestra.run.smithi073:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-08-02T23:31:48.599 DEBUG:teuthology.orchestra.run.smithi131:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-08-02T23:31:48.633 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-08-02T23:31:48.642 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-08-02T23:31:48.642 DEBUG:teuthology.orchestra.run.smithi073:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-08-02T23:31:48.670 DEBUG:teuthology.orchestra.run.smithi131:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-08-02T23:31:48.690 DEBUG:teuthology.orchestra.run.smithi073:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-08-02T23:31:48.776 DEBUG:teuthology.orchestra.run.smithi073:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-08-02T23:31:48.821 DEBUG:teuthology.orchestra.run.smithi073:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-08-02T23:31:48.908 DEBUG:teuthology.orchestra.run.smithi073:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-08-02T23:31:48.954 DEBUG:teuthology.orchestra.run.smithi073:> set -ex 2024-08-02T23:31:48.955 DEBUG:teuthology.orchestra.run.smithi073:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-08-02T23:31:49.023 DEBUG:teuthology.orchestra.run.smithi131:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-08-02T23:31:49.096 DEBUG:teuthology.orchestra.run.smithi131:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-08-02T23:31:49.159 DEBUG:teuthology.orchestra.run.smithi131:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-08-02T23:31:49.226 DEBUG:teuthology.orchestra.run.smithi131:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-08-02T23:31:49.303 DEBUG:teuthology.orchestra.run.smithi131:> set -ex 2024-08-02T23:31:49.303 DEBUG:teuthology.orchestra.run.smithi131:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-08-02T23:31:49.358 DEBUG:teuthology.orchestra.run.smithi073:> sudo service rsyslog restart 2024-08-02T23:31:49.365 DEBUG:teuthology.orchestra.run.smithi131:> sudo service rsyslog restart 2024-08-02T23:31:49.410 INFO:teuthology.orchestra.run.smithi073.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-08-02T23:31:49.487 INFO:teuthology.orchestra.run.smithi131.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-08-02T23:31:49.875 INFO:teuthology.run_tasks:Running task internal.timer... 2024-08-02T23:31:49.882 INFO:teuthology.task.internal:Starting timer... 2024-08-02T23:31:49.882 INFO:teuthology.run_tasks:Running task pcp... 2024-08-02T23:31:49.892 INFO:teuthology.run_tasks:Running task selinux... 2024-08-02T23:31:49.900 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:getty_t:s0']} 2024-08-02T23:31:49.901 DEBUG:teuthology.orchestra.run.smithi073:> sudo service auditd rotate 2024-08-02T23:31:49.974 INFO:teuthology.orchestra.run.smithi073.stdout:Rotating logs: 2024-08-02T23:31:49.976 DEBUG:teuthology.orchestra.run.smithi131:> sudo service auditd rotate 2024-08-02T23:31:50.025 INFO:teuthology.orchestra.run.smithi131.stdout:Rotating logs: 2024-08-02T23:31:50.027 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-08-02T23:31:50.027 DEBUG:teuthology.orchestra.run.smithi073:> /usr/sbin/getenforce 2024-08-02T23:31:50.053 INFO:teuthology.orchestra.run.smithi073.stdout:Permissive 2024-08-02T23:31:50.053 DEBUG:teuthology.orchestra.run.smithi131:> /usr/sbin/getenforce 2024-08-02T23:31:50.087 INFO:teuthology.orchestra.run.smithi131.stdout:Permissive 2024-08-02T23:31:50.087 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi073.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi131.front.sepia.ceph.com': 'permissive'} 2024-08-02T23:31:50.088 DEBUG:teuthology.orchestra.run.smithi073:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-08-02T23:31:50.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-02T23:31:50.125 DEBUG:teuthology.orchestra.run.smithi131:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-08-02T23:31:50.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-02T23:31:50.153 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-08-02T23:31:50.153 DEBUG:teuthology.orchestra.run.smithi073:> sudo /usr/sbin/setenforce permissive 2024-08-02T23:31:50.198 DEBUG:teuthology.orchestra.run.smithi131:> sudo /usr/sbin/setenforce permissive 2024-08-02T23:31:50.382 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-08-02T23:31:50.392 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-08-02T23:31:50.405 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-08-02T23:31:50.406 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi073.front.sepia.ceph.com,smithi131.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-08-02T23:36:34.540 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi073.front.sepia.ceph.com'), Remote(name='ubuntu@smithi131.front.sepia.ceph.com')] 2024-08-02T23:36:34.542 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi073.front.sepia.ceph.com' 2024-08-02T23:36:34.543 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi073.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-02T23:36:34.622 DEBUG:teuthology.orchestra.run.smithi073:> true 2024-08-02T23:36:34.711 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi073.front.sepia.ceph.com' 2024-08-02T23:36:34.712 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi131.front.sepia.ceph.com' 2024-08-02T23:36:34.712 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi131.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-02T23:36:34.794 DEBUG:teuthology.orchestra.run.smithi131:> true 2024-08-02T23:36:34.879 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi131.front.sepia.ceph.com' 2024-08-02T23:36:34.879 INFO:teuthology.run_tasks:Running task clock... 2024-08-02T23:36:34.893 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-08-02T23:36:34.893 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-08-02T23:36:34.894 DEBUG:teuthology.orchestra.run.smithi073:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-08-02T23:36:34.897 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-08-02T23:36:34.897 DEBUG:teuthology.orchestra.run.smithi131:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-08-02T23:36:34.928 INFO:teuthology.orchestra.run.smithi073.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-08-02T23:36:34.944 INFO:teuthology.orchestra.run.smithi073.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-08-02T23:36:34.950 INFO:teuthology.orchestra.run.smithi131.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-08-02T23:36:34.964 INFO:teuthology.orchestra.run.smithi131.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-08-02T23:36:34.970 INFO:teuthology.orchestra.run.smithi073.stderr:sudo: ntpd: command not found 2024-08-02T23:36:34.982 INFO:teuthology.orchestra.run.smithi073.stdout:506 Cannot talk to daemon 2024-08-02T23:36:34.989 INFO:teuthology.orchestra.run.smithi131.stderr:sudo: ntpd: command not found 2024-08-02T23:36:34.997 INFO:teuthology.orchestra.run.smithi073.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-08-02T23:36:35.000 INFO:teuthology.orchestra.run.smithi131.stdout:506 Cannot talk to daemon 2024-08-02T23:36:35.012 INFO:teuthology.orchestra.run.smithi073.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-08-02T23:36:35.014 INFO:teuthology.orchestra.run.smithi131.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-08-02T23:36:35.028 INFO:teuthology.orchestra.run.smithi131.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-08-02T23:36:35.075 INFO:teuthology.orchestra.run.smithi073.stderr:bash: line 1: ntpq: command not found 2024-08-02T23:36:35.079 INFO:teuthology.orchestra.run.smithi073.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-08-02T23:36:35.079 INFO:teuthology.orchestra.run.smithi073.stdout:=============================================================================== 2024-08-02T23:36:35.079 INFO:teuthology.orchestra.run.smithi073.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-08-02T23:36:35.079 INFO:teuthology.orchestra.run.smithi073.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-08-02T23:36:35.079 INFO:teuthology.orchestra.run.smithi073.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-08-02T23:36:35.079 INFO:teuthology.orchestra.run.smithi073.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-08-02T23:36:35.082 INFO:teuthology.orchestra.run.smithi131.stderr:bash: line 1: ntpq: command not found 2024-08-02T23:36:35.085 INFO:teuthology.orchestra.run.smithi131.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-08-02T23:36:35.086 INFO:teuthology.orchestra.run.smithi131.stdout:=============================================================================== 2024-08-02T23:36:35.086 INFO:teuthology.orchestra.run.smithi131.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-08-02T23:36:35.086 INFO:teuthology.orchestra.run.smithi131.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-08-02T23:36:35.086 INFO:teuthology.orchestra.run.smithi131.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-08-02T23:36:35.086 INFO:teuthology.orchestra.run.smithi131.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-08-02T23:36:35.087 INFO:teuthology.run_tasks:Running task install... 2024-08-02T23:36:35.095 DEBUG:teuthology.task.install:project ceph 2024-08-02T23:36:35.095 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'crimson', 'sha1': '26c3fb8e197dcf7a49a54d1f4c8a7362ee35a8ea'}} 2024-08-02T23:36:35.095 DEBUG:teuthology.task.install:config {'flavor': 'crimson', 'sha1': '26c3fb8e197dcf7a49a54d1f4c8a7362ee35a8ea'} 2024-08-02T23:36:35.095 INFO:teuthology.task.install:Using flavor: crimson 2024-08-02T23:36:35.101 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-08-02T23:36:35.101 INFO:teuthology.task.install:extra packages: [] 2024-08-02T23:36:35.101 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'crimson', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '26c3fb8e197dcf7a49a54d1f4c8a7362ee35a8ea', 'tag': None, 'wait_for_package': False} 2024-08-02T23:36:35.102 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=26c3fb8e197dcf7a49a54d1f4c8a7362ee35a8ea 2024-08-02T23:36:35.104 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'crimson', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '26c3fb8e197dcf7a49a54d1f4c8a7362ee35a8ea', 'tag': None, 'wait_for_package': False} 2024-08-02T23:36:35.104 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=26c3fb8e197dcf7a49a54d1f4c8a7362ee35a8ea 2024-08-02T23:36:35.253 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/squid-release/26c3fb8e197dcf7a49a54d1f4c8a7362ee35a8ea/centos/9/flavors/crimson/ 2024-08-02T23:36:35.253 INFO:teuthology.task.install.rpm:Package version is 19.1.0-1260.g26c3fb8e 2024-08-02T23:36:35.259 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/squid-release/26c3fb8e197dcf7a49a54d1f4c8a7362ee35a8ea/centos/9/flavors/crimson/ 2024-08-02T23:36:35.259 INFO:teuthology.task.install.rpm:Package version is 19.1.0-1260.g26c3fb8e 2024-08-02T23:36:35.362 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/squid-release/26c3fb8e197dcf7a49a54d1f4c8a7362ee35a8ea/centos/9/flavors/crimson/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/squid-release/26c3fb8e197dcf7a49a54d1f4c8a7362ee35a8ea/centos/9/flavors/crimson/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/squid-release/26c3fb8e197dcf7a49a54d1f4c8a7362ee35a8ea/centos/9/flavors/crimson/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-08-02T23:36:35.362 DEBUG:teuthology.orchestra.run.smithi073:> set -ex 2024-08-02T23:36:35.362 DEBUG:teuthology.orchestra.run.smithi073:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-08-02T23:36:35.370 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/squid-release/26c3fb8e197dcf7a49a54d1f4c8a7362ee35a8ea/centos/9/flavors/crimson/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/squid-release/26c3fb8e197dcf7a49a54d1f4c8a7362ee35a8ea/centos/9/flavors/crimson/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/squid-release/26c3fb8e197dcf7a49a54d1f4c8a7362ee35a8ea/centos/9/flavors/crimson/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-08-02T23:36:35.371 DEBUG:teuthology.orchestra.run.smithi131:> set -ex 2024-08-02T23:36:35.371 DEBUG:teuthology.orchestra.run.smithi131:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-08-02T23:36:35.388 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-08-02T23:36:35.388 DEBUG:teuthology.orchestra.run.smithi073:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/26c3fb8e197dcf7a49a54d1f4c8a7362ee35a8ea/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-08-02T23:36:35.396 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-08-02T23:36:35.396 DEBUG:teuthology.orchestra.run.smithi131:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/26c3fb8e197dcf7a49a54d1f4c8a7362ee35a8ea/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-08-02T23:36:35.454 DEBUG:teuthology.orchestra.run.smithi073:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-08-02T23:36:35.462 DEBUG:teuthology.orchestra.run.smithi131:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-08-02T23:36:35.527 DEBUG:teuthology.orchestra.run.smithi073:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-08-02T23:36:35.537 DEBUG:teuthology.orchestra.run.smithi131:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-08-02T23:36:35.591 INFO:teuthology.orchestra.run.smithi073.stdout:check_obsoletes = 1 2024-08-02T23:36:35.593 DEBUG:teuthology.orchestra.run.smithi073:> sudo yum clean all 2024-08-02T23:36:35.601 INFO:teuthology.orchestra.run.smithi131.stdout:check_obsoletes = 1 2024-08-02T23:36:35.602 DEBUG:teuthology.orchestra.run.smithi131:> sudo yum clean all 2024-08-02T23:36:35.836 INFO:teuthology.orchestra.run.smithi073.stdout:45 files removed 2024-08-02T23:36:35.840 INFO:teuthology.orchestra.run.smithi131.stdout:45 files removed 2024-08-02T23:36:35.865 DEBUG:teuthology.orchestra.run.smithi073:> sudo yum -y install ceph-radosgw 2024-08-02T23:36:35.870 DEBUG:teuthology.orchestra.run.smithi131:> sudo yum -y install ceph-radosgw 2024-08-02T23:36:36.449 INFO:teuthology.orchestra.run.smithi073.stdout:ceph packages for x86_64 285 kB/s | 87 kB 00:00 2024-08-02T23:36:36.470 INFO:teuthology.orchestra.run.smithi131.stdout:ceph packages for x86_64 270 kB/s | 87 kB 00:00 2024-08-02T23:36:36.783 INFO:teuthology.orchestra.run.smithi073.stdout:ceph noarch packages 41 kB/s | 12 kB 00:00 2024-08-02T23:36:36.796 INFO:teuthology.orchestra.run.smithi131.stdout:ceph noarch packages 42 kB/s | 12 kB 00:00 2024-08-02T23:36:37.109 INFO:teuthology.orchestra.run.smithi073.stdout:ceph source packages 7.0 kB/s | 2.0 kB 00:00 2024-08-02T23:36:37.121 INFO:teuthology.orchestra.run.smithi131.stdout:ceph source packages 7.1 kB/s | 2.0 kB 00:00 2024-08-02T23:36:37.897 INFO:teuthology.orchestra.run.smithi131.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 8.2 MB 00:00 2024-08-02T23:36:38.909 INFO:teuthology.orchestra.run.smithi073.stdout:CentOS Stream 9 - BaseOS 4.7 MB/s | 8.2 MB 00:01 2024-08-02T23:36:40.031 INFO:teuthology.orchestra.run.smithi131.stdout:CentOS Stream 9 - AppStream 19 MB/s | 20 MB 00:01 2024-08-02T23:36:40.693 INFO:teuthology.orchestra.run.smithi073.stdout:CentOS Stream 9 - AppStream 29 MB/s | 20 MB 00:00 2024-08-02T23:36:44.390 INFO:teuthology.orchestra.run.smithi131.stdout:CentOS Stream 9 - CRB 10 MB/s | 6.6 MB 00:00 2024-08-02T23:36:46.394 INFO:teuthology.orchestra.run.smithi073.stdout:CentOS Stream 9 - CRB 3.4 MB/s | 6.6 MB 00:01 2024-08-02T23:36:46.659 INFO:teuthology.orchestra.run.smithi131.stdout:CentOS Stream 9 - Extras packages 19 kB/s | 18 kB 00:00 2024-08-02T23:36:47.629 INFO:teuthology.orchestra.run.smithi131.stdout:Extra Packages for Enterprise Linux 26 MB/s | 22 MB 00:00 2024-08-02T23:36:47.986 INFO:teuthology.orchestra.run.smithi073.stdout:CentOS Stream 9 - Extras packages 69 kB/s | 18 kB 00:00 2024-08-02T23:36:49.216 INFO:teuthology.orchestra.run.smithi073.stdout:Extra Packages for Enterprise Linux 20 MB/s | 22 MB 00:01 2024-08-02T23:36:52.968 INFO:teuthology.orchestra.run.smithi131.stdout:lab-extras 48 kB/s | 1.7 kB 00:00 2024-08-02T23:36:54.646 INFO:teuthology.orchestra.run.smithi073.stdout:lab-extras 39 kB/s | 1.7 kB 00:00 2024-08-02T23:36:55.285 INFO:teuthology.orchestra.run.smithi131.stdout:Dependencies resolved. 2024-08-02T23:36:55.316 INFO:teuthology.orchestra.run.smithi131.stdout:================================================================================ 2024-08-02T23:36:55.316 INFO:teuthology.orchestra.run.smithi131.stdout: Package Arch Version Repository Size 2024-08-02T23:36:55.316 INFO:teuthology.orchestra.run.smithi131.stdout:================================================================================ 2024-08-02T23:36:55.316 INFO:teuthology.orchestra.run.smithi131.stdout:Installing: 2024-08-02T23:36:55.316 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-radosgw x86_64 2:19.1.0-1260.g26c3fb8e.el9 ceph 9.9 M 2024-08-02T23:36:55.316 INFO:teuthology.orchestra.run.smithi131.stdout:Upgrading: 2024-08-02T23:36:55.316 INFO:teuthology.orchestra.run.smithi131.stdout: librados2 x86_64 2:19.1.0-1260.g26c3fb8e.el9 ceph 3.2 M 2024-08-02T23:36:55.317 INFO:teuthology.orchestra.run.smithi131.stdout: librbd1 x86_64 2:19.1.0-1260.g26c3fb8e.el9 ceph 3.1 M 2024-08-02T23:36:55.317 INFO:teuthology.orchestra.run.smithi131.stdout:Installing dependencies: 2024-08-02T23:36:55.317 INFO:teuthology.orchestra.run.smithi131.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-08-02T23:36:55.317 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-base x86_64 2:19.1.0-1260.g26c3fb8e.el9 ceph 5.4 M 2024-08-02T23:36:55.317 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-common x86_64 2:19.1.0-1260.g26c3fb8e.el9 ceph 19 M 2024-08-02T23:36:55.317 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-selinux x86_64 2:19.1.0-1260.g26c3fb8e.el9 ceph 25 k 2024-08-02T23:36:55.317 INFO:teuthology.orchestra.run.smithi131.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-08-02T23:36:55.317 INFO:teuthology.orchestra.run.smithi131.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-08-02T23:36:55.317 INFO:teuthology.orchestra.run.smithi131.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-08-02T23:36:55.317 INFO:teuthology.orchestra.run.smithi131.stdout: libcephfs2 x86_64 2:19.1.0-1260.g26c3fb8e.el9 ceph 760 k 2024-08-02T23:36:55.317 INFO:teuthology.orchestra.run.smithi131.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-08-02T23:36:55.318 INFO:teuthology.orchestra.run.smithi131.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-08-02T23:36:55.318 INFO:teuthology.orchestra.run.smithi131.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-08-02T23:36:55.318 INFO:teuthology.orchestra.run.smithi131.stdout: libradosstriper1 x86_64 2:19.1.0-1260.g26c3fb8e.el9 ceph 501 k 2024-08-02T23:36:55.318 INFO:teuthology.orchestra.run.smithi131.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-08-02T23:36:55.318 INFO:teuthology.orchestra.run.smithi131.stdout: librgw2 x86_64 2:19.1.0-1260.g26c3fb8e.el9 ceph 5.1 M 2024-08-02T23:36:55.318 INFO:teuthology.orchestra.run.smithi131.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-08-02T23:36:55.318 INFO:teuthology.orchestra.run.smithi131.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-08-02T23:36:55.318 INFO:teuthology.orchestra.run.smithi131.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-08-02T23:36:55.318 INFO:teuthology.orchestra.run.smithi131.stdout: python3-ceph-argparse x86_64 2:19.1.0-1260.g26c3fb8e.el9 ceph 45 k 2024-08-02T23:36:55.319 INFO:teuthology.orchestra.run.smithi131.stdout: python3-ceph-common x86_64 2:19.1.0-1260.g26c3fb8e.el9 ceph 136 k 2024-08-02T23:36:55.319 INFO:teuthology.orchestra.run.smithi131.stdout: python3-cephfs x86_64 2:19.1.0-1260.g26c3fb8e.el9 ceph 188 k 2024-08-02T23:36:55.319 INFO:teuthology.orchestra.run.smithi131.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-08-02T23:36:55.319 INFO:teuthology.orchestra.run.smithi131.stdout: python3-rados x86_64 2:19.1.0-1260.g26c3fb8e.el9 ceph 372 k 2024-08-02T23:36:55.319 INFO:teuthology.orchestra.run.smithi131.stdout: python3-rbd x86_64 2:19.1.0-1260.g26c3fb8e.el9 ceph 350 k 2024-08-02T23:36:55.319 INFO:teuthology.orchestra.run.smithi131.stdout: python3-rgw x86_64 2:19.1.0-1260.g26c3fb8e.el9 ceph 116 k 2024-08-02T23:36:55.319 INFO:teuthology.orchestra.run.smithi131.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-08-02T23:36:55.320 INFO:teuthology.orchestra.run.smithi131.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-08-02T23:36:55.320 INFO:teuthology.orchestra.run.smithi131.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-08-02T23:36:55.320 INFO:teuthology.orchestra.run.smithi131.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-08-02T23:36:55.320 INFO:teuthology.orchestra.run.smithi131.stdout:Installing weak dependencies: 2024-08-02T23:36:55.320 INFO:teuthology.orchestra.run.smithi131.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-08-02T23:36:55.320 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-08-02T23:36:55.320 INFO:teuthology.orchestra.run.smithi131.stdout:Transaction Summary 2024-08-02T23:36:55.320 INFO:teuthology.orchestra.run.smithi131.stdout:================================================================================ 2024-08-02T23:36:55.320 INFO:teuthology.orchestra.run.smithi131.stdout:Install 30 Packages 2024-08-02T23:36:55.321 INFO:teuthology.orchestra.run.smithi131.stdout:Upgrade 2 Packages 2024-08-02T23:36:55.321 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-08-02T23:36:55.322 INFO:teuthology.orchestra.run.smithi131.stdout:Total download size: 58 M 2024-08-02T23:36:55.322 INFO:teuthology.orchestra.run.smithi131.stdout:Downloading Packages: 2024-08-02T23:36:56.403 INFO:teuthology.orchestra.run.smithi131.stdout:(1/32): ceph-base-19.1.0-1260.g26c3fb8e.el9.x86 9.3 MB/s | 5.4 MB 00:00 2024-08-02T23:36:56.462 INFO:teuthology.orchestra.run.smithi131.stdout:(2/32): ceph-selinux-19.1.0-1260.g26c3fb8e.el9. 431 kB/s | 25 kB 00:00 2024-08-02T23:36:56.562 INFO:teuthology.orchestra.run.smithi131.stdout:(3/32): libcephfs2-19.1.0-1260.g26c3fb8e.el9.x8 7.4 MB/s | 760 kB 00:00 2024-08-02T23:36:56.654 INFO:teuthology.orchestra.run.smithi131.stdout:(4/32): libradosstriper1-19.1.0-1260.g26c3fb8e. 5.4 MB/s | 501 kB 00:00 2024-08-02T23:36:56.862 INFO:teuthology.orchestra.run.smithi131.stdout:(5/32): ceph-common-19.1.0-1260.g26c3fb8e.el9.x 19 MB/s | 19 MB 00:01 2024-08-02T23:36:56.988 INFO:teuthology.orchestra.run.smithi131.stdout:(6/32): ceph-radosgw-19.1.0-1260.g26c3fb8e.el9. 8.5 MB/s | 9.9 MB 00:01 2024-08-02T23:36:56.989 INFO:teuthology.orchestra.run.smithi073.stdout:Dependencies resolved. 2024-08-02T23:36:57.007 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2024-08-02T23:36:57.007 INFO:teuthology.orchestra.run.smithi073.stdout: Package Arch Version Repository Size 2024-08-02T23:36:57.008 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2024-08-02T23:36:57.008 INFO:teuthology.orchestra.run.smithi073.stdout:Installing: 2024-08-02T23:36:57.008 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-radosgw x86_64 2:19.1.0-1260.g26c3fb8e.el9 ceph 9.9 M 2024-08-02T23:36:57.008 INFO:teuthology.orchestra.run.smithi073.stdout:Upgrading: 2024-08-02T23:36:57.008 INFO:teuthology.orchestra.run.smithi073.stdout: librados2 x86_64 2:19.1.0-1260.g26c3fb8e.el9 ceph 3.2 M 2024-08-02T23:36:57.008 INFO:teuthology.orchestra.run.smithi073.stdout: librbd1 x86_64 2:19.1.0-1260.g26c3fb8e.el9 ceph 3.1 M 2024-08-02T23:36:57.008 INFO:teuthology.orchestra.run.smithi073.stdout:Installing dependencies: 2024-08-02T23:36:57.008 INFO:teuthology.orchestra.run.smithi073.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-08-02T23:36:57.008 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-base x86_64 2:19.1.0-1260.g26c3fb8e.el9 ceph 5.4 M 2024-08-02T23:36:57.008 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-common x86_64 2:19.1.0-1260.g26c3fb8e.el9 ceph 19 M 2024-08-02T23:36:57.008 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-selinux x86_64 2:19.1.0-1260.g26c3fb8e.el9 ceph 25 k 2024-08-02T23:36:57.008 INFO:teuthology.orchestra.run.smithi073.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-08-02T23:36:57.009 INFO:teuthology.orchestra.run.smithi073.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-08-02T23:36:57.009 INFO:teuthology.orchestra.run.smithi073.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-08-02T23:36:57.009 INFO:teuthology.orchestra.run.smithi073.stdout: libcephfs2 x86_64 2:19.1.0-1260.g26c3fb8e.el9 ceph 760 k 2024-08-02T23:36:57.009 INFO:teuthology.orchestra.run.smithi073.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-08-02T23:36:57.009 INFO:teuthology.orchestra.run.smithi073.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-08-02T23:36:57.009 INFO:teuthology.orchestra.run.smithi073.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-08-02T23:36:57.009 INFO:teuthology.orchestra.run.smithi073.stdout: libradosstriper1 x86_64 2:19.1.0-1260.g26c3fb8e.el9 ceph 501 k 2024-08-02T23:36:57.009 INFO:teuthology.orchestra.run.smithi073.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-08-02T23:36:57.009 INFO:teuthology.orchestra.run.smithi073.stdout: librgw2 x86_64 2:19.1.0-1260.g26c3fb8e.el9 ceph 5.1 M 2024-08-02T23:36:57.009 INFO:teuthology.orchestra.run.smithi073.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-08-02T23:36:57.009 INFO:teuthology.orchestra.run.smithi073.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-08-02T23:36:57.009 INFO:teuthology.orchestra.run.smithi073.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-08-02T23:36:57.009 INFO:teuthology.orchestra.run.smithi073.stdout: python3-ceph-argparse x86_64 2:19.1.0-1260.g26c3fb8e.el9 ceph 45 k 2024-08-02T23:36:57.009 INFO:teuthology.orchestra.run.smithi073.stdout: python3-ceph-common x86_64 2:19.1.0-1260.g26c3fb8e.el9 ceph 136 k 2024-08-02T23:36:57.010 INFO:teuthology.orchestra.run.smithi073.stdout: python3-cephfs x86_64 2:19.1.0-1260.g26c3fb8e.el9 ceph 188 k 2024-08-02T23:36:57.010 INFO:teuthology.orchestra.run.smithi073.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-08-02T23:36:57.010 INFO:teuthology.orchestra.run.smithi073.stdout: python3-rados x86_64 2:19.1.0-1260.g26c3fb8e.el9 ceph 372 k 2024-08-02T23:36:57.010 INFO:teuthology.orchestra.run.smithi073.stdout: python3-rbd x86_64 2:19.1.0-1260.g26c3fb8e.el9 ceph 350 k 2024-08-02T23:36:57.010 INFO:teuthology.orchestra.run.smithi073.stdout: python3-rgw x86_64 2:19.1.0-1260.g26c3fb8e.el9 ceph 116 k 2024-08-02T23:36:57.010 INFO:teuthology.orchestra.run.smithi073.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-08-02T23:36:57.010 INFO:teuthology.orchestra.run.smithi073.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-08-02T23:36:57.010 INFO:teuthology.orchestra.run.smithi073.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-08-02T23:36:57.010 INFO:teuthology.orchestra.run.smithi073.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-08-02T23:36:57.010 INFO:teuthology.orchestra.run.smithi073.stdout:Installing weak dependencies: 2024-08-02T23:36:57.010 INFO:teuthology.orchestra.run.smithi073.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-08-02T23:36:57.010 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-08-02T23:36:57.010 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction Summary 2024-08-02T23:36:57.011 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2024-08-02T23:36:57.011 INFO:teuthology.orchestra.run.smithi073.stdout:Install 30 Packages 2024-08-02T23:36:57.011 INFO:teuthology.orchestra.run.smithi073.stdout:Upgrade 2 Packages 2024-08-02T23:36:57.011 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-08-02T23:36:57.011 INFO:teuthology.orchestra.run.smithi073.stdout:Total download size: 58 M 2024-08-02T23:36:57.011 INFO:teuthology.orchestra.run.smithi073.stdout:Downloading Packages: 2024-08-02T23:36:57.013 INFO:teuthology.orchestra.run.smithi131.stdout:(7/32): python3-ceph-argparse-19.1.0-1260.g26c3 300 kB/s | 45 kB 00:00 2024-08-02T23:36:57.038 INFO:teuthology.orchestra.run.smithi131.stdout:(8/32): python3-ceph-common-19.1.0-1260.g26c3fb 2.6 MB/s | 136 kB 00:00 2024-08-02T23:36:57.063 INFO:teuthology.orchestra.run.smithi131.stdout:(9/32): python3-cephfs-19.1.0-1260.g26c3fb8e.el 3.7 MB/s | 188 kB 00:00 2024-08-02T23:36:57.089 INFO:teuthology.orchestra.run.smithi131.stdout:(10/32): python3-rados-19.1.0-1260.g26c3fb8e.el 7.2 MB/s | 372 kB 00:00 2024-08-02T23:36:57.114 INFO:teuthology.orchestra.run.smithi131.stdout:(11/32): python3-rbd-19.1.0-1260.g26c3fb8e.el9. 6.8 MB/s | 350 kB 00:00 2024-08-02T23:36:57.148 INFO:teuthology.orchestra.run.smithi131.stdout:(12/32): python3-rgw-19.1.0-1260.g26c3fb8e.el9. 1.9 MB/s | 116 kB 00:00 2024-08-02T23:36:57.223 INFO:teuthology.orchestra.run.smithi131.stdout:(13/32): librgw2-19.1.0-1260.g26c3fb8e.el9.x86_ 8.9 MB/s | 5.1 MB 00:00 2024-08-02T23:36:57.423 INFO:teuthology.orchestra.run.smithi131.stdout:(14/32): boost-program-options-1.75.0-8.el9.x86 346 kB/s | 107 kB 00:00 2024-08-02T23:36:57.449 INFO:teuthology.orchestra.run.smithi131.stdout:(15/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 533 kB/s | 160 kB 00:00 2024-08-02T23:36:57.499 INFO:teuthology.orchestra.run.smithi131.stdout:(16/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 164 kB/s | 45 kB 00:00 2024-08-02T23:36:57.616 INFO:teuthology.orchestra.run.smithi131.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 462 kB/s | 42 kB 00:00 2024-08-02T23:36:57.642 INFO:teuthology.orchestra.run.smithi131.stdout:(18/32): librdkafka-1.6.1-102.el9.x86_64.rpm 3.0 MB/s | 662 kB 00:00 2024-08-02T23:36:57.692 INFO:teuthology.orchestra.run.smithi131.stdout:(19/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.3 MB/s | 292 kB 00:00 2024-08-02T23:36:57.775 INFO:teuthology.orchestra.run.smithi131.stdout:(20/32): qatlib-24.02.0-1.el9.x86_64.rpm 1.4 MB/s | 221 kB 00:00 2024-08-02T23:36:57.826 INFO:teuthology.orchestra.run.smithi131.stdout:(21/32): qatlib-service-24.02.0-1.el9.x86_64.rp 224 kB/s | 36 kB 00:00 2024-08-02T23:36:57.894 INFO:teuthology.orchestra.run.smithi131.stdout:(22/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 270 kB/s | 47 kB 00:00 2024-08-02T23:36:57.951 INFO:teuthology.orchestra.run.smithi131.stdout:(23/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.7 MB/s | 309 kB 00:00 2024-08-02T23:36:57.977 INFO:teuthology.orchestra.run.smithi131.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 2.1 MB/s | 49 kB 00:00 2024-08-02T23:36:57.981 INFO:teuthology.orchestra.run.smithi073.stdout:(1/32): ceph-base-19.1.0-1260.g26c3fb8e.el9.x86 10 MB/s | 5.4 MB 00:00 2024-08-02T23:36:58.002 INFO:teuthology.orchestra.run.smithi131.stdout:(25/32): libarrow-doc-9.0.0-11.el9.noarch.rpm 245 kB/s | 26 kB 00:00 2024-08-02T23:36:58.019 INFO:teuthology.orchestra.run.smithi131.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.6 MB/s | 67 kB 00:00 2024-08-02T23:36:58.031 INFO:teuthology.orchestra.run.smithi073.stdout:(2/32): ceph-selinux-19.1.0-1260.g26c3fb8e.el9. 502 kB/s | 25 kB 00:00 2024-08-02T23:36:58.053 INFO:teuthology.orchestra.run.smithi131.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 5.6 MB/s | 192 kB 00:00 2024-08-02T23:36:58.111 INFO:teuthology.orchestra.run.smithi131.stdout:(28/32): parquet-libs-9.0.0-11.el9.x86_64.rpm 7.5 MB/s | 839 kB 00:00 2024-08-02T23:36:58.131 INFO:teuthology.orchestra.run.smithi073.stdout:(3/32): libcephfs2-19.1.0-1260.g26c3fb8e.el9.x8 7.4 MB/s | 760 kB 00:00 2024-08-02T23:36:58.162 INFO:teuthology.orchestra.run.smithi131.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 15 MB/s | 1.6 MB 00:00 2024-08-02T23:36:58.207 INFO:teuthology.orchestra.run.smithi073.stdout:(4/32): libradosstriper1-19.1.0-1260.g26c3fb8e. 6.5 MB/s | 501 kB 00:00 2024-08-02T23:36:58.237 INFO:teuthology.orchestra.run.smithi131.stdout:(30/32): libarrow-9.0.0-11.el9.x86_64.rpm 11 MB/s | 4.4 MB 00:00 2024-08-02T23:36:58.424 INFO:teuthology.orchestra.run.smithi073.stdout:(5/32): ceph-common-19.1.0-1260.g26c3fb8e.el9.x 20 MB/s | 19 MB 00:00 2024-08-02T23:36:58.474 INFO:teuthology.orchestra.run.smithi073.stdout:(6/32): python3-ceph-argparse-19.1.0-1260.g26c3 907 kB/s | 45 kB 00:00 2024-08-02T23:36:58.549 INFO:teuthology.orchestra.run.smithi073.stdout:(7/32): librgw2-19.1.0-1260.g26c3fb8e.el9.x86_6 15 MB/s | 5.1 MB 00:00 2024-08-02T23:36:58.566 INFO:teuthology.orchestra.run.smithi073.stdout:(8/32): python3-ceph-common-19.1.0-1260.g26c3fb 1.4 MB/s | 136 kB 00:00 2024-08-02T23:36:58.570 INFO:teuthology.orchestra.run.smithi131.stdout:(31/32): librados2-19.1.0-1260.g26c3fb8e.el9.x8 7.0 MB/s | 3.2 MB 00:00 2024-08-02T23:36:58.591 INFO:teuthology.orchestra.run.smithi073.stdout:(9/32): python3-cephfs-19.1.0-1260.g26c3fb8e.el 4.4 MB/s | 188 kB 00:00 2024-08-02T23:36:58.616 INFO:teuthology.orchestra.run.smithi073.stdout:(10/32): python3-rados-19.1.0-1260.g26c3fb8e.el 7.2 MB/s | 372 kB 00:00 2024-08-02T23:36:58.629 INFO:teuthology.orchestra.run.smithi131.stdout:(32/32): librbd1-19.1.0-1260.g26c3fb8e.el9.x86_ 6.6 MB/s | 3.1 MB 00:00 2024-08-02T23:36:58.634 INFO:teuthology.orchestra.run.smithi131.stdout:-------------------------------------------------------------------------------- 2024-08-02T23:36:58.634 INFO:teuthology.orchestra.run.smithi131.stdout:Total 17 MB/s | 58 MB 00:03 2024-08-02T23:36:58.650 INFO:teuthology.orchestra.run.smithi073.stdout:(11/32): python3-rbd-19.1.0-1260.g26c3fb8e.el9. 5.8 MB/s | 350 kB 00:00 2024-08-02T23:36:58.730 INFO:teuthology.orchestra.run.smithi131.stdout:Running transaction check 2024-08-02T23:36:58.759 INFO:teuthology.orchestra.run.smithi073.stdout:(12/32): ceph-radosgw-19.1.0-1260.g26c3fb8e.el9 7.6 MB/s | 9.9 MB 00:01 2024-08-02T23:36:58.766 INFO:teuthology.orchestra.run.smithi131.stdout:Transaction check succeeded. 2024-08-02T23:36:58.766 INFO:teuthology.orchestra.run.smithi131.stdout:Running transaction test 2024-08-02T23:36:58.784 INFO:teuthology.orchestra.run.smithi073.stdout:(13/32): python3-rgw-19.1.0-1260.g26c3fb8e.el9. 693 kB/s | 116 kB 00:00 2024-08-02T23:36:58.859 INFO:teuthology.orchestra.run.smithi073.stdout:(14/32): boost-program-options-1.75.0-8.el9.x86 512 kB/s | 107 kB 00:00 2024-08-02T23:36:58.885 INFO:teuthology.orchestra.run.smithi073.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 452 kB/s | 45 kB 00:00 2024-08-02T23:36:58.910 INFO:teuthology.orchestra.run.smithi073.stdout:(16/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.0 MB/s | 160 kB 00:00 2024-08-02T23:36:58.952 INFO:teuthology.orchestra.run.smithi073.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 1.0 MB/s | 42 kB 00:00 2024-08-02T23:36:58.986 INFO:teuthology.orchestra.run.smithi073.stdout:(18/32): librdkafka-1.6.1-102.el9.x86_64.rpm 5.1 MB/s | 662 kB 00:00 2024-08-02T23:36:59.011 INFO:teuthology.orchestra.run.smithi073.stdout:(19/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.3 MB/s | 292 kB 00:00 2024-08-02T23:36:59.036 INFO:teuthology.orchestra.run.smithi073.stdout:(20/32): qatlib-service-24.02.0-1.el9.x86_64.rp 712 kB/s | 36 kB 00:00 2024-08-02T23:36:59.061 INFO:teuthology.orchestra.run.smithi073.stdout:(21/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 943 kB/s | 47 kB 00:00 2024-08-02T23:36:59.087 INFO:teuthology.orchestra.run.smithi073.stdout:(22/32): qatlib-24.02.0-1.el9.x86_64.rpm 1.6 MB/s | 221 kB 00:00 2024-08-02T23:36:59.195 INFO:teuthology.orchestra.run.smithi073.stdout:(23/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.9 MB/s | 309 kB 00:00 2024-08-02T23:36:59.220 INFO:teuthology.orchestra.run.smithi073.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 2.1 MB/s | 49 kB 00:00 2024-08-02T23:36:59.246 INFO:teuthology.orchestra.run.smithi073.stdout:(25/32): libarrow-doc-9.0.0-11.el9.noarch.rpm 240 kB/s | 26 kB 00:00 2024-08-02T23:36:59.271 INFO:teuthology.orchestra.run.smithi073.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-08-02T23:36:59.305 INFO:teuthology.orchestra.run.smithi073.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 5.6 MB/s | 192 kB 00:00 2024-08-02T23:36:59.331 INFO:teuthology.orchestra.run.smithi131.stdout:Transaction test succeeded. 2024-08-02T23:36:59.331 INFO:teuthology.orchestra.run.smithi131.stdout:Running transaction 2024-08-02T23:36:59.363 INFO:teuthology.orchestra.run.smithi073.stdout:(28/32): parquet-libs-9.0.0-11.el9.x86_64.rpm 7.0 MB/s | 839 kB 00:00 2024-08-02T23:36:59.405 INFO:teuthology.orchestra.run.smithi073.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 16 MB/s | 1.6 MB 00:00 2024-08-02T23:36:59.489 INFO:teuthology.orchestra.run.smithi073.stdout:(30/32): libarrow-9.0.0-11.el9.x86_64.rpm 10 MB/s | 4.4 MB 00:00 2024-08-02T23:36:59.781 INFO:teuthology.orchestra.run.smithi073.stdout:(31/32): librados2-19.1.0-1260.g26c3fb8e.el9.x8 7.7 MB/s | 3.2 MB 00:00 2024-08-02T23:36:59.839 INFO:teuthology.orchestra.run.smithi073.stdout:(32/32): librbd1-19.1.0-1260.g26c3fb8e.el9.x86_ 7.1 MB/s | 3.1 MB 00:00 2024-08-02T23:36:59.844 INFO:teuthology.orchestra.run.smithi073.stdout:-------------------------------------------------------------------------------- 2024-08-02T23:36:59.844 INFO:teuthology.orchestra.run.smithi073.stdout:Total 20 MB/s | 58 MB 00:02 2024-08-02T23:36:59.942 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction check 2024-08-02T23:36:59.977 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction check succeeded. 2024-08-02T23:36:59.978 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction test 2024-08-02T23:37:00.546 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction test succeeded. 2024-08-02T23:37:00.546 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction 2024-08-02T23:37:00.734 INFO:teuthology.orchestra.run.smithi131.stdout: Preparing : 1/1 2024-08-02T23:37:01.121 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-08-02T23:37:01.215 INFO:teuthology.orchestra.run.smithi131.stdout: Upgrading : librados2-2:19.1.0-1260.g26c3fb8e.el9.x86_64 2/34 2024-08-02T23:37:01.318 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: librados2-2:19.1.0-1260.g26c3fb8e.el9.x86_64 2/34 2024-08-02T23:37:01.410 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-rados-2:19.1.0-1260.g26c3fb8e.el9.x86_64 3/34 2024-08-02T23:37:01.433 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : libcephfs2-2:19.1.0-1260.g26c3fb8e.el9.x86_64 4/34 2024-08-02T23:37:01.526 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: libcephfs2-2:19.1.0-1260.g26c3fb8e.el9.x86_64 4/34 2024-08-02T23:37:01.554 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/34 2024-08-02T23:37:01.580 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 6/34 2024-08-02T23:37:01.603 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 7/34 2024-08-02T23:37:01.629 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 7/34 2024-08-02T23:37:01.822 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 8/34 2024-08-02T23:37:01.850 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 8/34 2024-08-02T23:37:01.879 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : liboath-2.6.7-2.el9.x86_64 9/34 2024-08-02T23:37:01.925 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-ceph-argparse-2:19.1.0-1260.g26c3fb8e.el9. 10/34 2024-08-02T23:37:01.950 INFO:teuthology.orchestra.run.smithi073.stdout: Preparing : 1/1 2024-08-02T23:37:01.952 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-cephfs-2:19.1.0-1260.g26c3fb8e.el9.x86_64 11/34 2024-08-02T23:37:02.037 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 12/34 2024-08-02T23:37:02.066 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : libradosstriper1-2:19.1.0-1260.g26c3fb8e.el9.x86_6 13/34 2024-08-02T23:37:02.156 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: libradosstriper1-2:19.1.0-1260.g26c3fb8e.el9.x86_6 13/34 2024-08-02T23:37:02.191 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : thrift-0.15.0-2.el9.x86_64 14/34 2024-08-02T23:37:02.304 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-08-02T23:37:02.342 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/34 2024-08-02T23:37:02.363 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-08-02T23:37:02.383 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-08-02T23:37:02.415 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-08-02T23:37:02.433 INFO:teuthology.orchestra.run.smithi073.stdout: Upgrading : librados2-2:19.1.0-1260.g26c3fb8e.el9.x86_64 2/34 2024-08-02T23:37:02.445 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/34 2024-08-02T23:37:02.477 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-08-02T23:37:02.529 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: librados2-2:19.1.0-1260.g26c3fb8e.el9.x86_64 2/34 2024-08-02T23:37:02.624 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-rados-2:19.1.0-1260.g26c3fb8e.el9.x86_64 3/34 2024-08-02T23:37:02.652 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : libcephfs2-2:19.1.0-1260.g26c3fb8e.el9.x86_64 4/34 2024-08-02T23:37:02.743 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: libcephfs2-2:19.1.0-1260.g26c3fb8e.el9.x86_64 4/34 2024-08-02T23:37:02.772 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/34 2024-08-02T23:37:02.806 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 6/34 2024-08-02T23:37:02.829 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 7/34 2024-08-02T23:37:02.843 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-08-02T23:37:02.853 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 7/34 2024-08-02T23:37:02.892 INFO:teuthology.orchestra.run.smithi131.stdout: Upgrading : librbd1-2:19.1.0-1260.g26c3fb8e.el9.x86_64 22/34 2024-08-02T23:37:02.996 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: librbd1-2:19.1.0-1260.g26c3fb8e.el9.x86_64 22/34 2024-08-02T23:37:03.031 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 8/34 2024-08-02T23:37:03.031 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-rbd-2:19.1.0-1260.g26c3fb8e.el9.x86_64 23/34 2024-08-02T23:37:03.059 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 8/34 2024-08-02T23:37:03.067 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-08-02T23:37:03.088 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : liboath-2.6.7-2.el9.x86_64 9/34 2024-08-02T23:37:03.134 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-ceph-argparse-2:19.1.0-1260.g26c3fb8e.el9. 10/34 2024-08-02T23:37:03.153 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-cephfs-2:19.1.0-1260.g26c3fb8e.el9.x86_64 11/34 2024-08-02T23:37:03.226 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 12/34 2024-08-02T23:37:03.250 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : libradosstriper1-2:19.1.0-1260.g26c3fb8e.el9.x86_6 13/34 2024-08-02T23:37:03.339 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: libradosstriper1-2:19.1.0-1260.g26c3fb8e.el9.x86_6 13/34 2024-08-02T23:37:03.374 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : thrift-0.15.0-2.el9.x86_64 14/34 2024-08-02T23:37:03.492 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-08-02T23:37:03.525 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/34 2024-08-02T23:37:03.558 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 25/34 2024-08-02T23:37:03.566 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-08-02T23:37:03.588 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : librgw2-2:19.1.0-1260.g26c3fb8e.el9.x86_64 26/34 2024-08-02T23:37:03.598 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-08-02T23:37:03.628 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/34 2024-08-02T23:37:03.660 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-08-02T23:37:03.661 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: librgw2-2:19.1.0-1260.g26c3fb8e.el9.x86_64 26/34 2024-08-02T23:37:03.736 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-rgw-2:19.1.0-1260.g26c3fb8e.el9.x86_64 27/34 2024-08-02T23:37:03.766 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-ceph-common-2:19.1.0-1260.g26c3fb8e.el9.x8 28/34 2024-08-02T23:37:04.002 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-08-02T23:37:04.049 INFO:teuthology.orchestra.run.smithi073.stdout: Upgrading : librbd1-2:19.1.0-1260.g26c3fb8e.el9.x86_64 22/34 2024-08-02T23:37:04.143 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: librbd1-2:19.1.0-1260.g26c3fb8e.el9.x86_64 22/34 2024-08-02T23:37:04.170 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-rbd-2:19.1.0-1260.g26c3fb8e.el9.x86_64 23/34 2024-08-02T23:37:04.206 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-08-02T23:37:04.719 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 25/34 2024-08-02T23:37:04.751 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : librgw2-2:19.1.0-1260.g26c3fb8e.el9.x86_64 26/34 2024-08-02T23:37:04.823 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: librgw2-2:19.1.0-1260.g26c3fb8e.el9.x86_64 26/34 2024-08-02T23:37:04.899 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-rgw-2:19.1.0-1260.g26c3fb8e.el9.x86_64 27/34 2024-08-02T23:37:04.928 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-ceph-common-2:19.1.0-1260.g26c3fb8e.el9.x8 28/34 2024-08-02T23:37:06.002 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: ceph-common-2:19.1.0-1260.g26c3fb8e.el9.x86_64 29/34 2024-08-02T23:37:06.038 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : ceph-common-2:19.1.0-1260.g26c3fb8e.el9.x86_64 29/34 2024-08-02T23:37:06.613 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: ceph-common-2:19.1.0-1260.g26c3fb8e.el9.x86_64 29/34 2024-08-02T23:37:06.652 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : ceph-base-2:19.1.0-1260.g26c3fb8e.el9.x86_64 30/34 2024-08-02T23:37:06.741 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: ceph-base-2:19.1.0-1260.g26c3fb8e.el9.x86_64 30/34 2024-08-02T23:37:06.741 INFO:teuthology.orchestra.run.smithi131.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-08-02T23:37:06.742 INFO:teuthology.orchestra.run.smithi131.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-08-02T23:37:06.742 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-08-02T23:37:06.779 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : ceph-selinux-2:19.1.0-1260.g26c3fb8e.el9.x86_64 31/34 2024-08-02T23:37:07.075 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: ceph-common-2:19.1.0-1260.g26c3fb8e.el9.x86_64 29/34 2024-08-02T23:37:07.117 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : ceph-common-2:19.1.0-1260.g26c3fb8e.el9.x86_64 29/34 2024-08-02T23:37:07.689 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: ceph-common-2:19.1.0-1260.g26c3fb8e.el9.x86_64 29/34 2024-08-02T23:37:07.729 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : ceph-base-2:19.1.0-1260.g26c3fb8e.el9.x86_64 30/34 2024-08-02T23:37:07.812 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: ceph-base-2:19.1.0-1260.g26c3fb8e.el9.x86_64 30/34 2024-08-02T23:37:07.812 INFO:teuthology.orchestra.run.smithi073.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-08-02T23:37:07.812 INFO:teuthology.orchestra.run.smithi073.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-08-02T23:37:07.812 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-08-02T23:37:07.847 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : ceph-selinux-2:19.1.0-1260.g26c3fb8e.el9.x86_64 31/34 2024-08-02T23:37:16.696 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: ceph-selinux-2:19.1.0-1260.g26c3fb8e.el9.x86_64 31/34 2024-08-02T23:37:16.697 INFO:teuthology.orchestra.run.smithi131.stdout:skipping the directory /sys 2024-08-02T23:37:16.697 INFO:teuthology.orchestra.run.smithi131.stdout:skipping the directory /proc 2024-08-02T23:37:16.697 INFO:teuthology.orchestra.run.smithi131.stdout:skipping the directory /mnt 2024-08-02T23:37:16.697 INFO:teuthology.orchestra.run.smithi131.stdout:skipping the directory /var/tmp 2024-08-02T23:37:16.697 INFO:teuthology.orchestra.run.smithi131.stdout:skipping the directory /home 2024-08-02T23:37:16.697 INFO:teuthology.orchestra.run.smithi131.stdout:skipping the directory /root 2024-08-02T23:37:16.698 INFO:teuthology.orchestra.run.smithi131.stdout:skipping the directory /tmp 2024-08-02T23:37:16.698 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-08-02T23:37:17.443 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: ceph-selinux-2:19.1.0-1260.g26c3fb8e.el9.x86_64 31/34 2024-08-02T23:37:17.443 INFO:teuthology.orchestra.run.smithi073.stdout:skipping the directory /sys 2024-08-02T23:37:17.443 INFO:teuthology.orchestra.run.smithi073.stdout:skipping the directory /proc 2024-08-02T23:37:17.443 INFO:teuthology.orchestra.run.smithi073.stdout:skipping the directory /mnt 2024-08-02T23:37:17.443 INFO:teuthology.orchestra.run.smithi073.stdout:skipping the directory /var/tmp 2024-08-02T23:37:17.444 INFO:teuthology.orchestra.run.smithi073.stdout:skipping the directory /home 2024-08-02T23:37:17.444 INFO:teuthology.orchestra.run.smithi073.stdout:skipping the directory /root 2024-08-02T23:37:17.444 INFO:teuthology.orchestra.run.smithi073.stdout:skipping the directory /tmp 2024-08-02T23:37:17.444 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-08-02T23:37:17.681 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : ceph-radosgw-2:19.1.0-1260.g26c3fb8e.el9.x86_64 32/34 2024-08-02T23:37:17.706 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: ceph-radosgw-2:19.1.0-1260.g26c3fb8e.el9.x86_64 32/34 2024-08-02T23:37:17.706 INFO:teuthology.orchestra.run.smithi131.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-02T23:37:17.706 INFO:teuthology.orchestra.run.smithi131.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-08-02T23:37:17.706 INFO:teuthology.orchestra.run.smithi131.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-08-02T23:37:17.707 INFO:teuthology.orchestra.run.smithi131.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-08-02T23:37:17.707 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-08-02T23:37:17.708 INFO:teuthology.orchestra.run.smithi131.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-08-02T23:37:17.775 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-08-02T23:37:17.775 INFO:teuthology.orchestra.run.smithi131.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-08-02T23:37:18.449 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : ceph-radosgw-2:19.1.0-1260.g26c3fb8e.el9.x86_64 32/34 2024-08-02T23:37:18.474 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: ceph-radosgw-2:19.1.0-1260.g26c3fb8e.el9.x86_64 32/34 2024-08-02T23:37:18.474 INFO:teuthology.orchestra.run.smithi073.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-02T23:37:18.474 INFO:teuthology.orchestra.run.smithi073.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-08-02T23:37:18.474 INFO:teuthology.orchestra.run.smithi073.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-08-02T23:37:18.474 INFO:teuthology.orchestra.run.smithi073.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-08-02T23:37:18.474 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-08-02T23:37:18.476 INFO:teuthology.orchestra.run.smithi073.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-08-02T23:37:18.559 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-08-02T23:37:18.560 INFO:teuthology.orchestra.run.smithi073.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-08-02T23:37:19.024 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-08-02T23:37:19.024 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : ceph-base-2:19.1.0-1260.g26c3fb8e.el9.x86_64 1/34 2024-08-02T23:37:19.024 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : ceph-common-2:19.1.0-1260.g26c3fb8e.el9.x86_64 2/34 2024-08-02T23:37:19.024 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : ceph-radosgw-2:19.1.0-1260.g26c3fb8e.el9.x86_64 3/34 2024-08-02T23:37:19.024 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : ceph-selinux-2:19.1.0-1260.g26c3fb8e.el9.x86_64 4/34 2024-08-02T23:37:19.024 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : libcephfs2-2:19.1.0-1260.g26c3fb8e.el9.x86_64 5/34 2024-08-02T23:37:19.025 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : libradosstriper1-2:19.1.0-1260.g26c3fb8e.el9.x86_6 6/34 2024-08-02T23:37:19.025 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : librgw2-2:19.1.0-1260.g26c3fb8e.el9.x86_64 7/34 2024-08-02T23:37:19.025 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-ceph-argparse-2:19.1.0-1260.g26c3fb8e.el9. 8/34 2024-08-02T23:37:19.025 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-ceph-common-2:19.1.0-1260.g26c3fb8e.el9.x8 9/34 2024-08-02T23:37:19.025 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-cephfs-2:19.1.0-1260.g26c3fb8e.el9.x86_64 10/34 2024-08-02T23:37:19.025 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-rados-2:19.1.0-1260.g26c3fb8e.el9.x86_64 11/34 2024-08-02T23:37:19.025 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-rbd-2:19.1.0-1260.g26c3fb8e.el9.x86_64 12/34 2024-08-02T23:37:19.025 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-rgw-2:19.1.0-1260.g26c3fb8e.el9.x86_64 13/34 2024-08-02T23:37:19.025 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-08-02T23:37:19.025 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-08-02T23:37:19.025 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-08-02T23:37:19.025 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-08-02T23:37:19.025 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-08-02T23:37:19.026 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-08-02T23:37:19.026 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-08-02T23:37:19.026 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-08-02T23:37:19.026 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-08-02T23:37:19.026 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-08-02T23:37:19.026 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 24/34 2024-08-02T23:37:19.026 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 25/34 2024-08-02T23:37:19.026 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-08-02T23:37:19.026 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-08-02T23:37:19.026 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 28/34 2024-08-02T23:37:19.026 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-08-02T23:37:19.026 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-08-02T23:37:19.027 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : librados2-2:19.1.0-1260.g26c3fb8e.el9.x86_64 31/34 2024-08-02T23:37:19.027 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-08-02T23:37:19.027 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : librbd1-2:19.1.0-1260.g26c3fb8e.el9.x86_64 33/34 2024-08-02T23:37:19.774 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-08-02T23:37:19.775 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-08-02T23:37:19.775 INFO:teuthology.orchestra.run.smithi131.stdout:Upgraded: 2024-08-02T23:37:19.775 INFO:teuthology.orchestra.run.smithi131.stdout: librados2-2:19.1.0-1260.g26c3fb8e.el9.x86_64 2024-08-02T23:37:19.775 INFO:teuthology.orchestra.run.smithi131.stdout: librbd1-2:19.1.0-1260.g26c3fb8e.el9.x86_64 2024-08-02T23:37:19.775 INFO:teuthology.orchestra.run.smithi131.stdout:Installed: 2024-08-02T23:37:19.775 INFO:teuthology.orchestra.run.smithi131.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-08-02T23:37:19.775 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-base-2:19.1.0-1260.g26c3fb8e.el9.x86_64 2024-08-02T23:37:19.775 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-common-2:19.1.0-1260.g26c3fb8e.el9.x86_64 2024-08-02T23:37:19.775 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-radosgw-2:19.1.0-1260.g26c3fb8e.el9.x86_64 2024-08-02T23:37:19.775 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-selinux-2:19.1.0-1260.g26c3fb8e.el9.x86_64 2024-08-02T23:37:19.776 INFO:teuthology.orchestra.run.smithi131.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-08-02T23:37:19.776 INFO:teuthology.orchestra.run.smithi131.stdout: libarrow-9.0.0-11.el9.x86_64 2024-08-02T23:37:19.776 INFO:teuthology.orchestra.run.smithi131.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-08-02T23:37:19.776 INFO:teuthology.orchestra.run.smithi131.stdout: libcephfs2-2:19.1.0-1260.g26c3fb8e.el9.x86_64 2024-08-02T23:37:19.776 INFO:teuthology.orchestra.run.smithi131.stdout: liboath-2.6.7-2.el9.x86_64 2024-08-02T23:37:19.776 INFO:teuthology.orchestra.run.smithi131.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-08-02T23:37:19.776 INFO:teuthology.orchestra.run.smithi131.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-08-02T23:37:19.776 INFO:teuthology.orchestra.run.smithi131.stdout: libradosstriper1-2:19.1.0-1260.g26c3fb8e.el9.x86_64 2024-08-02T23:37:19.776 INFO:teuthology.orchestra.run.smithi131.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-08-02T23:37:19.776 INFO:teuthology.orchestra.run.smithi131.stdout: librgw2-2:19.1.0-1260.g26c3fb8e.el9.x86_64 2024-08-02T23:37:19.776 INFO:teuthology.orchestra.run.smithi131.stdout: libunwind-1.6.2-1.el9.x86_64 2024-08-02T23:37:19.776 INFO:teuthology.orchestra.run.smithi131.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-08-02T23:37:19.776 INFO:teuthology.orchestra.run.smithi131.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-08-02T23:37:19.776 INFO:teuthology.orchestra.run.smithi131.stdout: python3-ceph-argparse-2:19.1.0-1260.g26c3fb8e.el9.x86_64 2024-08-02T23:37:19.777 INFO:teuthology.orchestra.run.smithi131.stdout: python3-ceph-common-2:19.1.0-1260.g26c3fb8e.el9.x86_64 2024-08-02T23:37:19.777 INFO:teuthology.orchestra.run.smithi131.stdout: python3-cephfs-2:19.1.0-1260.g26c3fb8e.el9.x86_64 2024-08-02T23:37:19.777 INFO:teuthology.orchestra.run.smithi131.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-08-02T23:37:19.777 INFO:teuthology.orchestra.run.smithi131.stdout: python3-rados-2:19.1.0-1260.g26c3fb8e.el9.x86_64 2024-08-02T23:37:19.777 INFO:teuthology.orchestra.run.smithi131.stdout: python3-rbd-2:19.1.0-1260.g26c3fb8e.el9.x86_64 2024-08-02T23:37:19.777 INFO:teuthology.orchestra.run.smithi131.stdout: python3-rgw-2:19.1.0-1260.g26c3fb8e.el9.x86_64 2024-08-02T23:37:19.777 INFO:teuthology.orchestra.run.smithi131.stdout: qatlib-24.02.0-1.el9.x86_64 2024-08-02T23:37:19.777 INFO:teuthology.orchestra.run.smithi131.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-08-02T23:37:19.777 INFO:teuthology.orchestra.run.smithi131.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-08-02T23:37:19.777 INFO:teuthology.orchestra.run.smithi131.stdout: re2-1:20211101-3.el9.x86_64 2024-08-02T23:37:19.777 INFO:teuthology.orchestra.run.smithi131.stdout: thrift-0.15.0-2.el9.x86_64 2024-08-02T23:37:19.777 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-08-02T23:37:19.777 INFO:teuthology.orchestra.run.smithi131.stdout:Complete! 2024-08-02T23:37:19.782 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-08-02T23:37:19.782 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : ceph-base-2:19.1.0-1260.g26c3fb8e.el9.x86_64 1/34 2024-08-02T23:37:19.782 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : ceph-common-2:19.1.0-1260.g26c3fb8e.el9.x86_64 2/34 2024-08-02T23:37:19.782 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : ceph-radosgw-2:19.1.0-1260.g26c3fb8e.el9.x86_64 3/34 2024-08-02T23:37:19.782 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : ceph-selinux-2:19.1.0-1260.g26c3fb8e.el9.x86_64 4/34 2024-08-02T23:37:19.782 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : libcephfs2-2:19.1.0-1260.g26c3fb8e.el9.x86_64 5/34 2024-08-02T23:37:19.782 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : libradosstriper1-2:19.1.0-1260.g26c3fb8e.el9.x86_6 6/34 2024-08-02T23:37:19.782 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : librgw2-2:19.1.0-1260.g26c3fb8e.el9.x86_64 7/34 2024-08-02T23:37:19.783 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-ceph-argparse-2:19.1.0-1260.g26c3fb8e.el9. 8/34 2024-08-02T23:37:19.783 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-ceph-common-2:19.1.0-1260.g26c3fb8e.el9.x8 9/34 2024-08-02T23:37:19.783 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-cephfs-2:19.1.0-1260.g26c3fb8e.el9.x86_64 10/34 2024-08-02T23:37:19.783 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-rados-2:19.1.0-1260.g26c3fb8e.el9.x86_64 11/34 2024-08-02T23:37:19.783 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-rbd-2:19.1.0-1260.g26c3fb8e.el9.x86_64 12/34 2024-08-02T23:37:19.783 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-rgw-2:19.1.0-1260.g26c3fb8e.el9.x86_64 13/34 2024-08-02T23:37:19.783 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-08-02T23:37:19.783 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-08-02T23:37:19.783 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-08-02T23:37:19.783 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-08-02T23:37:19.783 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-08-02T23:37:19.783 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-08-02T23:37:19.783 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-08-02T23:37:19.783 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-08-02T23:37:19.783 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-08-02T23:37:19.784 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-08-02T23:37:19.784 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 24/34 2024-08-02T23:37:19.784 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 25/34 2024-08-02T23:37:19.784 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-08-02T23:37:19.784 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-08-02T23:37:19.785 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 28/34 2024-08-02T23:37:19.785 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-08-02T23:37:19.785 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-08-02T23:37:19.785 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : librados2-2:19.1.0-1260.g26c3fb8e.el9.x86_64 31/34 2024-08-02T23:37:19.785 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-08-02T23:37:19.785 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : librbd1-2:19.1.0-1260.g26c3fb8e.el9.x86_64 33/34 2024-08-02T23:37:20.168 DEBUG:teuthology.orchestra.run.smithi131:> sudo yum -y install ceph-test 2024-08-02T23:37:20.633 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-08-02T23:37:20.633 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-08-02T23:37:20.633 INFO:teuthology.orchestra.run.smithi073.stdout:Upgraded: 2024-08-02T23:37:20.633 INFO:teuthology.orchestra.run.smithi073.stdout: librados2-2:19.1.0-1260.g26c3fb8e.el9.x86_64 2024-08-02T23:37:20.633 INFO:teuthology.orchestra.run.smithi073.stdout: librbd1-2:19.1.0-1260.g26c3fb8e.el9.x86_64 2024-08-02T23:37:20.633 INFO:teuthology.orchestra.run.smithi073.stdout:Installed: 2024-08-02T23:37:20.633 INFO:teuthology.orchestra.run.smithi073.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-08-02T23:37:20.634 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-base-2:19.1.0-1260.g26c3fb8e.el9.x86_64 2024-08-02T23:37:20.634 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-common-2:19.1.0-1260.g26c3fb8e.el9.x86_64 2024-08-02T23:37:20.634 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-radosgw-2:19.1.0-1260.g26c3fb8e.el9.x86_64 2024-08-02T23:37:20.634 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-selinux-2:19.1.0-1260.g26c3fb8e.el9.x86_64 2024-08-02T23:37:20.634 INFO:teuthology.orchestra.run.smithi073.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-08-02T23:37:20.634 INFO:teuthology.orchestra.run.smithi073.stdout: libarrow-9.0.0-11.el9.x86_64 2024-08-02T23:37:20.634 INFO:teuthology.orchestra.run.smithi073.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-08-02T23:37:20.634 INFO:teuthology.orchestra.run.smithi073.stdout: libcephfs2-2:19.1.0-1260.g26c3fb8e.el9.x86_64 2024-08-02T23:37:20.634 INFO:teuthology.orchestra.run.smithi073.stdout: liboath-2.6.7-2.el9.x86_64 2024-08-02T23:37:20.634 INFO:teuthology.orchestra.run.smithi073.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-08-02T23:37:20.634 INFO:teuthology.orchestra.run.smithi073.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-08-02T23:37:20.634 INFO:teuthology.orchestra.run.smithi073.stdout: libradosstriper1-2:19.1.0-1260.g26c3fb8e.el9.x86_64 2024-08-02T23:37:20.634 INFO:teuthology.orchestra.run.smithi073.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-08-02T23:37:20.635 INFO:teuthology.orchestra.run.smithi073.stdout: librgw2-2:19.1.0-1260.g26c3fb8e.el9.x86_64 2024-08-02T23:37:20.635 INFO:teuthology.orchestra.run.smithi073.stdout: libunwind-1.6.2-1.el9.x86_64 2024-08-02T23:37:20.635 INFO:teuthology.orchestra.run.smithi073.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-08-02T23:37:20.635 INFO:teuthology.orchestra.run.smithi073.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-08-02T23:37:20.635 INFO:teuthology.orchestra.run.smithi073.stdout: python3-ceph-argparse-2:19.1.0-1260.g26c3fb8e.el9.x86_64 2024-08-02T23:37:20.635 INFO:teuthology.orchestra.run.smithi073.stdout: python3-ceph-common-2:19.1.0-1260.g26c3fb8e.el9.x86_64 2024-08-02T23:37:20.635 INFO:teuthology.orchestra.run.smithi073.stdout: python3-cephfs-2:19.1.0-1260.g26c3fb8e.el9.x86_64 2024-08-02T23:37:20.635 INFO:teuthology.orchestra.run.smithi073.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-08-02T23:37:20.635 INFO:teuthology.orchestra.run.smithi073.stdout: python3-rados-2:19.1.0-1260.g26c3fb8e.el9.x86_64 2024-08-02T23:37:20.635 INFO:teuthology.orchestra.run.smithi073.stdout: python3-rbd-2:19.1.0-1260.g26c3fb8e.el9.x86_64 2024-08-02T23:37:20.635 INFO:teuthology.orchestra.run.smithi073.stdout: python3-rgw-2:19.1.0-1260.g26c3fb8e.el9.x86_64 2024-08-02T23:37:20.635 INFO:teuthology.orchestra.run.smithi073.stdout: qatlib-24.02.0-1.el9.x86_64 2024-08-02T23:37:20.635 INFO:teuthology.orchestra.run.smithi073.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-08-02T23:37:20.635 INFO:teuthology.orchestra.run.smithi073.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-08-02T23:37:20.636 INFO:teuthology.orchestra.run.smithi073.stdout: re2-1:20211101-3.el9.x86_64 2024-08-02T23:37:20.636 INFO:teuthology.orchestra.run.smithi073.stdout: thrift-0.15.0-2.el9.x86_64 2024-08-02T23:37:20.636 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-08-02T23:37:20.636 INFO:teuthology.orchestra.run.smithi073.stdout:Complete! 2024-08-02T23:37:20.721 INFO:teuthology.orchestra.run.smithi131.stdout:Last metadata expiration check: 0:00:28 ago on Fri 02 Aug 2024 11:36:52 PM UTC. 2024-08-02T23:37:20.848 INFO:teuthology.orchestra.run.smithi131.stdout:Dependencies resolved. 2024-08-02T23:37:20.849 INFO:teuthology.orchestra.run.smithi131.stdout:================================================================================ 2024-08-02T23:37:20.849 INFO:teuthology.orchestra.run.smithi131.stdout: Package Arch Version Repository Size 2024-08-02T23:37:20.849 INFO:teuthology.orchestra.run.smithi131.stdout:================================================================================ 2024-08-02T23:37:20.849 INFO:teuthology.orchestra.run.smithi131.stdout:Installing: 2024-08-02T23:37:20.849 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-test x86_64 2:19.1.0-1260.g26c3fb8e.el9 ceph 124 M 2024-08-02T23:37:20.849 INFO:teuthology.orchestra.run.smithi131.stdout:Installing dependencies: 2024-08-02T23:37:20.849 INFO:teuthology.orchestra.run.smithi131.stdout: libasan x86_64 11.4.1-3.el9 appstream 418 k 2024-08-02T23:37:20.850 INFO:teuthology.orchestra.run.smithi131.stdout: libcephsqlite x86_64 2:19.1.0-1260.g26c3fb8e.el9 ceph 158 k 2024-08-02T23:37:20.850 INFO:teuthology.orchestra.run.smithi131.stdout: libubsan x86_64 11.4.1-3.el9 appstream 187 k 2024-08-02T23:37:20.850 INFO:teuthology.orchestra.run.smithi131.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-08-02T23:37:20.850 INFO:teuthology.orchestra.run.smithi131.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-08-02T23:37:20.850 INFO:teuthology.orchestra.run.smithi131.stdout: yaml-cpp x86_64 0.6.3-5.el9 epel 122 k 2024-08-02T23:37:20.850 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-08-02T23:37:20.850 INFO:teuthology.orchestra.run.smithi131.stdout:Transaction Summary 2024-08-02T23:37:20.850 INFO:teuthology.orchestra.run.smithi131.stdout:================================================================================ 2024-08-02T23:37:20.850 INFO:teuthology.orchestra.run.smithi131.stdout:Install 7 Packages 2024-08-02T23:37:20.850 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-08-02T23:37:20.851 INFO:teuthology.orchestra.run.smithi131.stdout:Total download size: 125 M 2024-08-02T23:37:20.851 INFO:teuthology.orchestra.run.smithi131.stdout:Installed size: 1.1 G 2024-08-02T23:37:20.851 INFO:teuthology.orchestra.run.smithi131.stdout:Downloading Packages: 2024-08-02T23:37:20.984 DEBUG:teuthology.orchestra.run.smithi073:> sudo yum -y install ceph-test 2024-08-02T23:37:21.312 INFO:teuthology.orchestra.run.smithi131.stdout:(1/7): libasan-11.4.1-3.el9.x86_64.rpm 1.5 MB/s | 418 kB 00:00 2024-08-02T23:37:21.338 INFO:teuthology.orchestra.run.smithi131.stdout:(2/7): libcephsqlite-19.1.0-1260.g26c3fb8e.el9. 538 kB/s | 158 kB 00:00 2024-08-02T23:37:21.371 INFO:teuthology.orchestra.run.smithi131.stdout:(3/7): libubsan-11.4.1-3.el9.x86_64.rpm 3.1 MB/s | 187 kB 00:00 2024-08-02T23:37:21.421 INFO:teuthology.orchestra.run.smithi131.stdout:(4/7): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 64 kB 00:00 2024-08-02T23:37:21.480 INFO:teuthology.orchestra.run.smithi131.stdout:(5/7): yaml-cpp-0.6.3-5.el9.x86_64.rpm 2.0 MB/s | 122 kB 00:00 2024-08-02T23:37:21.536 INFO:teuthology.orchestra.run.smithi073.stdout:Last metadata expiration check: 0:00:27 ago on Fri 02 Aug 2024 11:36:54 PM UTC. 2024-08-02T23:37:21.564 INFO:teuthology.orchestra.run.smithi131.stdout:(6/7): socat-1.7.4.1-6.el9.x86_64.rpm 1.3 MB/s | 304 kB 00:00 2024-08-02T23:37:21.662 INFO:teuthology.orchestra.run.smithi073.stdout:Dependencies resolved. 2024-08-02T23:37:21.666 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2024-08-02T23:37:21.666 INFO:teuthology.orchestra.run.smithi073.stdout: Package Arch Version Repository Size 2024-08-02T23:37:21.666 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2024-08-02T23:37:21.666 INFO:teuthology.orchestra.run.smithi073.stdout:Installing: 2024-08-02T23:37:21.666 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-test x86_64 2:19.1.0-1260.g26c3fb8e.el9 ceph 124 M 2024-08-02T23:37:21.667 INFO:teuthology.orchestra.run.smithi073.stdout:Installing dependencies: 2024-08-02T23:37:21.667 INFO:teuthology.orchestra.run.smithi073.stdout: libasan x86_64 11.4.1-3.el9 appstream 418 k 2024-08-02T23:37:21.667 INFO:teuthology.orchestra.run.smithi073.stdout: libcephsqlite x86_64 2:19.1.0-1260.g26c3fb8e.el9 ceph 158 k 2024-08-02T23:37:21.667 INFO:teuthology.orchestra.run.smithi073.stdout: libubsan x86_64 11.4.1-3.el9 appstream 187 k 2024-08-02T23:37:21.667 INFO:teuthology.orchestra.run.smithi073.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-08-02T23:37:21.667 INFO:teuthology.orchestra.run.smithi073.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-08-02T23:37:21.667 INFO:teuthology.orchestra.run.smithi073.stdout: yaml-cpp x86_64 0.6.3-5.el9 epel 122 k 2024-08-02T23:37:21.668 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-08-02T23:37:21.668 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction Summary 2024-08-02T23:37:21.668 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2024-08-02T23:37:21.668 INFO:teuthology.orchestra.run.smithi073.stdout:Install 7 Packages 2024-08-02T23:37:21.668 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-08-02T23:37:21.668 INFO:teuthology.orchestra.run.smithi073.stdout:Total download size: 125 M 2024-08-02T23:37:21.668 INFO:teuthology.orchestra.run.smithi073.stdout:Installed size: 1.1 G 2024-08-02T23:37:21.668 INFO:teuthology.orchestra.run.smithi073.stdout:Downloading Packages: 2024-08-02T23:37:22.364 INFO:teuthology.orchestra.run.smithi073.stdout:(1/7): libcephsqlite-19.1.0-1260.g26c3fb8e.el9. 386 kB/s | 158 kB 00:00 2024-08-02T23:37:22.714 INFO:teuthology.orchestra.run.smithi073.stdout:(2/7): libasan-11.4.1-3.el9.x86_64.rpm 552 kB/s | 418 kB 00:00 2024-08-02T23:37:22.873 INFO:teuthology.orchestra.run.smithi073.stdout:(3/7): libubsan-11.4.1-3.el9.x86_64.rpm 367 kB/s | 187 kB 00:00 2024-08-02T23:37:22.898 INFO:teuthology.orchestra.run.smithi073.stdout:(4/7): socat-1.7.4.1-6.el9.x86_64.rpm 1.6 MB/s | 304 kB 00:00 2024-08-02T23:37:22.957 INFO:teuthology.orchestra.run.smithi073.stdout:(5/7): yaml-cpp-0.6.3-5.el9.x86_64.rpm 2.0 MB/s | 122 kB 00:00 2024-08-02T23:37:22.982 INFO:teuthology.orchestra.run.smithi073.stdout:(6/7): xmlstarlet-1.6.1-20.el9.x86_64.rpm 586 kB/s | 64 kB 00:00 2024-08-02T23:37:25.464 INFO:teuthology.orchestra.run.smithi131.stdout:(7/7): ceph-test-19.1.0-1260.g26c3fb8e.el9.x86_ 28 MB/s | 124 MB 00:04 2024-08-02T23:37:25.466 INFO:teuthology.orchestra.run.smithi131.stdout:-------------------------------------------------------------------------------- 2024-08-02T23:37:25.466 INFO:teuthology.orchestra.run.smithi131.stdout:Total 27 MB/s | 125 MB 00:04 2024-08-02T23:37:25.520 INFO:teuthology.orchestra.run.smithi131.stdout:Running transaction check 2024-08-02T23:37:25.541 INFO:teuthology.orchestra.run.smithi131.stdout:Transaction check succeeded. 2024-08-02T23:37:25.541 INFO:teuthology.orchestra.run.smithi131.stdout:Running transaction test 2024-08-02T23:37:26.357 INFO:teuthology.orchestra.run.smithi131.stdout:Transaction test succeeded. 2024-08-02T23:37:26.358 INFO:teuthology.orchestra.run.smithi131.stdout:Running transaction 2024-08-02T23:37:26.750 INFO:teuthology.orchestra.run.smithi073.stdout:(7/7): ceph-test-19.1.0-1260.g26c3fb8e.el9.x86_ 26 MB/s | 124 MB 00:04 2024-08-02T23:37:26.753 INFO:teuthology.orchestra.run.smithi073.stdout:-------------------------------------------------------------------------------- 2024-08-02T23:37:26.753 INFO:teuthology.orchestra.run.smithi073.stdout:Total 25 MB/s | 125 MB 00:05 2024-08-02T23:37:26.808 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction check 2024-08-02T23:37:26.831 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction check succeeded. 2024-08-02T23:37:26.831 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction test 2024-08-02T23:37:27.425 INFO:teuthology.orchestra.run.smithi131.stdout: Preparing : 1/1 2024-08-02T23:37:27.505 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : yaml-cpp-0.6.3-5.el9.x86_64 1/7 2024-08-02T23:37:27.558 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 2/7 2024-08-02T23:37:27.586 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 3/7 2024-08-02T23:37:27.615 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : libubsan-11.4.1-3.el9.x86_64 4/7 2024-08-02T23:37:27.644 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction test succeeded. 2024-08-02T23:37:27.644 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction 2024-08-02T23:37:27.656 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : libasan-11.4.1-3.el9.x86_64 5/7 2024-08-02T23:37:27.683 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : libcephsqlite-2:19.1.0-1260.g26c3fb8e.el9.x86_64 6/7 2024-08-02T23:37:28.751 INFO:teuthology.orchestra.run.smithi073.stdout: Preparing : 1/1 2024-08-02T23:37:28.798 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : yaml-cpp-0.6.3-5.el9.x86_64 1/7 2024-08-02T23:37:28.842 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 2/7 2024-08-02T23:37:28.870 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 3/7 2024-08-02T23:37:28.899 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : libubsan-11.4.1-3.el9.x86_64 4/7 2024-08-02T23:37:28.941 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : libasan-11.4.1-3.el9.x86_64 5/7 2024-08-02T23:37:28.968 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : libcephsqlite-2:19.1.0-1260.g26c3fb8e.el9.x86_64 6/7 2024-08-02T23:37:44.691 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: libcephsqlite-2:19.1.0-1260.g26c3fb8e.el9.x86_64 6/7 2024-08-02T23:37:44.765 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : ceph-test-2:19.1.0-1260.g26c3fb8e.el9.x86_64 7/7 2024-08-02T23:37:45.978 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: libcephsqlite-2:19.1.0-1260.g26c3fb8e.el9.x86_64 6/7 2024-08-02T23:37:46.058 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : ceph-test-2:19.1.0-1260.g26c3fb8e.el9.x86_64 7/7 2024-08-02T23:37:53.306 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: ceph-test-2:19.1.0-1260.g26c3fb8e.el9.x86_64 7/7 2024-08-02T23:37:53.307 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : ceph-test-2:19.1.0-1260.g26c3fb8e.el9.x86_64 1/7 2024-08-02T23:37:53.307 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : libcephsqlite-2:19.1.0-1260.g26c3fb8e.el9.x86_64 2/7 2024-08-02T23:37:53.307 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : libasan-11.4.1-3.el9.x86_64 3/7 2024-08-02T23:37:53.307 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : libubsan-11.4.1-3.el9.x86_64 4/7 2024-08-02T23:37:53.307 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 5/7 2024-08-02T23:37:53.307 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 6/7 2024-08-02T23:37:53.651 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : yaml-cpp-0.6.3-5.el9.x86_64 7/7 2024-08-02T23:37:53.651 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-08-02T23:37:53.651 INFO:teuthology.orchestra.run.smithi131.stdout:Installed: 2024-08-02T23:37:53.651 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-test-2:19.1.0-1260.g26c3fb8e.el9.x86_64 2024-08-02T23:37:53.652 INFO:teuthology.orchestra.run.smithi131.stdout: libasan-11.4.1-3.el9.x86_64 2024-08-02T23:37:53.652 INFO:teuthology.orchestra.run.smithi131.stdout: libcephsqlite-2:19.1.0-1260.g26c3fb8e.el9.x86_64 2024-08-02T23:37:53.653 INFO:teuthology.orchestra.run.smithi131.stdout: libubsan-11.4.1-3.el9.x86_64 2024-08-02T23:37:53.653 INFO:teuthology.orchestra.run.smithi131.stdout: socat-1.7.4.1-6.el9.x86_64 2024-08-02T23:37:53.653 INFO:teuthology.orchestra.run.smithi131.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-08-02T23:37:53.653 INFO:teuthology.orchestra.run.smithi131.stdout: yaml-cpp-0.6.3-5.el9.x86_64 2024-08-02T23:37:53.653 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-08-02T23:37:53.653 INFO:teuthology.orchestra.run.smithi131.stdout:Complete! 2024-08-02T23:37:53.910 DEBUG:teuthology.orchestra.run.smithi131:> sudo yum -y install ceph 2024-08-02T23:37:54.463 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: ceph-test-2:19.1.0-1260.g26c3fb8e.el9.x86_64 7/7 2024-08-02T23:37:54.463 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : ceph-test-2:19.1.0-1260.g26c3fb8e.el9.x86_64 1/7 2024-08-02T23:37:54.464 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : libcephsqlite-2:19.1.0-1260.g26c3fb8e.el9.x86_64 2/7 2024-08-02T23:37:54.464 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : libasan-11.4.1-3.el9.x86_64 3/7 2024-08-02T23:37:54.464 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : libubsan-11.4.1-3.el9.x86_64 4/7 2024-08-02T23:37:54.464 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 5/7 2024-08-02T23:37:54.464 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 6/7 2024-08-02T23:37:54.465 INFO:teuthology.orchestra.run.smithi131.stdout:Last metadata expiration check: 0:01:02 ago on Fri 02 Aug 2024 11:36:52 PM UTC. 2024-08-02T23:37:54.593 INFO:teuthology.orchestra.run.smithi131.stdout:Dependencies resolved. 2024-08-02T23:37:54.595 INFO:teuthology.orchestra.run.smithi131.stdout:================================================================================ 2024-08-02T23:37:54.595 INFO:teuthology.orchestra.run.smithi131.stdout: Package Arch Version Repository Size 2024-08-02T23:37:54.596 INFO:teuthology.orchestra.run.smithi131.stdout:================================================================================ 2024-08-02T23:37:54.596 INFO:teuthology.orchestra.run.smithi131.stdout:Installing: 2024-08-02T23:37:54.596 INFO:teuthology.orchestra.run.smithi131.stdout: ceph x86_64 2:19.1.0-1260.g26c3fb8e.el9 ceph 6.6 k 2024-08-02T23:37:54.596 INFO:teuthology.orchestra.run.smithi131.stdout:Installing dependencies: 2024-08-02T23:37:54.596 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-mds x86_64 2:19.1.0-1260.g26c3fb8e.el9 ceph 2.3 M 2024-08-02T23:37:54.596 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-mgr x86_64 2:19.1.0-1260.g26c3fb8e.el9 ceph 1.5 M 2024-08-02T23:37:54.596 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-mgr-modules-core noarch 2:19.1.0-1260.g26c3fb8e.el9 ceph-noarch 249 k 2024-08-02T23:37:54.597 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-mon x86_64 2:19.1.0-1260.g26c3fb8e.el9 ceph 4.9 M 2024-08-02T23:37:54.597 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-osd x86_64 2:19.1.0-1260.g26c3fb8e.el9 ceph 120 M 2024-08-02T23:37:54.597 INFO:teuthology.orchestra.run.smithi131.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-08-02T23:37:54.597 INFO:teuthology.orchestra.run.smithi131.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-08-02T23:37:54.597 INFO:teuthology.orchestra.run.smithi131.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-08-02T23:37:54.597 INFO:teuthology.orchestra.run.smithi131.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-08-02T23:37:54.597 INFO:teuthology.orchestra.run.smithi131.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-08-02T23:37:54.597 INFO:teuthology.orchestra.run.smithi131.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-08-02T23:37:54.597 INFO:teuthology.orchestra.run.smithi131.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-08-02T23:37:54.597 INFO:teuthology.orchestra.run.smithi131.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-08-02T23:37:54.598 INFO:teuthology.orchestra.run.smithi131.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-08-02T23:37:54.598 INFO:teuthology.orchestra.run.smithi131.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-08-02T23:37:54.598 INFO:teuthology.orchestra.run.smithi131.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-08-02T23:37:54.598 INFO:teuthology.orchestra.run.smithi131.stdout: python3-jaraco-collections 2024-08-02T23:37:54.598 INFO:teuthology.orchestra.run.smithi131.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-08-02T23:37:54.598 INFO:teuthology.orchestra.run.smithi131.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-08-02T23:37:54.598 INFO:teuthology.orchestra.run.smithi131.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-08-02T23:37:54.598 INFO:teuthology.orchestra.run.smithi131.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-08-02T23:37:54.598 INFO:teuthology.orchestra.run.smithi131.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-08-02T23:37:54.599 INFO:teuthology.orchestra.run.smithi131.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-08-02T23:37:54.599 INFO:teuthology.orchestra.run.smithi131.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-08-02T23:37:54.599 INFO:teuthology.orchestra.run.smithi131.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-08-02T23:37:54.599 INFO:teuthology.orchestra.run.smithi131.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-08-02T23:37:54.599 INFO:teuthology.orchestra.run.smithi131.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-08-02T23:37:54.599 INFO:teuthology.orchestra.run.smithi131.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-08-02T23:37:54.599 INFO:teuthology.orchestra.run.smithi131.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-08-02T23:37:54.599 INFO:teuthology.orchestra.run.smithi131.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-08-02T23:37:54.599 INFO:teuthology.orchestra.run.smithi131.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-08-02T23:37:54.599 INFO:teuthology.orchestra.run.smithi131.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-08-02T23:37:54.599 INFO:teuthology.orchestra.run.smithi131.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-08-02T23:37:54.599 INFO:teuthology.orchestra.run.smithi131.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-08-02T23:37:54.599 INFO:teuthology.orchestra.run.smithi131.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-08-02T23:37:54.599 INFO:teuthology.orchestra.run.smithi131.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-08-02T23:37:54.600 INFO:teuthology.orchestra.run.smithi131.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-08-02T23:37:54.600 INFO:teuthology.orchestra.run.smithi131.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-08-02T23:37:54.600 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-08-02T23:37:54.600 INFO:teuthology.orchestra.run.smithi131.stdout:Transaction Summary 2024-08-02T23:37:54.600 INFO:teuthology.orchestra.run.smithi131.stdout:================================================================================ 2024-08-02T23:37:54.600 INFO:teuthology.orchestra.run.smithi131.stdout:Install 38 Packages 2024-08-02T23:37:54.600 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-08-02T23:37:54.601 INFO:teuthology.orchestra.run.smithi131.stdout:Total download size: 134 M 2024-08-02T23:37:54.601 INFO:teuthology.orchestra.run.smithi131.stdout:Installed size: 1.4 G 2024-08-02T23:37:54.601 INFO:teuthology.orchestra.run.smithi131.stdout:Downloading Packages: 2024-08-02T23:37:54.789 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : yaml-cpp-0.6.3-5.el9.x86_64 7/7 2024-08-02T23:37:54.789 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-08-02T23:37:54.790 INFO:teuthology.orchestra.run.smithi073.stdout:Installed: 2024-08-02T23:37:54.790 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-test-2:19.1.0-1260.g26c3fb8e.el9.x86_64 2024-08-02T23:37:54.790 INFO:teuthology.orchestra.run.smithi073.stdout: libasan-11.4.1-3.el9.x86_64 2024-08-02T23:37:54.790 INFO:teuthology.orchestra.run.smithi073.stdout: libcephsqlite-2:19.1.0-1260.g26c3fb8e.el9.x86_64 2024-08-02T23:37:54.790 INFO:teuthology.orchestra.run.smithi073.stdout: libubsan-11.4.1-3.el9.x86_64 2024-08-02T23:37:54.790 INFO:teuthology.orchestra.run.smithi073.stdout: socat-1.7.4.1-6.el9.x86_64 2024-08-02T23:37:54.790 INFO:teuthology.orchestra.run.smithi073.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-08-02T23:37:54.790 INFO:teuthology.orchestra.run.smithi073.stdout: yaml-cpp-0.6.3-5.el9.x86_64 2024-08-02T23:37:54.790 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-08-02T23:37:54.791 INFO:teuthology.orchestra.run.smithi073.stdout:Complete! 2024-08-02T23:37:55.091 DEBUG:teuthology.orchestra.run.smithi073:> sudo yum -y install ceph 2024-08-02T23:37:55.641 INFO:teuthology.orchestra.run.smithi073.stdout:Last metadata expiration check: 0:01:01 ago on Fri 02 Aug 2024 11:36:54 PM UTC. 2024-08-02T23:37:55.749 INFO:teuthology.orchestra.run.smithi131.stdout:(1/38): ceph-19.1.0-1260.g26c3fb8e.el9.x86_64.r 40 kB/s | 6.6 kB 00:00 2024-08-02T23:37:55.770 INFO:teuthology.orchestra.run.smithi073.stdout:Dependencies resolved. 2024-08-02T23:37:55.773 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2024-08-02T23:37:55.773 INFO:teuthology.orchestra.run.smithi073.stdout: Package Arch Version Repository Size 2024-08-02T23:37:55.773 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2024-08-02T23:37:55.773 INFO:teuthology.orchestra.run.smithi073.stdout:Installing: 2024-08-02T23:37:55.773 INFO:teuthology.orchestra.run.smithi073.stdout: ceph x86_64 2:19.1.0-1260.g26c3fb8e.el9 ceph 6.6 k 2024-08-02T23:37:55.773 INFO:teuthology.orchestra.run.smithi073.stdout:Installing dependencies: 2024-08-02T23:37:55.773 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-mds x86_64 2:19.1.0-1260.g26c3fb8e.el9 ceph 2.3 M 2024-08-02T23:37:55.773 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-mgr x86_64 2:19.1.0-1260.g26c3fb8e.el9 ceph 1.5 M 2024-08-02T23:37:55.773 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-mgr-modules-core noarch 2:19.1.0-1260.g26c3fb8e.el9 ceph-noarch 249 k 2024-08-02T23:37:55.774 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-mon x86_64 2:19.1.0-1260.g26c3fb8e.el9 ceph 4.9 M 2024-08-02T23:37:55.774 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-osd x86_64 2:19.1.0-1260.g26c3fb8e.el9 ceph 120 M 2024-08-02T23:37:55.774 INFO:teuthology.orchestra.run.smithi073.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-08-02T23:37:55.774 INFO:teuthology.orchestra.run.smithi073.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-08-02T23:37:55.774 INFO:teuthology.orchestra.run.smithi073.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-08-02T23:37:55.774 INFO:teuthology.orchestra.run.smithi073.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-08-02T23:37:55.774 INFO:teuthology.orchestra.run.smithi073.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-08-02T23:37:55.774 INFO:teuthology.orchestra.run.smithi073.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-08-02T23:37:55.774 INFO:teuthology.orchestra.run.smithi073.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-08-02T23:37:55.774 INFO:teuthology.orchestra.run.smithi073.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-08-02T23:37:55.774 INFO:teuthology.orchestra.run.smithi073.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-08-02T23:37:55.774 INFO:teuthology.orchestra.run.smithi073.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-08-02T23:37:55.775 INFO:teuthology.orchestra.run.smithi073.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-08-02T23:37:55.775 INFO:teuthology.orchestra.run.smithi073.stdout: python3-jaraco-collections 2024-08-02T23:37:55.775 INFO:teuthology.orchestra.run.smithi073.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-08-02T23:37:55.775 INFO:teuthology.orchestra.run.smithi073.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-08-02T23:37:55.775 INFO:teuthology.orchestra.run.smithi073.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-08-02T23:37:55.775 INFO:teuthology.orchestra.run.smithi073.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-08-02T23:37:55.775 INFO:teuthology.orchestra.run.smithi073.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-08-02T23:37:55.775 INFO:teuthology.orchestra.run.smithi073.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-08-02T23:37:55.775 INFO:teuthology.orchestra.run.smithi073.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-08-02T23:37:55.775 INFO:teuthology.orchestra.run.smithi073.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-08-02T23:37:55.775 INFO:teuthology.orchestra.run.smithi073.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-08-02T23:37:55.775 INFO:teuthology.orchestra.run.smithi073.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-08-02T23:37:55.775 INFO:teuthology.orchestra.run.smithi073.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-08-02T23:37:55.775 INFO:teuthology.orchestra.run.smithi073.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-08-02T23:37:55.775 INFO:teuthology.orchestra.run.smithi073.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-08-02T23:37:55.776 INFO:teuthology.orchestra.run.smithi073.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-08-02T23:37:55.776 INFO:teuthology.orchestra.run.smithi073.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-08-02T23:37:55.776 INFO:teuthology.orchestra.run.smithi073.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-08-02T23:37:55.776 INFO:teuthology.orchestra.run.smithi073.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-08-02T23:37:55.776 INFO:teuthology.orchestra.run.smithi073.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-08-02T23:37:55.776 INFO:teuthology.orchestra.run.smithi073.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-08-02T23:37:55.776 INFO:teuthology.orchestra.run.smithi073.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-08-02T23:37:55.776 INFO:teuthology.orchestra.run.smithi073.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-08-02T23:37:55.776 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-08-02T23:37:55.776 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction Summary 2024-08-02T23:37:55.776 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2024-08-02T23:37:55.776 INFO:teuthology.orchestra.run.smithi073.stdout:Install 38 Packages 2024-08-02T23:37:55.776 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-08-02T23:37:55.777 INFO:teuthology.orchestra.run.smithi073.stdout:Total download size: 134 M 2024-08-02T23:37:55.777 INFO:teuthology.orchestra.run.smithi073.stdout:Installed size: 1.4 G 2024-08-02T23:37:55.777 INFO:teuthology.orchestra.run.smithi073.stdout:Downloading Packages: 2024-08-02T23:37:55.940 INFO:teuthology.orchestra.run.smithi131.stdout:(2/38): ceph-mds-19.1.0-1260.g26c3fb8e.el9.x86_ 6.4 MB/s | 2.3 MB 00:00 2024-08-02T23:37:55.982 INFO:teuthology.orchestra.run.smithi131.stdout:(3/38): ceph-mgr-19.1.0-1260.g26c3fb8e.el9.x86_ 3.8 MB/s | 1.5 MB 00:00 2024-08-02T23:37:56.074 INFO:teuthology.orchestra.run.smithi131.stdout:(4/38): ceph-mgr-modules-core-19.1.0-1260.g26c3 2.7 MB/s | 249 kB 00:00 2024-08-02T23:37:56.208 INFO:teuthology.orchestra.run.smithi131.stdout:(5/38): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.9 MB/s | 253 kB 00:00 2024-08-02T23:37:56.292 INFO:teuthology.orchestra.run.smithi131.stdout:(6/38): python3-cryptography-36.0.1-4.el9.x86_6 15 MB/s | 1.2 MB 00:00 2024-08-02T23:37:56.333 INFO:teuthology.orchestra.run.smithi131.stdout:(7/38): python3-ply-3.11-14.el9.noarch.rpm 2.5 MB/s | 106 kB 00:00 2024-08-02T23:37:56.417 INFO:teuthology.orchestra.run.smithi131.stdout:(8/38): ceph-mon-19.1.0-1260.g26c3fb8e.el9.x86_ 7.3 MB/s | 4.9 MB 00:00 2024-08-02T23:37:56.442 INFO:teuthology.orchestra.run.smithi131.stdout:(9/38): python3-pycparser-2.20-6.el9.noarch.rpm 1.2 MB/s | 135 kB 00:00 2024-08-02T23:37:56.484 INFO:teuthology.orchestra.run.smithi131.stdout:(10/38): python3-urllib3-1.26.5-6.el9.noarch.rp 5.0 MB/s | 215 kB 00:00 2024-08-02T23:37:56.534 INFO:teuthology.orchestra.run.smithi131.stdout:(11/38): python3-requests-2.25.1-8.el9.noarch.r 1.0 MB/s | 125 kB 00:00 2024-08-02T23:37:56.610 INFO:teuthology.orchestra.run.smithi131.stdout:(12/38): lua-5.4.4-4.el9.x86_64.rpm 1.5 MB/s | 188 kB 00:00 2024-08-02T23:37:56.668 INFO:teuthology.orchestra.run.smithi131.stdout:(13/38): python3-mako-1.1.4-6.el9.noarch.rpm 2.9 MB/s | 172 kB 00:00 2024-08-02T23:37:56.702 INFO:teuthology.orchestra.run.smithi131.stdout:(14/38): protobuf-3.14.0-13.el9.x86_64.rpm 6.0 MB/s | 1.0 MB 00:00 2024-08-02T23:37:56.727 INFO:teuthology.orchestra.run.smithi131.stdout:(15/38): python3-markupsafe-1.1.1-12.el9.x86_64 593 kB/s | 35 kB 00:00 2024-08-02T23:37:56.752 INFO:teuthology.orchestra.run.smithi131.stdout:(16/38): python3-packaging-20.9-5.el9.noarch.rp 1.5 MB/s | 77 kB 00:00 2024-08-02T23:37:56.778 INFO:teuthology.orchestra.run.smithi131.stdout:(17/38): python3-pytz-2021.1-5.el9.noarch.rpm 1.0 MB/s | 51 kB 00:00 2024-08-02T23:37:56.803 INFO:teuthology.orchestra.run.smithi131.stdout:(18/38): python3-toml-0.10.2-6.el9.noarch.rpm 826 kB/s | 42 kB 00:00 2024-08-02T23:37:56.862 INFO:teuthology.orchestra.run.smithi131.stdout:(19/38): luarocks-3.9.2-5.el9.noarch.rpm 2.5 MB/s | 151 kB 00:00 2024-08-02T23:37:56.887 INFO:teuthology.orchestra.run.smithi131.stdout:(20/38): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 1.7 MB/s | 43 kB 00:00 2024-08-02T23:37:56.912 INFO:teuthology.orchestra.run.smithi131.stdout:(21/38): python3-cheroot-8.6.0-4.el9.noarch.rpm 6.7 MB/s | 172 kB 00:00 2024-08-02T23:37:56.946 INFO:teuthology.orchestra.run.smithi131.stdout:(22/38): python3-cherrypy-18.6.1-2.el9.noarch.r 11 MB/s | 358 kB 00:00 2024-08-02T23:37:56.952 INFO:teuthology.orchestra.run.smithi073.stdout:(1/38): ceph-19.1.0-1260.g26c3fb8e.el9.x86_64.r 34 kB/s | 6.6 kB 00:00 2024-08-02T23:37:56.971 INFO:teuthology.orchestra.run.smithi131.stdout:(23/38): python3-jaraco-8.2.1-3.el9.noarch.rpm 426 kB/s | 11 kB 00:00 2024-08-02T23:37:56.996 INFO:teuthology.orchestra.run.smithi131.stdout:(24/38): python3-jaraco-classes-3.2.1-5.el9.noa 707 kB/s | 18 kB 00:00 2024-08-02T23:37:57.021 INFO:teuthology.orchestra.run.smithi131.stdout:(25/38): python3-jaraco-collections-3.0.0-8.el9 930 kB/s | 23 kB 00:00 2024-08-02T23:37:57.047 INFO:teuthology.orchestra.run.smithi131.stdout:(26/38): python3-jaraco-functools-3.5.0-2.el9.n 773 kB/s | 19 kB 00:00 2024-08-02T23:37:57.097 INFO:teuthology.orchestra.run.smithi131.stdout:(27/38): python3-jaraco-text-3.2.0-6.el9.noarch 752 kB/s | 20 kB 00:00 2024-08-02T23:37:57.103 INFO:teuthology.orchestra.run.smithi073.stdout:(2/38): ceph-mgr-19.1.0-1260.g26c3fb8e.el9.x86_ 4.4 MB/s | 1.5 MB 00:00 2024-08-02T23:37:57.122 INFO:teuthology.orchestra.run.smithi131.stdout:(28/38): python3-logutils-0.3.5-21.el9.noarch.r 1.8 MB/s | 46 kB 00:00 2024-08-02T23:37:57.148 INFO:teuthology.orchestra.run.smithi131.stdout:(29/38): python3-more-itertools-8.12.0-2.el9.no 3.1 MB/s | 79 kB 00:00 2024-08-02T23:37:57.153 INFO:teuthology.orchestra.run.smithi073.stdout:(3/38): ceph-mds-19.1.0-1260.g26c3fb8e.el9.x86_ 5.8 MB/s | 2.3 MB 00:00 2024-08-02T23:37:57.190 INFO:teuthology.orchestra.run.smithi131.stdout:(30/38): python3-pecan-1.4.2-3.el9.noarch.rpm 6.5 MB/s | 272 kB 00:00 2024-08-02T23:37:57.215 INFO:teuthology.orchestra.run.smithi131.stdout:(31/38): python3-portend-3.1.0-2.el9.noarch.rpm 662 kB/s | 16 kB 00:00 2024-08-02T23:37:57.220 INFO:teuthology.orchestra.run.smithi073.stdout:(4/38): ceph-mgr-modules-core-19.1.0-1260.g26c3 3.7 MB/s | 249 kB 00:00 2024-08-02T23:37:57.240 INFO:teuthology.orchestra.run.smithi131.stdout:(32/38): python3-pyOpenSSL-21.0.0-1.el9.noarch. 3.5 MB/s | 90 kB 00:00 2024-08-02T23:37:57.257 INFO:teuthology.orchestra.run.smithi131.stdout:(33/38): python3-tempora-5.0.0-2.el9.noarch.rpm 2.1 MB/s | 36 kB 00:00 2024-08-02T23:37:57.291 INFO:teuthology.orchestra.run.smithi131.stdout:(34/38): python3-webob-1.8.7-6.el9.noarch.rpm 6.7 MB/s | 230 kB 00:00 2024-08-02T23:37:57.324 INFO:teuthology.orchestra.run.smithi131.stdout:(35/38): python3-werkzeug-2.0.3-3.el9.1.noarch. 13 MB/s | 427 kB 00:00 2024-08-02T23:37:57.341 INFO:teuthology.orchestra.run.smithi131.stdout:(36/38): python3-zc-lockfile-2.0-10.el9.noarch. 1.2 MB/s | 20 kB 00:00 2024-08-02T23:37:57.400 INFO:teuthology.orchestra.run.smithi131.stdout:(37/38): lua-devel-5.4.4-4.el9.x86_64.rpm 36 kB/s | 22 kB 00:00 2024-08-02T23:37:57.453 INFO:teuthology.orchestra.run.smithi073.stdout:(5/38): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.1 MB/s | 253 kB 00:00 2024-08-02T23:37:57.587 INFO:teuthology.orchestra.run.smithi073.stdout:(6/38): python3-cryptography-36.0.1-4.el9.x86_6 9.2 MB/s | 1.2 MB 00:00 2024-08-02T23:37:57.646 INFO:teuthology.orchestra.run.smithi073.stdout:(7/38): python3-ply-3.11-14.el9.noarch.rpm 1.8 MB/s | 106 kB 00:00 2024-08-02T23:37:57.713 INFO:teuthology.orchestra.run.smithi073.stdout:(8/38): python3-pycparser-2.20-6.el9.noarch.rpm 2.0 MB/s | 135 kB 00:00 2024-08-02T23:37:57.771 INFO:teuthology.orchestra.run.smithi073.stdout:(9/38): python3-requests-2.25.1-8.el9.noarch.rp 2.1 MB/s | 125 kB 00:00 2024-08-02T23:37:57.830 INFO:teuthology.orchestra.run.smithi073.stdout:(10/38): python3-urllib3-1.26.5-6.el9.noarch.rp 3.6 MB/s | 215 kB 00:00 2024-08-02T23:37:57.889 INFO:teuthology.orchestra.run.smithi073.stdout:(11/38): lua-5.4.4-4.el9.x86_64.rpm 3.2 MB/s | 188 kB 00:00 2024-08-02T23:37:57.964 INFO:teuthology.orchestra.run.smithi073.stdout:(12/38): protobuf-3.14.0-13.el9.x86_64.rpm 13 MB/s | 1.0 MB 00:00 2024-08-02T23:37:58.039 INFO:teuthology.orchestra.run.smithi073.stdout:(13/38): ceph-mon-19.1.0-1260.g26c3fb8e.el9.x86 4.5 MB/s | 4.9 MB 00:01 2024-08-02T23:37:58.064 INFO:teuthology.orchestra.run.smithi073.stdout:(14/38): python3-mako-1.1.4-6.el9.noarch.rpm 1.7 MB/s | 172 kB 00:00 2024-08-02T23:37:58.123 INFO:teuthology.orchestra.run.smithi073.stdout:(15/38): python3-packaging-20.9-5.el9.noarch.rp 1.3 MB/s | 77 kB 00:00 2024-08-02T23:37:58.149 INFO:teuthology.orchestra.run.smithi073.stdout:(16/38): python3-markupsafe-1.1.1-12.el9.x86_64 319 kB/s | 35 kB 00:00 2024-08-02T23:37:58.165 INFO:teuthology.orchestra.run.smithi073.stdout:(17/38): python3-pytz-2021.1-5.el9.noarch.rpm 1.2 MB/s | 51 kB 00:00 2024-08-02T23:37:58.257 INFO:teuthology.orchestra.run.smithi073.stdout:(18/38): lua-devel-5.4.4-4.el9.x86_64.rpm 244 kB/s | 22 kB 00:00 2024-08-02T23:37:58.282 INFO:teuthology.orchestra.run.smithi073.stdout:(19/38): python3-toml-0.10.2-6.el9.noarch.rpm 310 kB/s | 42 kB 00:00 2024-08-02T23:37:58.366 INFO:teuthology.orchestra.run.smithi073.stdout:(20/38): luarocks-3.9.2-5.el9.noarch.rpm 1.4 MB/s | 151 kB 00:00 2024-08-02T23:37:58.383 INFO:teuthology.orchestra.run.smithi073.stdout:(21/38): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 433 kB/s | 43 kB 00:00 2024-08-02T23:37:58.408 INFO:teuthology.orchestra.run.smithi073.stdout:(22/38): python3-cheroot-8.6.0-4.el9.noarch.rpm 4.2 MB/s | 172 kB 00:00 2024-08-02T23:37:58.425 INFO:teuthology.orchestra.run.smithi073.stdout:(23/38): python3-jaraco-8.2.1-3.el9.noarch.rpm 644 kB/s | 11 kB 00:00 2024-08-02T23:37:58.442 INFO:teuthology.orchestra.run.smithi073.stdout:(24/38): python3-jaraco-classes-3.2.1-5.el9.noa 1.0 MB/s | 18 kB 00:00 2024-08-02T23:37:58.459 INFO:teuthology.orchestra.run.smithi073.stdout:(25/38): python3-jaraco-collections-3.0.0-8.el9 1.4 MB/s | 23 kB 00:00 2024-08-02T23:37:58.476 INFO:teuthology.orchestra.run.smithi073.stdout:(26/38): python3-jaraco-functools-3.5.0-2.el9.n 1.1 MB/s | 19 kB 00:00 2024-08-02T23:37:58.501 INFO:teuthology.orchestra.run.smithi073.stdout:(27/38): python3-cherrypy-18.6.1-2.el9.noarch.r 3.0 MB/s | 358 kB 00:00 2024-08-02T23:37:58.518 INFO:teuthology.orchestra.run.smithi073.stdout:(28/38): python3-jaraco-text-3.2.0-6.el9.noarch 468 kB/s | 20 kB 00:00 2024-08-02T23:37:58.536 INFO:teuthology.orchestra.run.smithi073.stdout:(29/38): python3-logutils-0.3.5-21.el9.noarch.r 1.3 MB/s | 46 kB 00:00 2024-08-02T23:37:58.561 INFO:teuthology.orchestra.run.smithi073.stdout:(30/38): python3-more-itertools-8.12.0-2.el9.no 1.8 MB/s | 79 kB 00:00 2024-08-02T23:37:58.586 INFO:teuthology.orchestra.run.smithi073.stdout:(31/38): python3-portend-3.1.0-2.el9.noarch.rpm 657 kB/s | 16 kB 00:00 2024-08-02T23:37:58.611 INFO:teuthology.orchestra.run.smithi073.stdout:(32/38): python3-pecan-1.4.2-3.el9.noarch.rpm 3.5 MB/s | 272 kB 00:00 2024-08-02T23:37:58.637 INFO:teuthology.orchestra.run.smithi073.stdout:(33/38): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.8 MB/s | 90 kB 00:00 2024-08-02T23:37:58.662 INFO:teuthology.orchestra.run.smithi073.stdout:(34/38): python3-tempora-5.0.0-2.el9.noarch.rpm 1.4 MB/s | 36 kB 00:00 2024-08-02T23:37:58.687 INFO:teuthology.orchestra.run.smithi073.stdout:(35/38): python3-webob-1.8.7-6.el9.noarch.rpm 4.5 MB/s | 230 kB 00:00 2024-08-02T23:37:58.712 INFO:teuthology.orchestra.run.smithi073.stdout:(36/38): python3-zc-lockfile-2.0-10.el9.noarch. 797 kB/s | 20 kB 00:00 2024-08-02T23:37:58.737 INFO:teuthology.orchestra.run.smithi073.stdout:(37/38): python3-werkzeug-2.0.3-3.el9.1.noarch. 5.5 MB/s | 427 kB 00:00 2024-08-02T23:38:00.400 INFO:teuthology.orchestra.run.smithi131.stdout:(38/38): ceph-osd-19.1.0-1260.g26c3fb8e.el9.x86 27 MB/s | 120 MB 00:04 2024-08-02T23:38:00.404 INFO:teuthology.orchestra.run.smithi131.stdout:-------------------------------------------------------------------------------- 2024-08-02T23:38:00.404 INFO:teuthology.orchestra.run.smithi131.stdout:Total 23 MB/s | 134 MB 00:05 2024-08-02T23:38:00.576 INFO:teuthology.orchestra.run.smithi131.stdout:Running transaction check 2024-08-02T23:38:00.616 INFO:teuthology.orchestra.run.smithi131.stdout:Transaction check succeeded. 2024-08-02T23:38:00.617 INFO:teuthology.orchestra.run.smithi131.stdout:Running transaction test 2024-08-02T23:38:01.654 INFO:teuthology.orchestra.run.smithi073.stdout:(38/38): ceph-osd-19.1.0-1260.g26c3fb8e.el9.x86 26 MB/s | 120 MB 00:04 2024-08-02T23:38:01.659 INFO:teuthology.orchestra.run.smithi073.stdout:-------------------------------------------------------------------------------- 2024-08-02T23:38:01.659 INFO:teuthology.orchestra.run.smithi073.stdout:Total 23 MB/s | 134 MB 00:05 2024-08-02T23:38:01.691 INFO:teuthology.orchestra.run.smithi131.stdout:Transaction test succeeded. 2024-08-02T23:38:01.691 INFO:teuthology.orchestra.run.smithi131.stdout:Running transaction 2024-08-02T23:38:01.826 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction check 2024-08-02T23:38:01.867 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction check succeeded. 2024-08-02T23:38:01.867 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction test 2024-08-02T23:38:02.910 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction test succeeded. 2024-08-02T23:38:02.910 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction 2024-08-02T23:38:03.732 INFO:teuthology.orchestra.run.smithi131.stdout: Preparing : 1/1 2024-08-02T23:38:03.803 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/38 2024-08-02T23:38:03.835 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/38 2024-08-02T23:38:03.872 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/38 2024-08-02T23:38:03.946 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/38 2024-08-02T23:38:04.004 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/38 2024-08-02T23:38:04.045 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/38 2024-08-02T23:38:04.136 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/38 2024-08-02T23:38:04.216 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/38 2024-08-02T23:38:04.255 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/38 2024-08-02T23:38:04.291 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/38 2024-08-02T23:38:04.321 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/38 2024-08-02T23:38:04.347 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/38 2024-08-02T23:38:04.380 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/38 2024-08-02T23:38:04.421 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/38 2024-08-02T23:38:04.455 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/38 2024-08-02T23:38:04.490 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/38 2024-08-02T23:38:04.538 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/38 2024-08-02T23:38:04.570 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-packaging-20.9-5.el9.noarch 18/38 2024-08-02T23:38:04.645 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 19/38 2024-08-02T23:38:04.791 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 20/38 2024-08-02T23:38:04.843 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 21/38 2024-08-02T23:38:04.924 INFO:teuthology.orchestra.run.smithi073.stdout: Preparing : 1/1 2024-08-02T23:38:04.999 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/38 2024-08-02T23:38:05.031 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/38 2024-08-02T23:38:05.059 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/38 2024-08-02T23:38:05.130 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/38 2024-08-02T23:38:05.175 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/38 2024-08-02T23:38:05.208 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/38 2024-08-02T23:38:05.301 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/38 2024-08-02T23:38:05.371 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/38 2024-08-02T23:38:05.419 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/38 2024-08-02T23:38:05.454 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/38 2024-08-02T23:38:05.484 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/38 2024-08-02T23:38:05.519 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/38 2024-08-02T23:38:05.552 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/38 2024-08-02T23:38:05.592 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/38 2024-08-02T23:38:05.627 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/38 2024-08-02T23:38:05.662 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/38 2024-08-02T23:38:05.711 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/38 2024-08-02T23:38:05.742 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-packaging-20.9-5.el9.noarch 18/38 2024-08-02T23:38:05.824 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 19/38 2024-08-02T23:38:05.962 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 20/38 2024-08-02T23:38:06.015 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 21/38 2024-08-02T23:38:23.563 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 22/38 2024-08-02T23:38:23.681 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : ceph-osd-2:19.1.0-1260.g26c3fb8e.el9.x86_64 23/38 2024-08-02T23:38:23.710 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: ceph-osd-2:19.1.0-1260.g26c3fb8e.el9.x86_64 23/38 2024-08-02T23:38:23.710 INFO:teuthology.orchestra.run.smithi131.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-02T23:38:23.710 INFO:teuthology.orchestra.run.smithi131.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-08-02T23:38:23.710 INFO:teuthology.orchestra.run.smithi131.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-08-02T23:38:23.710 INFO:teuthology.orchestra.run.smithi131.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-08-02T23:38:23.710 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-08-02T23:38:23.813 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 24/38 2024-08-02T23:38:23.852 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 25/38 2024-08-02T23:38:23.901 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-ply-3.11-14.el9.noarch 26/38 2024-08-02T23:38:23.964 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 27/38 2024-08-02T23:38:24.151 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 28/38 2024-08-02T23:38:24.198 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 29/38 2024-08-02T23:38:24.276 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 30/38 2024-08-02T23:38:24.375 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 31/38 2024-08-02T23:38:24.455 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 32/38 2024-08-02T23:38:24.573 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 33/38 2024-08-02T23:38:24.735 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : ceph-mgr-modules-core-2:19.1.0-1260.g26c3fb8e.el9. 34/38 2024-08-02T23:38:24.758 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 22/38 2024-08-02T23:38:24.759 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : ceph-mgr-2:19.1.0-1260.g26c3fb8e.el9.x86_64 35/38 2024-08-02T23:38:24.784 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: ceph-mgr-2:19.1.0-1260.g26c3fb8e.el9.x86_64 35/38 2024-08-02T23:38:24.785 INFO:teuthology.orchestra.run.smithi131.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-02T23:38:24.785 INFO:teuthology.orchestra.run.smithi131.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-08-02T23:38:24.785 INFO:teuthology.orchestra.run.smithi131.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-08-02T23:38:24.785 INFO:teuthology.orchestra.run.smithi131.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-08-02T23:38:24.785 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-08-02T23:38:24.794 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : ceph-osd-2:19.1.0-1260.g26c3fb8e.el9.x86_64 23/38 2024-08-02T23:38:24.823 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: ceph-osd-2:19.1.0-1260.g26c3fb8e.el9.x86_64 23/38 2024-08-02T23:38:24.824 INFO:teuthology.orchestra.run.smithi073.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-02T23:38:24.824 INFO:teuthology.orchestra.run.smithi073.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-08-02T23:38:24.824 INFO:teuthology.orchestra.run.smithi073.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-08-02T23:38:24.824 INFO:teuthology.orchestra.run.smithi073.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-08-02T23:38:24.824 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-08-02T23:38:24.919 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 24/38 2024-08-02T23:38:24.966 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 25/38 2024-08-02T23:38:25.015 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-ply-3.11-14.el9.noarch 26/38 2024-08-02T23:38:25.071 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 27/38 2024-08-02T23:38:25.260 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 28/38 2024-08-02T23:38:25.301 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : ceph-mon-2:19.1.0-1260.g26c3fb8e.el9.x86_64 36/38 2024-08-02T23:38:25.305 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 29/38 2024-08-02T23:38:25.325 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: ceph-mon-2:19.1.0-1260.g26c3fb8e.el9.x86_64 36/38 2024-08-02T23:38:25.325 INFO:teuthology.orchestra.run.smithi131.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-02T23:38:25.325 INFO:teuthology.orchestra.run.smithi131.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-08-02T23:38:25.325 INFO:teuthology.orchestra.run.smithi131.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-08-02T23:38:25.325 INFO:teuthology.orchestra.run.smithi131.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-08-02T23:38:25.326 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-08-02T23:38:25.384 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 30/38 2024-08-02T23:38:25.482 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 31/38 2024-08-02T23:38:25.524 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 32/38 2024-08-02T23:38:25.552 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : ceph-mds-2:19.1.0-1260.g26c3fb8e.el9.x86_64 37/38 2024-08-02T23:38:25.575 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: ceph-mds-2:19.1.0-1260.g26c3fb8e.el9.x86_64 37/38 2024-08-02T23:38:25.575 INFO:teuthology.orchestra.run.smithi131.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-02T23:38:25.575 INFO:teuthology.orchestra.run.smithi131.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-08-02T23:38:25.576 INFO:teuthology.orchestra.run.smithi131.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-08-02T23:38:25.576 INFO:teuthology.orchestra.run.smithi131.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-08-02T23:38:25.576 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-08-02T23:38:25.640 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : ceph-2:19.1.0-1260.g26c3fb8e.el9.x86_64 38/38 2024-08-02T23:38:25.649 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 33/38 2024-08-02T23:38:25.828 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : ceph-mgr-modules-core-2:19.1.0-1260.g26c3fb8e.el9. 34/38 2024-08-02T23:38:25.856 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : ceph-mgr-2:19.1.0-1260.g26c3fb8e.el9.x86_64 35/38 2024-08-02T23:38:25.888 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: ceph-mgr-2:19.1.0-1260.g26c3fb8e.el9.x86_64 35/38 2024-08-02T23:38:25.888 INFO:teuthology.orchestra.run.smithi073.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-02T23:38:25.888 INFO:teuthology.orchestra.run.smithi073.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-08-02T23:38:25.889 INFO:teuthology.orchestra.run.smithi073.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-08-02T23:38:25.889 INFO:teuthology.orchestra.run.smithi073.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-08-02T23:38:25.889 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-08-02T23:38:26.382 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : ceph-mon-2:19.1.0-1260.g26c3fb8e.el9.x86_64 36/38 2024-08-02T23:38:26.405 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: ceph-mon-2:19.1.0-1260.g26c3fb8e.el9.x86_64 36/38 2024-08-02T23:38:26.405 INFO:teuthology.orchestra.run.smithi073.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-02T23:38:26.405 INFO:teuthology.orchestra.run.smithi073.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-08-02T23:38:26.405 INFO:teuthology.orchestra.run.smithi073.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-08-02T23:38:26.406 INFO:teuthology.orchestra.run.smithi073.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-08-02T23:38:26.406 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-08-02T23:38:26.633 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : ceph-mds-2:19.1.0-1260.g26c3fb8e.el9.x86_64 37/38 2024-08-02T23:38:26.656 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: ceph-mds-2:19.1.0-1260.g26c3fb8e.el9.x86_64 37/38 2024-08-02T23:38:26.656 INFO:teuthology.orchestra.run.smithi073.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-02T23:38:26.656 INFO:teuthology.orchestra.run.smithi073.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-08-02T23:38:26.656 INFO:teuthology.orchestra.run.smithi073.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-08-02T23:38:26.657 INFO:teuthology.orchestra.run.smithi073.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-08-02T23:38:26.657 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-08-02T23:38:26.723 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : ceph-2:19.1.0-1260.g26c3fb8e.el9.x86_64 38/38 2024-08-02T23:38:36.701 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: ceph-2:19.1.0-1260.g26c3fb8e.el9.x86_64 38/38 2024-08-02T23:38:36.701 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : ceph-2:19.1.0-1260.g26c3fb8e.el9.x86_64 1/38 2024-08-02T23:38:36.702 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : ceph-mds-2:19.1.0-1260.g26c3fb8e.el9.x86_64 2/38 2024-08-02T23:38:36.702 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : ceph-mgr-2:19.1.0-1260.g26c3fb8e.el9.x86_64 3/38 2024-08-02T23:38:36.702 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : ceph-mon-2:19.1.0-1260.g26c3fb8e.el9.x86_64 4/38 2024-08-02T23:38:36.702 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : ceph-osd-2:19.1.0-1260.g26c3fb8e.el9.x86_64 5/38 2024-08-02T23:38:36.702 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : ceph-mgr-modules-core-2:19.1.0-1260.g26c3fb8e.el9. 6/38 2024-08-02T23:38:36.702 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/38 2024-08-02T23:38:36.702 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/38 2024-08-02T23:38:36.702 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/38 2024-08-02T23:38:36.702 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/38 2024-08-02T23:38:36.702 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/38 2024-08-02T23:38:36.702 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/38 2024-08-02T23:38:36.702 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/38 2024-08-02T23:38:36.703 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 14/38 2024-08-02T23:38:36.703 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 15/38 2024-08-02T23:38:36.703 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 16/38 2024-08-02T23:38:36.703 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 17/38 2024-08-02T23:38:36.703 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 18/38 2024-08-02T23:38:36.703 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 19/38 2024-08-02T23:38:36.703 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 20/38 2024-08-02T23:38:36.703 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 21/38 2024-08-02T23:38:36.703 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 22/38 2024-08-02T23:38:36.703 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 23/38 2024-08-02T23:38:36.703 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 24/38 2024-08-02T23:38:36.703 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 25/38 2024-08-02T23:38:36.703 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 26/38 2024-08-02T23:38:36.704 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 27/38 2024-08-02T23:38:36.704 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 28/38 2024-08-02T23:38:36.705 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 29/38 2024-08-02T23:38:36.705 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 30/38 2024-08-02T23:38:36.705 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 31/38 2024-08-02T23:38:36.705 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 32/38 2024-08-02T23:38:36.705 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 33/38 2024-08-02T23:38:36.705 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 34/38 2024-08-02T23:38:36.705 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 35/38 2024-08-02T23:38:36.705 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 36/38 2024-08-02T23:38:36.705 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 37/38 2024-08-02T23:38:37.681 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 38/38 2024-08-02T23:38:37.681 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-08-02T23:38:37.681 INFO:teuthology.orchestra.run.smithi131.stdout:Installed: 2024-08-02T23:38:37.681 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-2:19.1.0-1260.g26c3fb8e.el9.x86_64 2024-08-02T23:38:37.681 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-mds-2:19.1.0-1260.g26c3fb8e.el9.x86_64 2024-08-02T23:38:37.681 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-mgr-2:19.1.0-1260.g26c3fb8e.el9.x86_64 2024-08-02T23:38:37.681 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-mgr-modules-core-2:19.1.0-1260.g26c3fb8e.el9.noarch 2024-08-02T23:38:37.681 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-mon-2:19.1.0-1260.g26c3fb8e.el9.x86_64 2024-08-02T23:38:37.681 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-osd-2:19.1.0-1260.g26c3fb8e.el9.x86_64 2024-08-02T23:38:37.681 INFO:teuthology.orchestra.run.smithi131.stdout: lua-5.4.4-4.el9.x86_64 2024-08-02T23:38:37.682 INFO:teuthology.orchestra.run.smithi131.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-08-02T23:38:37.682 INFO:teuthology.orchestra.run.smithi131.stdout: luarocks-3.9.2-5.el9.noarch 2024-08-02T23:38:37.682 INFO:teuthology.orchestra.run.smithi131.stdout: protobuf-3.14.0-13.el9.x86_64 2024-08-02T23:38:37.682 INFO:teuthology.orchestra.run.smithi131.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-08-02T23:38:37.682 INFO:teuthology.orchestra.run.smithi131.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-08-02T23:38:37.682 INFO:teuthology.orchestra.run.smithi131.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-08-02T23:38:37.682 INFO:teuthology.orchestra.run.smithi131.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-08-02T23:38:37.682 INFO:teuthology.orchestra.run.smithi131.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-08-02T23:38:37.682 INFO:teuthology.orchestra.run.smithi131.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-08-02T23:38:37.682 INFO:teuthology.orchestra.run.smithi131.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-08-02T23:38:37.682 INFO:teuthology.orchestra.run.smithi131.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-08-02T23:38:37.682 INFO:teuthology.orchestra.run.smithi131.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-08-02T23:38:37.683 INFO:teuthology.orchestra.run.smithi131.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-08-02T23:38:37.683 INFO:teuthology.orchestra.run.smithi131.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-08-02T23:38:37.683 INFO:teuthology.orchestra.run.smithi131.stdout: python3-mako-1.1.4-6.el9.noarch 2024-08-02T23:38:37.683 INFO:teuthology.orchestra.run.smithi131.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-08-02T23:38:37.683 INFO:teuthology.orchestra.run.smithi131.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-08-02T23:38:37.683 INFO:teuthology.orchestra.run.smithi131.stdout: python3-packaging-20.9-5.el9.noarch 2024-08-02T23:38:37.683 INFO:teuthology.orchestra.run.smithi131.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-08-02T23:38:37.683 INFO:teuthology.orchestra.run.smithi131.stdout: python3-ply-3.11-14.el9.noarch 2024-08-02T23:38:37.683 INFO:teuthology.orchestra.run.smithi131.stdout: python3-portend-3.1.0-2.el9.noarch 2024-08-02T23:38:37.683 INFO:teuthology.orchestra.run.smithi131.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-08-02T23:38:37.683 INFO:teuthology.orchestra.run.smithi131.stdout: python3-pycparser-2.20-6.el9.noarch 2024-08-02T23:38:37.683 INFO:teuthology.orchestra.run.smithi131.stdout: python3-pytz-2021.1-5.el9.noarch 2024-08-02T23:38:37.683 INFO:teuthology.orchestra.run.smithi131.stdout: python3-requests-2.25.1-8.el9.noarch 2024-08-02T23:38:37.683 INFO:teuthology.orchestra.run.smithi131.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-08-02T23:38:37.684 INFO:teuthology.orchestra.run.smithi131.stdout: python3-toml-0.10.2-6.el9.noarch 2024-08-02T23:38:37.684 INFO:teuthology.orchestra.run.smithi131.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-08-02T23:38:37.684 INFO:teuthology.orchestra.run.smithi131.stdout: python3-webob-1.8.7-6.el9.noarch 2024-08-02T23:38:37.684 INFO:teuthology.orchestra.run.smithi131.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-08-02T23:38:37.684 INFO:teuthology.orchestra.run.smithi131.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-08-02T23:38:37.684 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-08-02T23:38:37.684 INFO:teuthology.orchestra.run.smithi131.stdout:Complete! 2024-08-02T23:38:37.687 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: ceph-2:19.1.0-1260.g26c3fb8e.el9.x86_64 38/38 2024-08-02T23:38:37.687 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : ceph-2:19.1.0-1260.g26c3fb8e.el9.x86_64 1/38 2024-08-02T23:38:37.687 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : ceph-mds-2:19.1.0-1260.g26c3fb8e.el9.x86_64 2/38 2024-08-02T23:38:37.687 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : ceph-mgr-2:19.1.0-1260.g26c3fb8e.el9.x86_64 3/38 2024-08-02T23:38:37.687 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : ceph-mon-2:19.1.0-1260.g26c3fb8e.el9.x86_64 4/38 2024-08-02T23:38:37.687 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : ceph-osd-2:19.1.0-1260.g26c3fb8e.el9.x86_64 5/38 2024-08-02T23:38:37.687 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : ceph-mgr-modules-core-2:19.1.0-1260.g26c3fb8e.el9. 6/38 2024-08-02T23:38:37.688 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/38 2024-08-02T23:38:37.688 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/38 2024-08-02T23:38:37.688 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/38 2024-08-02T23:38:37.688 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/38 2024-08-02T23:38:37.688 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/38 2024-08-02T23:38:37.688 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/38 2024-08-02T23:38:37.688 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/38 2024-08-02T23:38:37.688 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 14/38 2024-08-02T23:38:37.688 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 15/38 2024-08-02T23:38:37.688 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 16/38 2024-08-02T23:38:37.688 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 17/38 2024-08-02T23:38:37.688 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 18/38 2024-08-02T23:38:37.688 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 19/38 2024-08-02T23:38:37.689 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 20/38 2024-08-02T23:38:37.689 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 21/38 2024-08-02T23:38:37.689 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 22/38 2024-08-02T23:38:37.689 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 23/38 2024-08-02T23:38:37.689 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 24/38 2024-08-02T23:38:37.689 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 25/38 2024-08-02T23:38:37.689 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 26/38 2024-08-02T23:38:37.689 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 27/38 2024-08-02T23:38:37.689 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 28/38 2024-08-02T23:38:37.689 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 29/38 2024-08-02T23:38:37.689 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 30/38 2024-08-02T23:38:37.689 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 31/38 2024-08-02T23:38:37.689 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 32/38 2024-08-02T23:38:37.689 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 33/38 2024-08-02T23:38:37.690 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 34/38 2024-08-02T23:38:37.690 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 35/38 2024-08-02T23:38:37.690 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 36/38 2024-08-02T23:38:37.690 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 37/38 2024-08-02T23:38:37.907 DEBUG:teuthology.orchestra.run.smithi131:> sudo yum -y install ceph-base 2024-08-02T23:38:38.457 INFO:teuthology.orchestra.run.smithi131.stdout:Last metadata expiration check: 0:01:46 ago on Fri 02 Aug 2024 11:36:52 PM UTC. 2024-08-02T23:38:38.542 INFO:teuthology.orchestra.run.smithi131.stdout:Package ceph-base-2:19.1.0-1260.g26c3fb8e.el9.x86_64 is already installed. 2024-08-02T23:38:38.583 INFO:teuthology.orchestra.run.smithi131.stdout:Dependencies resolved. 2024-08-02T23:38:38.584 INFO:teuthology.orchestra.run.smithi131.stdout:Nothing to do. 2024-08-02T23:38:38.584 INFO:teuthology.orchestra.run.smithi131.stdout:Complete! 2024-08-02T23:38:38.590 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 38/38 2024-08-02T23:38:38.590 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-08-02T23:38:38.590 INFO:teuthology.orchestra.run.smithi073.stdout:Installed: 2024-08-02T23:38:38.591 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-2:19.1.0-1260.g26c3fb8e.el9.x86_64 2024-08-02T23:38:38.591 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-mds-2:19.1.0-1260.g26c3fb8e.el9.x86_64 2024-08-02T23:38:38.591 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-mgr-2:19.1.0-1260.g26c3fb8e.el9.x86_64 2024-08-02T23:38:38.591 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-mgr-modules-core-2:19.1.0-1260.g26c3fb8e.el9.noarch 2024-08-02T23:38:38.591 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-mon-2:19.1.0-1260.g26c3fb8e.el9.x86_64 2024-08-02T23:38:38.591 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-osd-2:19.1.0-1260.g26c3fb8e.el9.x86_64 2024-08-02T23:38:38.591 INFO:teuthology.orchestra.run.smithi073.stdout: lua-5.4.4-4.el9.x86_64 2024-08-02T23:38:38.591 INFO:teuthology.orchestra.run.smithi073.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-08-02T23:38:38.591 INFO:teuthology.orchestra.run.smithi073.stdout: luarocks-3.9.2-5.el9.noarch 2024-08-02T23:38:38.591 INFO:teuthology.orchestra.run.smithi073.stdout: protobuf-3.14.0-13.el9.x86_64 2024-08-02T23:38:38.591 INFO:teuthology.orchestra.run.smithi073.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-08-02T23:38:38.591 INFO:teuthology.orchestra.run.smithi073.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-08-02T23:38:38.591 INFO:teuthology.orchestra.run.smithi073.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-08-02T23:38:38.592 INFO:teuthology.orchestra.run.smithi073.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-08-02T23:38:38.592 INFO:teuthology.orchestra.run.smithi073.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-08-02T23:38:38.592 INFO:teuthology.orchestra.run.smithi073.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-08-02T23:38:38.592 INFO:teuthology.orchestra.run.smithi073.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-08-02T23:38:38.592 INFO:teuthology.orchestra.run.smithi073.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-08-02T23:38:38.592 INFO:teuthology.orchestra.run.smithi073.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-08-02T23:38:38.592 INFO:teuthology.orchestra.run.smithi073.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-08-02T23:38:38.592 INFO:teuthology.orchestra.run.smithi073.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-08-02T23:38:38.592 INFO:teuthology.orchestra.run.smithi073.stdout: python3-mako-1.1.4-6.el9.noarch 2024-08-02T23:38:38.592 INFO:teuthology.orchestra.run.smithi073.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-08-02T23:38:38.592 INFO:teuthology.orchestra.run.smithi073.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-08-02T23:38:38.592 INFO:teuthology.orchestra.run.smithi073.stdout: python3-packaging-20.9-5.el9.noarch 2024-08-02T23:38:38.592 INFO:teuthology.orchestra.run.smithi073.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-08-02T23:38:38.593 INFO:teuthology.orchestra.run.smithi073.stdout: python3-ply-3.11-14.el9.noarch 2024-08-02T23:38:38.593 INFO:teuthology.orchestra.run.smithi073.stdout: python3-portend-3.1.0-2.el9.noarch 2024-08-02T23:38:38.593 INFO:teuthology.orchestra.run.smithi073.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-08-02T23:38:38.593 INFO:teuthology.orchestra.run.smithi073.stdout: python3-pycparser-2.20-6.el9.noarch 2024-08-02T23:38:38.593 INFO:teuthology.orchestra.run.smithi073.stdout: python3-pytz-2021.1-5.el9.noarch 2024-08-02T23:38:38.593 INFO:teuthology.orchestra.run.smithi073.stdout: python3-requests-2.25.1-8.el9.noarch 2024-08-02T23:38:38.593 INFO:teuthology.orchestra.run.smithi073.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-08-02T23:38:38.593 INFO:teuthology.orchestra.run.smithi073.stdout: python3-toml-0.10.2-6.el9.noarch 2024-08-02T23:38:38.593 INFO:teuthology.orchestra.run.smithi073.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-08-02T23:38:38.593 INFO:teuthology.orchestra.run.smithi073.stdout: python3-webob-1.8.7-6.el9.noarch 2024-08-02T23:38:38.593 INFO:teuthology.orchestra.run.smithi073.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-08-02T23:38:38.593 INFO:teuthology.orchestra.run.smithi073.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-08-02T23:38:38.593 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-08-02T23:38:38.593 INFO:teuthology.orchestra.run.smithi073.stdout:Complete! 2024-08-02T23:38:38.648 DEBUG:teuthology.orchestra.run.smithi131:> sudo yum -y install cephadm 2024-08-02T23:38:38.847 DEBUG:teuthology.orchestra.run.smithi073:> sudo yum -y install ceph-base 2024-08-02T23:38:39.190 INFO:teuthology.orchestra.run.smithi131.stdout:Last metadata expiration check: 0:01:47 ago on Fri 02 Aug 2024 11:36:52 PM UTC. 2024-08-02T23:38:39.315 INFO:teuthology.orchestra.run.smithi131.stdout:Dependencies resolved. 2024-08-02T23:38:39.316 INFO:teuthology.orchestra.run.smithi131.stdout:================================================================================ 2024-08-02T23:38:39.316 INFO:teuthology.orchestra.run.smithi131.stdout: Package Arch Version Repository Size 2024-08-02T23:38:39.316 INFO:teuthology.orchestra.run.smithi131.stdout:================================================================================ 2024-08-02T23:38:39.316 INFO:teuthology.orchestra.run.smithi131.stdout:Installing: 2024-08-02T23:38:39.316 INFO:teuthology.orchestra.run.smithi131.stdout: cephadm noarch 2:19.1.0-1260.g26c3fb8e.el9 ceph-noarch 765 k 2024-08-02T23:38:39.316 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-08-02T23:38:39.316 INFO:teuthology.orchestra.run.smithi131.stdout:Transaction Summary 2024-08-02T23:38:39.317 INFO:teuthology.orchestra.run.smithi131.stdout:================================================================================ 2024-08-02T23:38:39.317 INFO:teuthology.orchestra.run.smithi131.stdout:Install 1 Package 2024-08-02T23:38:39.317 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-08-02T23:38:39.317 INFO:teuthology.orchestra.run.smithi131.stdout:Total download size: 765 k 2024-08-02T23:38:39.317 INFO:teuthology.orchestra.run.smithi131.stdout:Installed size: 771 k 2024-08-02T23:38:39.317 INFO:teuthology.orchestra.run.smithi131.stdout:Downloading Packages: 2024-08-02T23:38:39.394 INFO:teuthology.orchestra.run.smithi073.stdout:Last metadata expiration check: 0:01:45 ago on Fri 02 Aug 2024 11:36:54 PM UTC. 2024-08-02T23:38:39.478 INFO:teuthology.orchestra.run.smithi073.stdout:Package ceph-base-2:19.1.0-1260.g26c3fb8e.el9.x86_64 is already installed. 2024-08-02T23:38:39.519 INFO:teuthology.orchestra.run.smithi073.stdout:Dependencies resolved. 2024-08-02T23:38:39.520 INFO:teuthology.orchestra.run.smithi073.stdout:Nothing to do. 2024-08-02T23:38:39.521 INFO:teuthology.orchestra.run.smithi073.stdout:Complete! 2024-08-02T23:38:39.583 DEBUG:teuthology.orchestra.run.smithi073:> sudo yum -y install cephadm 2024-08-02T23:38:39.876 INFO:teuthology.orchestra.run.smithi131.stdout:cephadm-19.1.0-1260.g26c3fb8e.el9.noarch.rpm 1.3 MB/s | 765 kB 00:00 2024-08-02T23:38:39.877 INFO:teuthology.orchestra.run.smithi131.stdout:-------------------------------------------------------------------------------- 2024-08-02T23:38:39.877 INFO:teuthology.orchestra.run.smithi131.stdout:Total 1.3 MB/s | 765 kB 00:00 2024-08-02T23:38:39.877 INFO:teuthology.orchestra.run.smithi131.stdout:Running transaction check 2024-08-02T23:38:39.882 INFO:teuthology.orchestra.run.smithi131.stdout:Transaction check succeeded. 2024-08-02T23:38:39.882 INFO:teuthology.orchestra.run.smithi131.stdout:Running transaction test 2024-08-02T23:38:39.888 INFO:teuthology.orchestra.run.smithi131.stdout:Transaction test succeeded. 2024-08-02T23:38:39.888 INFO:teuthology.orchestra.run.smithi131.stdout:Running transaction 2024-08-02T23:38:39.963 INFO:teuthology.orchestra.run.smithi131.stdout: Preparing : 1/1 2024-08-02T23:38:40.125 INFO:teuthology.orchestra.run.smithi073.stdout:Last metadata expiration check: 0:01:46 ago on Fri 02 Aug 2024 11:36:54 PM UTC. 2024-08-02T23:38:40.251 INFO:teuthology.orchestra.run.smithi073.stdout:Dependencies resolved. 2024-08-02T23:38:40.251 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2024-08-02T23:38:40.251 INFO:teuthology.orchestra.run.smithi073.stdout: Package Arch Version Repository Size 2024-08-02T23:38:40.252 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2024-08-02T23:38:40.252 INFO:teuthology.orchestra.run.smithi073.stdout:Installing: 2024-08-02T23:38:40.252 INFO:teuthology.orchestra.run.smithi073.stdout: cephadm noarch 2:19.1.0-1260.g26c3fb8e.el9 ceph-noarch 765 k 2024-08-02T23:38:40.252 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-08-02T23:38:40.252 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction Summary 2024-08-02T23:38:40.252 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2024-08-02T23:38:40.252 INFO:teuthology.orchestra.run.smithi073.stdout:Install 1 Package 2024-08-02T23:38:40.252 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-08-02T23:38:40.252 INFO:teuthology.orchestra.run.smithi073.stdout:Total download size: 765 k 2024-08-02T23:38:40.252 INFO:teuthology.orchestra.run.smithi073.stdout:Installed size: 771 k 2024-08-02T23:38:40.252 INFO:teuthology.orchestra.run.smithi073.stdout:Downloading Packages: 2024-08-02T23:38:40.437 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: cephadm-2:19.1.0-1260.g26c3fb8e.el9.noarch 1/1 2024-08-02T23:38:40.510 INFO:teuthology.orchestra.run.smithi073.stdout:cephadm-19.1.0-1260.g26c3fb8e.el9.noarch.rpm 2.9 MB/s | 765 kB 00:00 2024-08-02T23:38:40.510 INFO:teuthology.orchestra.run.smithi073.stdout:-------------------------------------------------------------------------------- 2024-08-02T23:38:40.511 INFO:teuthology.orchestra.run.smithi073.stdout:Total 2.9 MB/s | 765 kB 00:00 2024-08-02T23:38:40.511 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction check 2024-08-02T23:38:40.516 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction check succeeded. 2024-08-02T23:38:40.516 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction test 2024-08-02T23:38:40.522 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction test succeeded. 2024-08-02T23:38:40.522 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction 2024-08-02T23:38:40.530 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : cephadm-2:19.1.0-1260.g26c3fb8e.el9.noarch 1/1 2024-08-02T23:38:40.598 INFO:teuthology.orchestra.run.smithi073.stdout: Preparing : 1/1 2024-08-02T23:38:40.890 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: cephadm-2:19.1.0-1260.g26c3fb8e.el9.noarch 1/1 2024-08-02T23:38:41.037 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: cephadm-2:19.1.0-1260.g26c3fb8e.el9.noarch 1/1 2024-08-02T23:38:41.138 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : cephadm-2:19.1.0-1260.g26c3fb8e.el9.noarch 1/1 2024-08-02T23:38:41.147 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : cephadm-2:19.1.0-1260.g26c3fb8e.el9.noarch 1/1 2024-08-02T23:38:41.147 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-08-02T23:38:41.147 INFO:teuthology.orchestra.run.smithi131.stdout:Installed: 2024-08-02T23:38:41.147 INFO:teuthology.orchestra.run.smithi131.stdout: cephadm-2:19.1.0-1260.g26c3fb8e.el9.noarch 2024-08-02T23:38:41.147 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-08-02T23:38:41.147 INFO:teuthology.orchestra.run.smithi131.stdout:Complete! 2024-08-02T23:38:41.362 DEBUG:teuthology.orchestra.run.smithi131:> sudo yum -y install ceph-immutable-object-cache 2024-08-02T23:38:41.482 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: cephadm-2:19.1.0-1260.g26c3fb8e.el9.noarch 1/1 2024-08-02T23:38:41.729 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : cephadm-2:19.1.0-1260.g26c3fb8e.el9.noarch 1/1 2024-08-02T23:38:41.730 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-08-02T23:38:41.730 INFO:teuthology.orchestra.run.smithi073.stdout:Installed: 2024-08-02T23:38:41.730 INFO:teuthology.orchestra.run.smithi073.stdout: cephadm-2:19.1.0-1260.g26c3fb8e.el9.noarch 2024-08-02T23:38:41.730 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-08-02T23:38:41.730 INFO:teuthology.orchestra.run.smithi073.stdout:Complete! 2024-08-02T23:38:41.907 DEBUG:teuthology.orchestra.run.smithi073:> sudo yum -y install ceph-immutable-object-cache 2024-08-02T23:38:41.920 INFO:teuthology.orchestra.run.smithi131.stdout:Last metadata expiration check: 0:01:49 ago on Fri 02 Aug 2024 11:36:52 PM UTC. 2024-08-02T23:38:42.049 INFO:teuthology.orchestra.run.smithi131.stdout:Dependencies resolved. 2024-08-02T23:38:42.050 INFO:teuthology.orchestra.run.smithi131.stdout:================================================================================ 2024-08-02T23:38:42.050 INFO:teuthology.orchestra.run.smithi131.stdout: Package Arch Version Repo Size 2024-08-02T23:38:42.050 INFO:teuthology.orchestra.run.smithi131.stdout:================================================================================ 2024-08-02T23:38:42.050 INFO:teuthology.orchestra.run.smithi131.stdout:Installing: 2024-08-02T23:38:42.050 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-immutable-object-cache x86_64 2:19.1.0-1260.g26c3fb8e.el9 ceph 148 k 2024-08-02T23:38:42.050 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-08-02T23:38:42.050 INFO:teuthology.orchestra.run.smithi131.stdout:Transaction Summary 2024-08-02T23:38:42.050 INFO:teuthology.orchestra.run.smithi131.stdout:================================================================================ 2024-08-02T23:38:42.050 INFO:teuthology.orchestra.run.smithi131.stdout:Install 1 Package 2024-08-02T23:38:42.050 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-08-02T23:38:42.050 INFO:teuthology.orchestra.run.smithi131.stdout:Total download size: 148 k 2024-08-02T23:38:42.051 INFO:teuthology.orchestra.run.smithi131.stdout:Installed size: 481 k 2024-08-02T23:38:42.051 INFO:teuthology.orchestra.run.smithi131.stdout:Downloading Packages: 2024-08-02T23:38:42.274 INFO:teuthology.orchestra.run.smithi131.stdout:ceph-immutable-object-cache-19.1.0-1260.g26c3fb 662 kB/s | 148 kB 00:00 2024-08-02T23:38:42.275 INFO:teuthology.orchestra.run.smithi131.stdout:-------------------------------------------------------------------------------- 2024-08-02T23:38:42.275 INFO:teuthology.orchestra.run.smithi131.stdout:Total 656 kB/s | 148 kB 00:00 2024-08-02T23:38:42.276 INFO:teuthology.orchestra.run.smithi131.stdout:Running transaction check 2024-08-02T23:38:42.282 INFO:teuthology.orchestra.run.smithi131.stdout:Transaction check succeeded. 2024-08-02T23:38:42.283 INFO:teuthology.orchestra.run.smithi131.stdout:Running transaction test 2024-08-02T23:38:42.339 INFO:teuthology.orchestra.run.smithi131.stdout:Transaction test succeeded. 2024-08-02T23:38:42.340 INFO:teuthology.orchestra.run.smithi131.stdout:Running transaction 2024-08-02T23:38:42.465 INFO:teuthology.orchestra.run.smithi073.stdout:Last metadata expiration check: 0:01:48 ago on Fri 02 Aug 2024 11:36:54 PM UTC. 2024-08-02T23:38:42.480 INFO:teuthology.orchestra.run.smithi131.stdout: Preparing : 1/1 2024-08-02T23:38:42.575 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : ceph-immutable-object-cache-2:19.1.0-1260.g26c3fb8e. 1/1 2024-08-02T23:38:42.596 INFO:teuthology.orchestra.run.smithi073.stdout:Dependencies resolved. 2024-08-02T23:38:42.597 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2024-08-02T23:38:42.597 INFO:teuthology.orchestra.run.smithi073.stdout: Package Arch Version Repo Size 2024-08-02T23:38:42.597 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2024-08-02T23:38:42.597 INFO:teuthology.orchestra.run.smithi073.stdout:Installing: 2024-08-02T23:38:42.597 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-immutable-object-cache x86_64 2:19.1.0-1260.g26c3fb8e.el9 ceph 148 k 2024-08-02T23:38:42.597 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-08-02T23:38:42.597 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction Summary 2024-08-02T23:38:42.597 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2024-08-02T23:38:42.597 INFO:teuthology.orchestra.run.smithi073.stdout:Install 1 Package 2024-08-02T23:38:42.597 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-08-02T23:38:42.597 INFO:teuthology.orchestra.run.smithi073.stdout:Total download size: 148 k 2024-08-02T23:38:42.598 INFO:teuthology.orchestra.run.smithi073.stdout:Installed size: 481 k 2024-08-02T23:38:42.598 INFO:teuthology.orchestra.run.smithi073.stdout:Downloading Packages: 2024-08-02T23:38:42.600 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.1.0-1260.g26c3fb8e. 1/1 2024-08-02T23:38:42.600 INFO:teuthology.orchestra.run.smithi131.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-02T23:38:42.601 INFO:teuthology.orchestra.run.smithi131.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-08-02T23:38:42.601 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-08-02T23:38:42.833 INFO:teuthology.orchestra.run.smithi073.stdout:ceph-immutable-object-cache-19.1.0-1260.g26c3fb 628 kB/s | 148 kB 00:00 2024-08-02T23:38:42.833 INFO:teuthology.orchestra.run.smithi073.stdout:-------------------------------------------------------------------------------- 2024-08-02T23:38:42.834 INFO:teuthology.orchestra.run.smithi073.stdout:Total 624 kB/s | 148 kB 00:00 2024-08-02T23:38:42.834 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction check 2024-08-02T23:38:42.841 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction check succeeded. 2024-08-02T23:38:42.842 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction test 2024-08-02T23:38:42.898 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction test succeeded. 2024-08-02T23:38:42.898 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction 2024-08-02T23:38:43.049 INFO:teuthology.orchestra.run.smithi073.stdout: Preparing : 1/1 2024-08-02T23:38:43.117 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : ceph-immutable-object-cache-2:19.1.0-1260.g26c3fb8e. 1/1 2024-08-02T23:38:43.142 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.1.0-1260.g26c3fb8e. 1/1 2024-08-02T23:38:43.142 INFO:teuthology.orchestra.run.smithi073.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-02T23:38:43.143 INFO:teuthology.orchestra.run.smithi073.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-08-02T23:38:43.143 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-08-02T23:38:43.481 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : ceph-immutable-object-cache-2:19.1.0-1260.g26c3fb8e. 1/1 2024-08-02T23:38:43.481 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-08-02T23:38:43.481 INFO:teuthology.orchestra.run.smithi131.stdout:Installed: 2024-08-02T23:38:43.481 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-immutable-object-cache-2:19.1.0-1260.g26c3fb8e.el9.x86_64 2024-08-02T23:38:43.481 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-08-02T23:38:43.481 INFO:teuthology.orchestra.run.smithi131.stdout:Complete! 2024-08-02T23:38:43.694 DEBUG:teuthology.orchestra.run.smithi131:> sudo yum -y install ceph-mgr 2024-08-02T23:38:44.052 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : ceph-immutable-object-cache-2:19.1.0-1260.g26c3fb8e. 1/1 2024-08-02T23:38:44.052 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-08-02T23:38:44.052 INFO:teuthology.orchestra.run.smithi073.stdout:Installed: 2024-08-02T23:38:44.052 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-immutable-object-cache-2:19.1.0-1260.g26c3fb8e.el9.x86_64 2024-08-02T23:38:44.052 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-08-02T23:38:44.053 INFO:teuthology.orchestra.run.smithi073.stdout:Complete! 2024-08-02T23:38:44.243 DEBUG:teuthology.orchestra.run.smithi073:> sudo yum -y install ceph-mgr 2024-08-02T23:38:44.259 INFO:teuthology.orchestra.run.smithi131.stdout:Last metadata expiration check: 0:01:52 ago on Fri 02 Aug 2024 11:36:52 PM UTC. 2024-08-02T23:38:44.348 INFO:teuthology.orchestra.run.smithi131.stdout:Package ceph-mgr-2:19.1.0-1260.g26c3fb8e.el9.x86_64 is already installed. 2024-08-02T23:38:44.389 INFO:teuthology.orchestra.run.smithi131.stdout:Dependencies resolved. 2024-08-02T23:38:44.390 INFO:teuthology.orchestra.run.smithi131.stdout:Nothing to do. 2024-08-02T23:38:44.391 INFO:teuthology.orchestra.run.smithi131.stdout:Complete! 2024-08-02T23:38:44.461 DEBUG:teuthology.orchestra.run.smithi131:> sudo yum -y install ceph-mgr-dashboard 2024-08-02T23:38:44.805 INFO:teuthology.orchestra.run.smithi073.stdout:Last metadata expiration check: 0:01:50 ago on Fri 02 Aug 2024 11:36:54 PM UTC. 2024-08-02T23:38:44.891 INFO:teuthology.orchestra.run.smithi073.stdout:Package ceph-mgr-2:19.1.0-1260.g26c3fb8e.el9.x86_64 is already installed. 2024-08-02T23:38:44.934 INFO:teuthology.orchestra.run.smithi073.stdout:Dependencies resolved. 2024-08-02T23:38:44.935 INFO:teuthology.orchestra.run.smithi073.stdout:Nothing to do. 2024-08-02T23:38:44.935 INFO:teuthology.orchestra.run.smithi073.stdout:Complete! 2024-08-02T23:38:44.994 DEBUG:teuthology.orchestra.run.smithi073:> sudo yum -y install ceph-mgr-dashboard 2024-08-02T23:38:45.005 INFO:teuthology.orchestra.run.smithi131.stdout:Last metadata expiration check: 0:01:53 ago on Fri 02 Aug 2024 11:36:52 PM UTC. 2024-08-02T23:38:45.134 INFO:teuthology.orchestra.run.smithi131.stdout:Dependencies resolved. 2024-08-02T23:38:45.135 INFO:teuthology.orchestra.run.smithi131.stdout:================================================================================ 2024-08-02T23:38:45.135 INFO:teuthology.orchestra.run.smithi131.stdout: Package Arch Version Repository Size 2024-08-02T23:38:45.135 INFO:teuthology.orchestra.run.smithi131.stdout:================================================================================ 2024-08-02T23:38:45.135 INFO:teuthology.orchestra.run.smithi131.stdout:Installing: 2024-08-02T23:38:45.135 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-mgr-dashboard noarch 2:19.1.0-1260.g26c3fb8e.el9 ceph-noarch 3.6 M 2024-08-02T23:38:45.135 INFO:teuthology.orchestra.run.smithi131.stdout:Installing dependencies: 2024-08-02T23:38:45.135 INFO:teuthology.orchestra.run.smithi131.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-08-02T23:38:45.135 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-grafana-dashboards noarch 2:19.1.0-1260.g26c3fb8e.el9 ceph-noarch 32 k 2024-08-02T23:38:45.135 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-prometheus-alerts noarch 2:19.1.0-1260.g26c3fb8e.el9 ceph-noarch 16 k 2024-08-02T23:38:45.135 INFO:teuthology.orchestra.run.smithi131.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-08-02T23:38:45.136 INFO:teuthology.orchestra.run.smithi131.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-08-02T23:38:45.136 INFO:teuthology.orchestra.run.smithi131.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-08-02T23:38:45.136 INFO:teuthology.orchestra.run.smithi131.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-08-02T23:38:45.136 INFO:teuthology.orchestra.run.smithi131.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-08-02T23:38:45.136 INFO:teuthology.orchestra.run.smithi131.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-08-02T23:38:45.136 INFO:teuthology.orchestra.run.smithi131.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-08-02T23:38:45.136 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-08-02T23:38:45.136 INFO:teuthology.orchestra.run.smithi131.stdout:Transaction Summary 2024-08-02T23:38:45.136 INFO:teuthology.orchestra.run.smithi131.stdout:================================================================================ 2024-08-02T23:38:45.136 INFO:teuthology.orchestra.run.smithi131.stdout:Install 11 Packages 2024-08-02T23:38:45.136 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-08-02T23:38:45.137 INFO:teuthology.orchestra.run.smithi131.stdout:Total download size: 7.7 M 2024-08-02T23:38:45.137 INFO:teuthology.orchestra.run.smithi131.stdout:Installed size: 101 M 2024-08-02T23:38:45.137 INFO:teuthology.orchestra.run.smithi131.stdout:Downloading Packages: 2024-08-02T23:38:45.530 INFO:teuthology.orchestra.run.smithi073.stdout:Last metadata expiration check: 0:01:51 ago on Fri 02 Aug 2024 11:36:54 PM UTC. 2024-08-02T23:38:45.658 INFO:teuthology.orchestra.run.smithi073.stdout:Dependencies resolved. 2024-08-02T23:38:45.659 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2024-08-02T23:38:45.659 INFO:teuthology.orchestra.run.smithi073.stdout: Package Arch Version Repository Size 2024-08-02T23:38:45.659 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2024-08-02T23:38:45.659 INFO:teuthology.orchestra.run.smithi073.stdout:Installing: 2024-08-02T23:38:45.659 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-mgr-dashboard noarch 2:19.1.0-1260.g26c3fb8e.el9 ceph-noarch 3.6 M 2024-08-02T23:38:45.659 INFO:teuthology.orchestra.run.smithi073.stdout:Installing dependencies: 2024-08-02T23:38:45.659 INFO:teuthology.orchestra.run.smithi073.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-08-02T23:38:45.659 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-grafana-dashboards noarch 2:19.1.0-1260.g26c3fb8e.el9 ceph-noarch 32 k 2024-08-02T23:38:45.659 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-prometheus-alerts noarch 2:19.1.0-1260.g26c3fb8e.el9 ceph-noarch 16 k 2024-08-02T23:38:45.659 INFO:teuthology.orchestra.run.smithi073.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-08-02T23:38:45.660 INFO:teuthology.orchestra.run.smithi073.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-08-02T23:38:45.660 INFO:teuthology.orchestra.run.smithi073.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-08-02T23:38:45.660 INFO:teuthology.orchestra.run.smithi073.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-08-02T23:38:45.660 INFO:teuthology.orchestra.run.smithi073.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-08-02T23:38:45.660 INFO:teuthology.orchestra.run.smithi073.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-08-02T23:38:45.660 INFO:teuthology.orchestra.run.smithi073.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-08-02T23:38:45.660 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-08-02T23:38:45.660 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction Summary 2024-08-02T23:38:45.660 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2024-08-02T23:38:45.660 INFO:teuthology.orchestra.run.smithi073.stdout:Install 11 Packages 2024-08-02T23:38:45.660 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-08-02T23:38:45.661 INFO:teuthology.orchestra.run.smithi073.stdout:Total download size: 7.7 M 2024-08-02T23:38:45.661 INFO:teuthology.orchestra.run.smithi073.stdout:Installed size: 101 M 2024-08-02T23:38:45.661 INFO:teuthology.orchestra.run.smithi073.stdout:Downloading Packages: 2024-08-02T23:38:45.948 INFO:teuthology.orchestra.run.smithi131.stdout:(1/11): ceph-grafana-dashboards-19.1.0-1260.g26 159 kB/s | 32 kB 00:00 2024-08-02T23:38:45.973 INFO:teuthology.orchestra.run.smithi131.stdout:(2/11): ceph-prometheus-alerts-19.1.0-1260.g26c 73 kB/s | 16 kB 00:00 2024-08-02T23:38:46.157 INFO:teuthology.orchestra.run.smithi131.stdout:(3/11): ceph-mgr-dashboard-19.1.0-1260.g26c3fb8 8.7 MB/s | 3.6 MB 00:00 2024-08-02T23:38:46.216 INFO:teuthology.orchestra.run.smithi131.stdout:(4/11): python3-protobuf-3.14.0-13.el9.noarch.r 1.0 MB/s | 269 kB 00:00 2024-08-02T23:38:46.324 INFO:teuthology.orchestra.run.smithi131.stdout:(5/11): grpc-data-1.46.7-10.el9.noarch.rpm 183 kB/s | 19 kB 00:00 2024-08-02T23:38:46.358 INFO:teuthology.orchestra.run.smithi131.stdout:(6/11): abseil-cpp-20211102.0-3.el9.x86_64.rpm 2.7 MB/s | 548 kB 00:00 2024-08-02T23:38:46.375 INFO:teuthology.orchestra.run.smithi131.stdout:(7/11): python3-grpcio-tools-1.46.7-10.el9.x86_ 8.9 MB/s | 144 kB 00:00 2024-08-02T23:38:46.400 INFO:teuthology.orchestra.run.smithi131.stdout:(8/11): python3-repoze-lru-0.7-16.el9.noarch.rp 1.2 MB/s | 31 kB 00:00 2024-08-02T23:38:46.425 INFO:teuthology.orchestra.run.smithi131.stdout:(9/11): python3-routes-2.5.1-5.el9.noarch.rpm 7.4 MB/s | 188 kB 00:00 2024-08-02T23:38:46.470 INFO:teuthology.orchestra.run.smithi073.stdout:(1/11): ceph-prometheus-alerts-19.1.0-1260.g26c 85 kB/s | 16 kB 00:00 2024-08-02T23:38:46.484 INFO:teuthology.orchestra.run.smithi131.stdout:(10/11): python3-grpcio-1.46.7-10.el9.x86_64.rp 13 MB/s | 2.0 MB 00:00 2024-08-02T23:38:46.495 INFO:teuthology.orchestra.run.smithi073.stdout:(2/11): ceph-grafana-dashboards-19.1.0-1260.g26 146 kB/s | 32 kB 00:00 2024-08-02T23:38:46.671 INFO:teuthology.orchestra.run.smithi073.stdout:(3/11): ceph-mgr-dashboard-19.1.0-1260.g26c3fb8 9.0 MB/s | 3.6 MB 00:00 2024-08-02T23:38:46.743 INFO:teuthology.orchestra.run.smithi073.stdout:(4/11): abseil-cpp-20211102.0-3.el9.x86_64.rpm 7.5 MB/s | 548 kB 00:00 2024-08-02T23:38:46.769 INFO:teuthology.orchestra.run.smithi073.stdout:(5/11): grpc-data-1.46.7-10.el9.noarch.rpm 769 kB/s | 19 kB 00:00 2024-08-02T23:38:46.827 INFO:teuthology.orchestra.run.smithi073.stdout:(6/11): python3-grpcio-1.46.7-10.el9.x86_64.rpm 35 MB/s | 2.0 MB 00:00 2024-08-02T23:38:46.853 INFO:teuthology.orchestra.run.smithi073.stdout:(7/11): python3-protobuf-3.14.0-13.el9.noarch.r 704 kB/s | 269 kB 00:00 2024-08-02T23:38:46.886 INFO:teuthology.orchestra.run.smithi073.stdout:(8/11): protobuf-compiler-3.14.0-13.el9.x86_64. 2.2 MB/s | 863 kB 00:00 2024-08-02T23:38:46.970 INFO:teuthology.orchestra.run.smithi073.stdout:(9/11): python3-grpcio-tools-1.46.7-10.el9.x86_ 1.0 MB/s | 144 kB 00:00 2024-08-02T23:38:46.984 INFO:teuthology.orchestra.run.smithi131.stdout:(11/11): protobuf-compiler-3.14.0-13.el9.x86_64 854 kB/s | 863 kB 00:01 2024-08-02T23:38:46.989 INFO:teuthology.orchestra.run.smithi131.stdout:-------------------------------------------------------------------------------- 2024-08-02T23:38:46.989 INFO:teuthology.orchestra.run.smithi131.stdout:Total 4.1 MB/s | 7.7 MB 00:01 2024-08-02T23:38:47.001 INFO:teuthology.orchestra.run.smithi073.stdout:(10/11): python3-repoze-lru-0.7-16.el9.noarch.r 268 kB/s | 31 kB 00:00 2024-08-02T23:38:47.019 INFO:teuthology.orchestra.run.smithi131.stdout:Running transaction check 2024-08-02T23:38:47.027 INFO:teuthology.orchestra.run.smithi073.stdout:(11/11): python3-routes-2.5.1-5.el9.noarch.rpm 1.3 MB/s | 188 kB 00:00 2024-08-02T23:38:47.031 INFO:teuthology.orchestra.run.smithi073.stdout:-------------------------------------------------------------------------------- 2024-08-02T23:38:47.031 INFO:teuthology.orchestra.run.smithi073.stdout:Total 5.6 MB/s | 7.7 MB 00:01 2024-08-02T23:38:47.035 INFO:teuthology.orchestra.run.smithi131.stdout:Transaction check succeeded. 2024-08-02T23:38:47.036 INFO:teuthology.orchestra.run.smithi131.stdout:Running transaction test 2024-08-02T23:38:47.060 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction check 2024-08-02T23:38:47.077 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction check succeeded. 2024-08-02T23:38:47.077 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction test 2024-08-02T23:38:47.205 INFO:teuthology.orchestra.run.smithi131.stdout:Transaction test succeeded. 2024-08-02T23:38:47.206 INFO:teuthology.orchestra.run.smithi131.stdout:Running transaction 2024-08-02T23:38:47.245 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction test succeeded. 2024-08-02T23:38:47.246 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction 2024-08-02T23:38:47.690 INFO:teuthology.orchestra.run.smithi131.stdout: Preparing : 1/1 2024-08-02T23:38:47.741 INFO:teuthology.orchestra.run.smithi073.stdout: Preparing : 1/1 2024-08-02T23:38:47.766 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 1/11 2024-08-02T23:38:47.804 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 2/11 2024-08-02T23:38:47.825 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 1/11 2024-08-02T23:38:47.855 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 2/11 2024-08-02T23:38:47.905 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 3/11 2024-08-02T23:38:47.963 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 3/11 2024-08-02T23:38:48.030 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 4/11 2024-08-02T23:38:48.091 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 5/11 2024-08-02T23:38:48.098 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 4/11 2024-08-02T23:38:48.150 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 5/11 2024-08-02T23:38:48.175 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 6/11 2024-08-02T23:38:48.218 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 7/11 2024-08-02T23:38:48.234 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 6/11 2024-08-02T23:38:48.247 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 8/11 2024-08-02T23:38:48.278 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 7/11 2024-08-02T23:38:48.278 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : ceph-prometheus-alerts-2:19.1.0-1260.g26c3fb8e.el9 9/11 2024-08-02T23:38:48.306 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 8/11 2024-08-02T23:38:48.337 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : ceph-prometheus-alerts-2:19.1.0-1260.g26c3fb8e.el9 9/11 2024-08-02T23:38:49.334 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : ceph-grafana-dashboards-2:19.1.0-1260.g26c3fb8e.el 10/11 2024-08-02T23:38:49.371 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : ceph-mgr-dashboard-2:19.1.0-1260.g26c3fb8e.el9.noa 11/11 2024-08-02T23:38:49.384 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : ceph-grafana-dashboards-2:19.1.0-1260.g26c3fb8e.el 10/11 2024-08-02T23:38:49.421 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : ceph-mgr-dashboard-2:19.1.0-1260.g26c3fb8e.el9.noa 11/11 2024-08-02T23:38:50.791 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.1.0-1260.g26c3fb8e.el9.noa 11/11 2024-08-02T23:38:50.792 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : ceph-grafana-dashboards-2:19.1.0-1260.g26c3fb8e.el 1/11 2024-08-02T23:38:50.792 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : ceph-mgr-dashboard-2:19.1.0-1260.g26c3fb8e.el9.noa 2/11 2024-08-02T23:38:50.792 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : ceph-prometheus-alerts-2:19.1.0-1260.g26c3fb8e.el9 3/11 2024-08-02T23:38:50.792 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 4/11 2024-08-02T23:38:50.792 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 5/11 2024-08-02T23:38:50.792 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 6/11 2024-08-02T23:38:50.792 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 7/11 2024-08-02T23:38:50.793 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 8/11 2024-08-02T23:38:50.793 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/11 2024-08-02T23:38:50.793 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 10/11 2024-08-02T23:38:50.795 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.1.0-1260.g26c3fb8e.el9.noa 11/11 2024-08-02T23:38:50.795 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : ceph-grafana-dashboards-2:19.1.0-1260.g26c3fb8e.el 1/11 2024-08-02T23:38:50.795 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : ceph-mgr-dashboard-2:19.1.0-1260.g26c3fb8e.el9.noa 2/11 2024-08-02T23:38:50.795 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : ceph-prometheus-alerts-2:19.1.0-1260.g26c3fb8e.el9 3/11 2024-08-02T23:38:50.795 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 4/11 2024-08-02T23:38:50.795 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 5/11 2024-08-02T23:38:50.795 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 6/11 2024-08-02T23:38:50.795 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 7/11 2024-08-02T23:38:50.795 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 8/11 2024-08-02T23:38:50.795 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/11 2024-08-02T23:38:50.795 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 10/11 2024-08-02T23:38:51.093 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 11/11 2024-08-02T23:38:51.093 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-08-02T23:38:51.093 INFO:teuthology.orchestra.run.smithi073.stdout:Installed: 2024-08-02T23:38:51.094 INFO:teuthology.orchestra.run.smithi073.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-08-02T23:38:51.094 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-grafana-dashboards-2:19.1.0-1260.g26c3fb8e.el9.noarch 2024-08-02T23:38:51.094 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-mgr-dashboard-2:19.1.0-1260.g26c3fb8e.el9.noarch 2024-08-02T23:38:51.094 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-prometheus-alerts-2:19.1.0-1260.g26c3fb8e.el9.noarch 2024-08-02T23:38:51.094 INFO:teuthology.orchestra.run.smithi073.stdout: grpc-data-1.46.7-10.el9.noarch 2024-08-02T23:38:51.094 INFO:teuthology.orchestra.run.smithi073.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-08-02T23:38:51.094 INFO:teuthology.orchestra.run.smithi073.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-08-02T23:38:51.094 INFO:teuthology.orchestra.run.smithi073.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-08-02T23:38:51.094 INFO:teuthology.orchestra.run.smithi073.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-08-02T23:38:51.094 INFO:teuthology.orchestra.run.smithi073.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-08-02T23:38:51.094 INFO:teuthology.orchestra.run.smithi073.stdout: python3-routes-2.5.1-5.el9.noarch 2024-08-02T23:38:51.094 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-08-02T23:38:51.094 INFO:teuthology.orchestra.run.smithi073.stdout:Complete! 2024-08-02T23:38:51.114 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 11/11 2024-08-02T23:38:51.115 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-08-02T23:38:51.115 INFO:teuthology.orchestra.run.smithi131.stdout:Installed: 2024-08-02T23:38:51.115 INFO:teuthology.orchestra.run.smithi131.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-08-02T23:38:51.115 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-grafana-dashboards-2:19.1.0-1260.g26c3fb8e.el9.noarch 2024-08-02T23:38:51.115 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-mgr-dashboard-2:19.1.0-1260.g26c3fb8e.el9.noarch 2024-08-02T23:38:51.115 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-prometheus-alerts-2:19.1.0-1260.g26c3fb8e.el9.noarch 2024-08-02T23:38:51.116 INFO:teuthology.orchestra.run.smithi131.stdout: grpc-data-1.46.7-10.el9.noarch 2024-08-02T23:38:51.116 INFO:teuthology.orchestra.run.smithi131.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-08-02T23:38:51.116 INFO:teuthology.orchestra.run.smithi131.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-08-02T23:38:51.116 INFO:teuthology.orchestra.run.smithi131.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-08-02T23:38:51.116 INFO:teuthology.orchestra.run.smithi131.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-08-02T23:38:51.117 INFO:teuthology.orchestra.run.smithi131.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-08-02T23:38:51.117 INFO:teuthology.orchestra.run.smithi131.stdout: python3-routes-2.5.1-5.el9.noarch 2024-08-02T23:38:51.117 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-08-02T23:38:51.117 INFO:teuthology.orchestra.run.smithi131.stdout:Complete! 2024-08-02T23:38:51.328 DEBUG:teuthology.orchestra.run.smithi073:> sudo yum -y install ceph-mgr-diskprediction-local 2024-08-02T23:38:51.362 DEBUG:teuthology.orchestra.run.smithi131:> sudo yum -y install ceph-mgr-diskprediction-local 2024-08-02T23:38:51.869 INFO:teuthology.orchestra.run.smithi073.stdout:Last metadata expiration check: 0:01:57 ago on Fri 02 Aug 2024 11:36:54 PM UTC. 2024-08-02T23:38:51.912 INFO:teuthology.orchestra.run.smithi131.stdout:Last metadata expiration check: 0:01:59 ago on Fri 02 Aug 2024 11:36:52 PM UTC. 2024-08-02T23:38:51.996 INFO:teuthology.orchestra.run.smithi073.stdout:Dependencies resolved. 2024-08-02T23:38:51.997 INFO:teuthology.orchestra.run.smithi073.stdout:======================================================================================= 2024-08-02T23:38:51.997 INFO:teuthology.orchestra.run.smithi073.stdout: Package Arch Version Repository Size 2024-08-02T23:38:51.997 INFO:teuthology.orchestra.run.smithi073.stdout:======================================================================================= 2024-08-02T23:38:51.997 INFO:teuthology.orchestra.run.smithi073.stdout:Installing: 2024-08-02T23:38:51.997 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-mgr-diskprediction-local noarch 2:19.1.0-1260.g26c3fb8e.el9 ceph-noarch 7.4 M 2024-08-02T23:38:51.997 INFO:teuthology.orchestra.run.smithi073.stdout:Installing dependencies: 2024-08-02T23:38:51.997 INFO:teuthology.orchestra.run.smithi073.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-08-02T23:38:51.997 INFO:teuthology.orchestra.run.smithi073.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-08-02T23:38:51.998 INFO:teuthology.orchestra.run.smithi073.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-08-02T23:38:51.998 INFO:teuthology.orchestra.run.smithi073.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-08-02T23:38:51.998 INFO:teuthology.orchestra.run.smithi073.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-08-02T23:38:51.998 INFO:teuthology.orchestra.run.smithi073.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-08-02T23:38:51.998 INFO:teuthology.orchestra.run.smithi073.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-08-02T23:38:51.998 INFO:teuthology.orchestra.run.smithi073.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-08-02T23:38:51.998 INFO:teuthology.orchestra.run.smithi073.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-08-02T23:38:51.998 INFO:teuthology.orchestra.run.smithi073.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-08-02T23:38:51.998 INFO:teuthology.orchestra.run.smithi073.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-08-02T23:38:51.998 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-08-02T23:38:51.998 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction Summary 2024-08-02T23:38:51.998 INFO:teuthology.orchestra.run.smithi073.stdout:======================================================================================= 2024-08-02T23:38:51.998 INFO:teuthology.orchestra.run.smithi073.stdout:Install 12 Packages 2024-08-02T23:38:51.999 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-08-02T23:38:52.002 INFO:teuthology.orchestra.run.smithi073.stdout:Total download size: 38 M 2024-08-02T23:38:52.003 INFO:teuthology.orchestra.run.smithi073.stdout:Installed size: 209 M 2024-08-02T23:38:52.003 INFO:teuthology.orchestra.run.smithi073.stdout:Downloading Packages: 2024-08-02T23:38:52.040 INFO:teuthology.orchestra.run.smithi131.stdout:Dependencies resolved. 2024-08-02T23:38:52.041 INFO:teuthology.orchestra.run.smithi131.stdout:======================================================================================= 2024-08-02T23:38:52.042 INFO:teuthology.orchestra.run.smithi131.stdout: Package Arch Version Repository Size 2024-08-02T23:38:52.042 INFO:teuthology.orchestra.run.smithi131.stdout:======================================================================================= 2024-08-02T23:38:52.042 INFO:teuthology.orchestra.run.smithi131.stdout:Installing: 2024-08-02T23:38:52.042 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-mgr-diskprediction-local noarch 2:19.1.0-1260.g26c3fb8e.el9 ceph-noarch 7.4 M 2024-08-02T23:38:52.042 INFO:teuthology.orchestra.run.smithi131.stdout:Installing dependencies: 2024-08-02T23:38:52.042 INFO:teuthology.orchestra.run.smithi131.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-08-02T23:38:52.042 INFO:teuthology.orchestra.run.smithi131.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-08-02T23:38:52.042 INFO:teuthology.orchestra.run.smithi131.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-08-02T23:38:52.042 INFO:teuthology.orchestra.run.smithi131.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-08-02T23:38:52.042 INFO:teuthology.orchestra.run.smithi131.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-08-02T23:38:52.042 INFO:teuthology.orchestra.run.smithi131.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-08-02T23:38:52.042 INFO:teuthology.orchestra.run.smithi131.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-08-02T23:38:52.042 INFO:teuthology.orchestra.run.smithi131.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-08-02T23:38:52.043 INFO:teuthology.orchestra.run.smithi131.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-08-02T23:38:52.043 INFO:teuthology.orchestra.run.smithi131.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-08-02T23:38:52.043 INFO:teuthology.orchestra.run.smithi131.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-08-02T23:38:52.043 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-08-02T23:38:52.043 INFO:teuthology.orchestra.run.smithi131.stdout:Transaction Summary 2024-08-02T23:38:52.043 INFO:teuthology.orchestra.run.smithi131.stdout:======================================================================================= 2024-08-02T23:38:52.043 INFO:teuthology.orchestra.run.smithi131.stdout:Install 12 Packages 2024-08-02T23:38:52.043 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-08-02T23:38:52.044 INFO:teuthology.orchestra.run.smithi131.stdout:Total download size: 38 M 2024-08-02T23:38:52.044 INFO:teuthology.orchestra.run.smithi131.stdout:Installed size: 209 M 2024-08-02T23:38:52.044 INFO:teuthology.orchestra.run.smithi131.stdout:Downloading Packages: 2024-08-02T23:38:52.476 INFO:teuthology.orchestra.run.smithi131.stdout:(1/12): libquadmath-11.4.1-3.el9.x86_64.rpm 1.0 MB/s | 195 kB 00:00 2024-08-02T23:38:52.559 INFO:teuthology.orchestra.run.smithi131.stdout:(2/12): flexiblas-3.0.4-8.el9.x86_64.rpm 393 kB/s | 33 kB 00:00 2024-08-02T23:38:52.576 INFO:teuthology.orchestra.run.smithi073.stdout:(1/12): libquadmath-11.4.1-3.el9.x86_64.rpm 818 kB/s | 195 kB 00:00 2024-08-02T23:38:52.618 INFO:teuthology.orchestra.run.smithi131.stdout:(3/12): libgfortran-11.4.1-3.el9.x86_64.rpm 2.3 MB/s | 803 kB 00:00 2024-08-02T23:38:52.643 INFO:teuthology.orchestra.run.smithi073.stdout:(2/12): libgfortran-11.4.1-3.el9.x86_64.rpm 2.6 MB/s | 803 kB 00:00 2024-08-02T23:38:52.668 INFO:teuthology.orchestra.run.smithi073.stdout:(3/12): flexiblas-3.0.4-8.el9.x86_64.rpm 356 kB/s | 33 kB 00:00 2024-08-02T23:38:52.718 INFO:teuthology.orchestra.run.smithi073.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 365 kB/s | 18 kB 00:00 2024-08-02T23:38:52.777 INFO:teuthology.orchestra.run.smithi131.stdout:(4/12): ceph-mgr-diskprediction-local-19.1.0-12 15 MB/s | 7.4 MB 00:00 2024-08-02T23:38:52.777 INFO:teuthology.orchestra.run.smithi073.stdout:(5/12): openblas-0.3.21-2.el9.x86_64.rpm 603 kB/s | 35 kB 00:00 2024-08-02T23:38:52.802 INFO:teuthology.orchestra.run.smithi131.stdout:(5/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 99 kB/s | 18 kB 00:00 2024-08-02T23:38:52.836 INFO:teuthology.orchestra.run.smithi131.stdout:(6/12): openblas-0.3.21-2.el9.x86_64.rpm 602 kB/s | 35 kB 00:00 2024-08-02T23:38:52.894 INFO:teuthology.orchestra.run.smithi131.stdout:(7/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 8.9 MB/s | 3.0 MB 00:00 2024-08-02T23:38:52.910 INFO:teuthology.orchestra.run.smithi073.stdout:(6/12): ceph-mgr-diskprediction-local-19.1.0-12 13 MB/s | 7.4 MB 00:00 2024-08-02T23:38:52.969 INFO:teuthology.orchestra.run.smithi131.stdout:(8/12): python3-devel-3.9.18-3.el9.x86_64.rpm 1.8 MB/s | 244 kB 00:00 2024-08-02T23:38:52.977 INFO:teuthology.orchestra.run.smithi073.stdout:(7/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 8.9 MB/s | 3.0 MB 00:00 2024-08-02T23:38:53.061 INFO:teuthology.orchestra.run.smithi131.stdout:(9/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 31 MB/s | 5.1 MB 00:00 2024-08-02T23:38:53.087 INFO:teuthology.orchestra.run.smithi131.stdout:(10/12): python3-numpy-f2py-1.20.1-5.el9.x86_64 2.1 MB/s | 250 kB 00:00 2024-08-02T23:38:53.120 INFO:teuthology.orchestra.run.smithi073.stdout:(8/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 36 MB/s | 5.1 MB 00:00 2024-08-02T23:38:53.170 INFO:teuthology.orchestra.run.smithi131.stdout:(11/12): openblas-openmp-0.3.21-2.el9.x86_64.rp 13 MB/s | 4.7 MB 00:00 2024-08-02T23:38:53.186 INFO:teuthology.orchestra.run.smithi073.stdout:(9/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 12 MB/s | 4.7 MB 00:00 2024-08-02T23:38:53.211 INFO:teuthology.orchestra.run.smithi073.stdout:(10/12): python3-devel-3.9.18-3.el9.x86_64.rpm 812 kB/s | 244 kB 00:00 2024-08-02T23:38:53.237 INFO:teuthology.orchestra.run.smithi073.stdout:(11/12): python3-numpy-f2py-1.20.1-5.el9.x86_64 2.1 MB/s | 250 kB 00:00 2024-08-02T23:38:53.420 INFO:teuthology.orchestra.run.smithi131.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 45 MB/s | 16 MB 00:00 2024-08-02T23:38:53.422 INFO:teuthology.orchestra.run.smithi131.stdout:-------------------------------------------------------------------------------- 2024-08-02T23:38:53.422 INFO:teuthology.orchestra.run.smithi131.stdout:Total 28 MB/s | 38 MB 00:01 2024-08-02T23:38:53.487 INFO:teuthology.orchestra.run.smithi073.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 54 MB/s | 16 MB 00:00 2024-08-02T23:38:53.488 INFO:teuthology.orchestra.run.smithi073.stdout:-------------------------------------------------------------------------------- 2024-08-02T23:38:53.488 INFO:teuthology.orchestra.run.smithi073.stdout:Total 26 MB/s | 38 MB 00:01 2024-08-02T23:38:53.769 INFO:teuthology.orchestra.run.smithi131.stdout:Running transaction check 2024-08-02T23:38:53.795 INFO:teuthology.orchestra.run.smithi131.stdout:Transaction check succeeded. 2024-08-02T23:38:53.795 INFO:teuthology.orchestra.run.smithi131.stdout:Running transaction test 2024-08-02T23:38:53.838 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction check 2024-08-02T23:38:53.866 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction check succeeded. 2024-08-02T23:38:53.866 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction test 2024-08-02T23:38:54.223 INFO:teuthology.orchestra.run.smithi131.stdout:Transaction test succeeded. 2024-08-02T23:38:54.223 INFO:teuthology.orchestra.run.smithi131.stdout:Running transaction 2024-08-02T23:38:54.298 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction test succeeded. 2024-08-02T23:38:54.298 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction 2024-08-02T23:38:55.006 INFO:teuthology.orchestra.run.smithi131.stdout: Preparing : 1/1 2024-08-02T23:38:55.076 INFO:teuthology.orchestra.run.smithi073.stdout: Preparing : 1/1 2024-08-02T23:38:55.076 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-08-02T23:38:55.119 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 2/12 2024-08-02T23:38:55.143 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-08-02T23:38:55.185 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 2/12 2024-08-02T23:38:55.218 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 3/12 2024-08-02T23:38:55.253 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 4/12 2024-08-02T23:38:55.285 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 3/12 2024-08-02T23:38:55.320 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 4/12 2024-08-02T23:38:55.429 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-08-02T23:38:55.461 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-08-02T23:38:55.497 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-08-02T23:38:55.527 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-08-02T23:38:55.544 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-08-02T23:38:55.614 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-08-02T23:38:56.191 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-08-02T23:38:56.257 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-08-02T23:38:56.315 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-08-02T23:38:56.374 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-08-02T23:38:57.533 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-08-02T23:38:57.601 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-08-02T23:38:58.568 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-08-02T23:38:58.595 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : ceph-mgr-diskprediction-local-2:19.1.0-1260.g26c3f 12/12 2024-08-02T23:38:58.637 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-08-02T23:38:58.660 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : ceph-mgr-diskprediction-local-2:19.1.0-1260.g26c3f 12/12 2024-08-02T23:39:01.171 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.1.0-1260.g26c3f 12/12 2024-08-02T23:39:01.171 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.1.0-1260.g26c3f 1/12 2024-08-02T23:39:01.171 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-08-02T23:39:01.172 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 3/12 2024-08-02T23:39:01.172 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-08-02T23:39:01.172 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-08-02T23:39:01.172 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-08-02T23:39:01.172 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 7/12 2024-08-02T23:39:01.172 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 8/12 2024-08-02T23:39:01.172 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 9/12 2024-08-02T23:39:01.172 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-08-02T23:39:01.172 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-08-02T23:39:01.410 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.1.0-1260.g26c3f 12/12 2024-08-02T23:39:01.411 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.1.0-1260.g26c3f 1/12 2024-08-02T23:39:01.411 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-08-02T23:39:01.411 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 3/12 2024-08-02T23:39:01.411 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-08-02T23:39:01.411 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-08-02T23:39:01.411 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-08-02T23:39:01.411 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 7/12 2024-08-02T23:39:01.411 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 8/12 2024-08-02T23:39:01.411 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 9/12 2024-08-02T23:39:01.411 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-08-02T23:39:01.411 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-08-02T23:39:01.595 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-08-02T23:39:01.595 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-08-02T23:39:01.595 INFO:teuthology.orchestra.run.smithi073.stdout:Installed: 2024-08-02T23:39:01.595 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-mgr-diskprediction-local-2:19.1.0-1260.g26c3fb8e.el9.noarch 2024-08-02T23:39:01.596 INFO:teuthology.orchestra.run.smithi073.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-08-02T23:39:01.596 INFO:teuthology.orchestra.run.smithi073.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-08-02T23:39:01.596 INFO:teuthology.orchestra.run.smithi073.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-08-02T23:39:01.596 INFO:teuthology.orchestra.run.smithi073.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-08-02T23:39:01.596 INFO:teuthology.orchestra.run.smithi073.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-08-02T23:39:01.596 INFO:teuthology.orchestra.run.smithi073.stdout: openblas-0.3.21-2.el9.x86_64 2024-08-02T23:39:01.596 INFO:teuthology.orchestra.run.smithi073.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-08-02T23:39:01.596 INFO:teuthology.orchestra.run.smithi073.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-08-02T23:39:01.596 INFO:teuthology.orchestra.run.smithi073.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-08-02T23:39:01.596 INFO:teuthology.orchestra.run.smithi073.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-08-02T23:39:01.596 INFO:teuthology.orchestra.run.smithi073.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-08-02T23:39:01.596 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-08-02T23:39:01.596 INFO:teuthology.orchestra.run.smithi073.stdout:Complete! 2024-08-02T23:39:01.814 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-08-02T23:39:01.814 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-08-02T23:39:01.814 INFO:teuthology.orchestra.run.smithi131.stdout:Installed: 2024-08-02T23:39:01.814 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-mgr-diskprediction-local-2:19.1.0-1260.g26c3fb8e.el9.noarch 2024-08-02T23:39:01.814 INFO:teuthology.orchestra.run.smithi131.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-08-02T23:39:01.814 INFO:teuthology.orchestra.run.smithi131.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-08-02T23:39:01.814 INFO:teuthology.orchestra.run.smithi131.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-08-02T23:39:01.815 INFO:teuthology.orchestra.run.smithi131.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-08-02T23:39:01.815 INFO:teuthology.orchestra.run.smithi131.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-08-02T23:39:01.815 INFO:teuthology.orchestra.run.smithi131.stdout: openblas-0.3.21-2.el9.x86_64 2024-08-02T23:39:01.815 INFO:teuthology.orchestra.run.smithi131.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-08-02T23:39:01.815 INFO:teuthology.orchestra.run.smithi131.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-08-02T23:39:01.815 INFO:teuthology.orchestra.run.smithi131.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-08-02T23:39:01.815 INFO:teuthology.orchestra.run.smithi131.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-08-02T23:39:01.815 INFO:teuthology.orchestra.run.smithi131.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-08-02T23:39:01.815 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-08-02T23:39:01.815 INFO:teuthology.orchestra.run.smithi131.stdout:Complete! 2024-08-02T23:39:02.022 DEBUG:teuthology.orchestra.run.smithi073:> sudo yum -y install ceph-mgr-rook 2024-08-02T23:39:02.206 DEBUG:teuthology.orchestra.run.smithi131:> sudo yum -y install ceph-mgr-rook 2024-08-02T23:39:02.589 INFO:teuthology.orchestra.run.smithi073.stdout:Last metadata expiration check: 0:02:08 ago on Fri 02 Aug 2024 11:36:54 PM UTC. 2024-08-02T23:39:02.717 INFO:teuthology.orchestra.run.smithi073.stdout:Dependencies resolved. 2024-08-02T23:39:02.718 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2024-08-02T23:39:02.718 INFO:teuthology.orchestra.run.smithi073.stdout: Package Arch Version Repository Size 2024-08-02T23:39:02.718 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2024-08-02T23:39:02.718 INFO:teuthology.orchestra.run.smithi073.stdout:Installing: 2024-08-02T23:39:02.718 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-mgr-rook noarch 2:19.1.0-1260.g26c3fb8e.el9 ceph-noarch 49 k 2024-08-02T23:39:02.718 INFO:teuthology.orchestra.run.smithi073.stdout:Installing dependencies: 2024-08-02T23:39:02.718 INFO:teuthology.orchestra.run.smithi073.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-08-02T23:39:02.718 INFO:teuthology.orchestra.run.smithi073.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-08-02T23:39:02.719 INFO:teuthology.orchestra.run.smithi073.stdout: python3-google-auth noarch 1:2.32.0-1.el9 epel 216 k 2024-08-02T23:39:02.719 INFO:teuthology.orchestra.run.smithi073.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-08-02T23:39:02.719 INFO:teuthology.orchestra.run.smithi073.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-08-02T23:39:02.719 INFO:teuthology.orchestra.run.smithi073.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-08-02T23:39:02.719 INFO:teuthology.orchestra.run.smithi073.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-08-02T23:39:02.719 INFO:teuthology.orchestra.run.smithi073.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-08-02T23:39:02.719 INFO:teuthology.orchestra.run.smithi073.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-08-02T23:39:02.719 INFO:teuthology.orchestra.run.smithi073.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-08-02T23:39:02.719 INFO:teuthology.orchestra.run.smithi073.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-08-02T23:39:02.719 INFO:teuthology.orchestra.run.smithi073.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-08-02T23:39:02.719 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-08-02T23:39:02.719 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction Summary 2024-08-02T23:39:02.719 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2024-08-02T23:39:02.719 INFO:teuthology.orchestra.run.smithi073.stdout:Install 13 Packages 2024-08-02T23:39:02.720 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-08-02T23:39:02.720 INFO:teuthology.orchestra.run.smithi073.stdout:Total download size: 2.2 M 2024-08-02T23:39:02.720 INFO:teuthology.orchestra.run.smithi073.stdout:Installed size: 26 M 2024-08-02T23:39:02.720 INFO:teuthology.orchestra.run.smithi073.stdout:Downloading Packages: 2024-08-02T23:39:02.763 INFO:teuthology.orchestra.run.smithi131.stdout:Last metadata expiration check: 0:02:10 ago on Fri 02 Aug 2024 11:36:52 PM UTC. 2024-08-02T23:39:02.891 INFO:teuthology.orchestra.run.smithi131.stdout:Dependencies resolved. 2024-08-02T23:39:02.892 INFO:teuthology.orchestra.run.smithi131.stdout:================================================================================ 2024-08-02T23:39:02.892 INFO:teuthology.orchestra.run.smithi131.stdout: Package Arch Version Repository Size 2024-08-02T23:39:02.892 INFO:teuthology.orchestra.run.smithi131.stdout:================================================================================ 2024-08-02T23:39:02.892 INFO:teuthology.orchestra.run.smithi131.stdout:Installing: 2024-08-02T23:39:02.892 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-mgr-rook noarch 2:19.1.0-1260.g26c3fb8e.el9 ceph-noarch 49 k 2024-08-02T23:39:02.892 INFO:teuthology.orchestra.run.smithi131.stdout:Installing dependencies: 2024-08-02T23:39:02.892 INFO:teuthology.orchestra.run.smithi131.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-08-02T23:39:02.892 INFO:teuthology.orchestra.run.smithi131.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-08-02T23:39:02.892 INFO:teuthology.orchestra.run.smithi131.stdout: python3-google-auth noarch 1:2.32.0-1.el9 epel 216 k 2024-08-02T23:39:02.893 INFO:teuthology.orchestra.run.smithi131.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-08-02T23:39:02.893 INFO:teuthology.orchestra.run.smithi131.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-08-02T23:39:02.893 INFO:teuthology.orchestra.run.smithi131.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-08-02T23:39:02.893 INFO:teuthology.orchestra.run.smithi131.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-08-02T23:39:02.893 INFO:teuthology.orchestra.run.smithi131.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-08-02T23:39:02.893 INFO:teuthology.orchestra.run.smithi131.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-08-02T23:39:02.893 INFO:teuthology.orchestra.run.smithi131.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-08-02T23:39:02.893 INFO:teuthology.orchestra.run.smithi131.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-08-02T23:39:02.893 INFO:teuthology.orchestra.run.smithi131.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-08-02T23:39:02.893 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-08-02T23:39:02.893 INFO:teuthology.orchestra.run.smithi131.stdout:Transaction Summary 2024-08-02T23:39:02.893 INFO:teuthology.orchestra.run.smithi131.stdout:================================================================================ 2024-08-02T23:39:02.893 INFO:teuthology.orchestra.run.smithi131.stdout:Install 13 Packages 2024-08-02T23:39:02.893 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-08-02T23:39:02.894 INFO:teuthology.orchestra.run.smithi131.stdout:Total download size: 2.2 M 2024-08-02T23:39:02.894 INFO:teuthology.orchestra.run.smithi131.stdout:Installed size: 26 M 2024-08-02T23:39:02.894 INFO:teuthology.orchestra.run.smithi131.stdout:Downloading Packages: 2024-08-02T23:39:03.069 INFO:teuthology.orchestra.run.smithi073.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 159 kB/s | 19 kB 00:00 2024-08-02T23:39:03.094 INFO:teuthology.orchestra.run.smithi073.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 175 kB/s | 26 kB 00:00 2024-08-02T23:39:03.153 INFO:teuthology.orchestra.run.smithi073.stdout:(3/13): ceph-mgr-rook-19.1.0-1260.g26c3fb8e.el9 241 kB/s | 49 kB 00:00 2024-08-02T23:39:03.178 INFO:teuthology.orchestra.run.smithi073.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.9 MB/s | 159 kB 00:00 2024-08-02T23:39:03.203 INFO:teuthology.orchestra.run.smithi073.stdout:(5/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.6 MB/s | 222 kB 00:00 2024-08-02T23:39:03.228 INFO:teuthology.orchestra.run.smithi073.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 1.0 MB/s | 54 kB 00:00 2024-08-02T23:39:03.320 INFO:teuthology.orchestra.run.smithi073.stdout:(7/13): python3-certifi-2023.05.07-4.el9.noarch 154 kB/s | 14 kB 00:00 2024-08-02T23:39:03.330 INFO:teuthology.orchestra.run.smithi131.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 216 kB/s | 19 kB 00:00 2024-08-02T23:39:03.345 INFO:teuthology.orchestra.run.smithi073.stdout:(8/13): python3-pyasn1-modules-0.4.8-6.el9.noar 1.4 MB/s | 279 kB 00:00 2024-08-02T23:39:03.355 INFO:teuthology.orchestra.run.smithi131.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 225 kB/s | 26 kB 00:00 2024-08-02T23:39:03.371 INFO:teuthology.orchestra.run.smithi073.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 192 kB/s | 32 kB 00:00 2024-08-02T23:39:03.421 INFO:teuthology.orchestra.run.smithi073.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 2.3 MB/s | 59 kB 00:00 2024-08-02T23:39:03.446 INFO:teuthology.orchestra.run.smithi073.stdout:(11/13): python3-google-auth-2.32.0-1.el9.noarc 1.7 MB/s | 216 kB 00:00 2024-08-02T23:39:03.456 INFO:teuthology.orchestra.run.smithi131.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.7 MB/s | 222 kB 00:00 2024-08-02T23:39:03.472 INFO:teuthology.orchestra.run.smithi073.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.8 MB/s | 90 kB 00:00 2024-08-02T23:39:03.481 INFO:teuthology.orchestra.run.smithi131.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.2 MB/s | 159 kB 00:00 2024-08-02T23:39:03.507 INFO:teuthology.orchestra.run.smithi131.stdout:(5/13): ceph-mgr-rook-19.1.0-1260.g26c3fb8e.el9 186 kB/s | 49 kB 00:00 2024-08-02T23:39:03.513 INFO:teuthology.orchestra.run.smithi073.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 6.1 MB/s | 1.0 MB 00:00 2024-08-02T23:39:03.516 INFO:teuthology.orchestra.run.smithi073.stdout:-------------------------------------------------------------------------------- 2024-08-02T23:39:03.516 INFO:teuthology.orchestra.run.smithi073.stdout:Total 2.8 MB/s | 2.2 MB 00:00 2024-08-02T23:39:03.532 INFO:teuthology.orchestra.run.smithi131.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 1.0 MB/s | 54 kB 00:00 2024-08-02T23:39:03.565 INFO:teuthology.orchestra.run.smithi131.stdout:(7/13): python3-pyasn1-modules-0.4.8-6.el9.noar 2.5 MB/s | 279 kB 00:00 2024-08-02T23:39:03.579 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction check 2024-08-02T23:39:03.594 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction check succeeded. 2024-08-02T23:39:03.594 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction test 2024-08-02T23:39:03.666 INFO:teuthology.orchestra.run.smithi131.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 202 kB/s | 32 kB 00:00 2024-08-02T23:39:03.678 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction test succeeded. 2024-08-02T23:39:03.680 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction 2024-08-02T23:39:03.691 INFO:teuthology.orchestra.run.smithi131.stdout:(9/13): python3-certifi-2023.05.07-4.el9.noarch 89 kB/s | 14 kB 00:00 2024-08-02T23:39:03.716 INFO:teuthology.orchestra.run.smithi131.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 2.3 MB/s | 59 kB 00:00 2024-08-02T23:39:03.742 INFO:teuthology.orchestra.run.smithi131.stdout:(11/13): python3-google-auth-2.32.0-1.el9.noarc 1.6 MB/s | 216 kB 00:00 2024-08-02T23:39:03.767 INFO:teuthology.orchestra.run.smithi131.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.7 MB/s | 90 kB 00:00 2024-08-02T23:39:03.809 INFO:teuthology.orchestra.run.smithi131.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 7.3 MB/s | 1.0 MB 00:00 2024-08-02T23:39:03.811 INFO:teuthology.orchestra.run.smithi131.stdout:-------------------------------------------------------------------------------- 2024-08-02T23:39:03.811 INFO:teuthology.orchestra.run.smithi131.stdout:Total 2.4 MB/s | 2.2 MB 00:00 2024-08-02T23:39:03.873 INFO:teuthology.orchestra.run.smithi131.stdout:Running transaction check 2024-08-02T23:39:03.887 INFO:teuthology.orchestra.run.smithi131.stdout:Transaction check succeeded. 2024-08-02T23:39:03.887 INFO:teuthology.orchestra.run.smithi131.stdout:Running transaction test 2024-08-02T23:39:03.972 INFO:teuthology.orchestra.run.smithi131.stdout:Transaction test succeeded. 2024-08-02T23:39:03.973 INFO:teuthology.orchestra.run.smithi131.stdout:Running transaction 2024-08-02T23:39:04.189 INFO:teuthology.orchestra.run.smithi073.stdout: Preparing : 1/1 2024-08-02T23:39:04.343 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-08-02T23:39:04.398 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-08-02T23:39:04.453 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-08-02T23:39:04.491 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-08-02T23:39:04.508 INFO:teuthology.orchestra.run.smithi131.stdout: Preparing : 1/1 2024-08-02T23:39:04.532 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-08-02T23:39:04.637 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-08-02T23:39:04.683 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-08-02T23:39:04.736 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-08-02T23:39:04.742 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-google-auth-1:2.32.0-1.el9.noarch 7/13 2024-08-02T23:39:04.792 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-08-02T23:39:04.799 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-08-02T23:39:04.836 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-08-02T23:39:04.878 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-08-02T23:39:04.976 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-08-02T23:39:05.082 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-google-auth-1:2.32.0-1.el9.noarch 7/13 2024-08-02T23:39:05.147 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-08-02T23:39:05.500 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-08-02T23:39:05.548 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-08-02T23:39:05.598 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-08-02T23:39:05.646 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-08-02T23:39:05.668 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : ceph-mgr-rook-2:19.1.0-1260.g26c3fb8e.el9.noarch 13/13 2024-08-02T23:39:05.882 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-08-02T23:39:05.944 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-08-02T23:39:05.977 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-08-02T23:39:06.026 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-08-02T23:39:06.047 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : ceph-mgr-rook-2:19.1.0-1260.g26c3fb8e.el9.noarch 13/13 2024-08-02T23:39:06.449 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: ceph-mgr-rook-2:19.1.0-1260.g26c3fb8e.el9.noarch 13/13 2024-08-02T23:39:06.449 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : ceph-mgr-rook-2:19.1.0-1260.g26c3fb8e.el9.noarch 1/13 2024-08-02T23:39:06.449 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-08-02T23:39:06.449 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-08-02T23:39:06.449 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-08-02T23:39:06.449 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-08-02T23:39:06.449 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-08-02T23:39:06.449 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-08-02T23:39:06.449 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-08-02T23:39:06.449 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-08-02T23:39:06.450 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-google-auth-1:2.32.0-1.el9.noarch 10/13 2024-08-02T23:39:06.450 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-08-02T23:39:06.450 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-08-02T23:39:06.777 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-08-02T23:39:06.777 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-08-02T23:39:06.777 INFO:teuthology.orchestra.run.smithi073.stdout:Installed: 2024-08-02T23:39:06.777 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-mgr-rook-2:19.1.0-1260.g26c3fb8e.el9.noarch 2024-08-02T23:39:06.777 INFO:teuthology.orchestra.run.smithi073.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-08-02T23:39:06.777 INFO:teuthology.orchestra.run.smithi073.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-08-02T23:39:06.777 INFO:teuthology.orchestra.run.smithi073.stdout: python3-google-auth-1:2.32.0-1.el9.noarch 2024-08-02T23:39:06.778 INFO:teuthology.orchestra.run.smithi073.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-08-02T23:39:06.778 INFO:teuthology.orchestra.run.smithi073.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-08-02T23:39:06.778 INFO:teuthology.orchestra.run.smithi073.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-08-02T23:39:06.778 INFO:teuthology.orchestra.run.smithi073.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-08-02T23:39:06.778 INFO:teuthology.orchestra.run.smithi073.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-08-02T23:39:06.778 INFO:teuthology.orchestra.run.smithi073.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-08-02T23:39:06.778 INFO:teuthology.orchestra.run.smithi073.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-08-02T23:39:06.778 INFO:teuthology.orchestra.run.smithi073.stdout: python3-rsa-4.9-2.el9.noarch 2024-08-02T23:39:06.778 INFO:teuthology.orchestra.run.smithi073.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-08-02T23:39:06.778 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-08-02T23:39:06.778 INFO:teuthology.orchestra.run.smithi073.stdout:Complete! 2024-08-02T23:39:06.899 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: ceph-mgr-rook-2:19.1.0-1260.g26c3fb8e.el9.noarch 13/13 2024-08-02T23:39:06.900 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : ceph-mgr-rook-2:19.1.0-1260.g26c3fb8e.el9.noarch 1/13 2024-08-02T23:39:06.900 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-08-02T23:39:06.900 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-08-02T23:39:06.900 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-08-02T23:39:06.900 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-08-02T23:39:06.900 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-08-02T23:39:06.900 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-08-02T23:39:06.900 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-08-02T23:39:06.900 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-08-02T23:39:06.900 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-google-auth-1:2.32.0-1.el9.noarch 10/13 2024-08-02T23:39:06.900 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-08-02T23:39:06.900 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-08-02T23:39:07.023 DEBUG:teuthology.orchestra.run.smithi073:> sudo yum -y install ceph-mgr-cephadm 2024-08-02T23:39:07.253 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-08-02T23:39:07.253 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-08-02T23:39:07.253 INFO:teuthology.orchestra.run.smithi131.stdout:Installed: 2024-08-02T23:39:07.253 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-mgr-rook-2:19.1.0-1260.g26c3fb8e.el9.noarch 2024-08-02T23:39:07.253 INFO:teuthology.orchestra.run.smithi131.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-08-02T23:39:07.254 INFO:teuthology.orchestra.run.smithi131.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-08-02T23:39:07.254 INFO:teuthology.orchestra.run.smithi131.stdout: python3-google-auth-1:2.32.0-1.el9.noarch 2024-08-02T23:39:07.254 INFO:teuthology.orchestra.run.smithi131.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-08-02T23:39:07.254 INFO:teuthology.orchestra.run.smithi131.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-08-02T23:39:07.254 INFO:teuthology.orchestra.run.smithi131.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-08-02T23:39:07.254 INFO:teuthology.orchestra.run.smithi131.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-08-02T23:39:07.254 INFO:teuthology.orchestra.run.smithi131.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-08-02T23:39:07.254 INFO:teuthology.orchestra.run.smithi131.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-08-02T23:39:07.254 INFO:teuthology.orchestra.run.smithi131.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-08-02T23:39:07.254 INFO:teuthology.orchestra.run.smithi131.stdout: python3-rsa-4.9-2.el9.noarch 2024-08-02T23:39:07.254 INFO:teuthology.orchestra.run.smithi131.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-08-02T23:39:07.254 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-08-02T23:39:07.254 INFO:teuthology.orchestra.run.smithi131.stdout:Complete! 2024-08-02T23:39:07.507 DEBUG:teuthology.orchestra.run.smithi131:> sudo yum -y install ceph-mgr-cephadm 2024-08-02T23:39:07.563 INFO:teuthology.orchestra.run.smithi073.stdout:Last metadata expiration check: 0:02:13 ago on Fri 02 Aug 2024 11:36:54 PM UTC. 2024-08-02T23:39:07.689 INFO:teuthology.orchestra.run.smithi073.stdout:Dependencies resolved. 2024-08-02T23:39:07.690 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2024-08-02T23:39:07.690 INFO:teuthology.orchestra.run.smithi073.stdout: Package Arch Version Repository Size 2024-08-02T23:39:07.690 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2024-08-02T23:39:07.690 INFO:teuthology.orchestra.run.smithi073.stdout:Installing: 2024-08-02T23:39:07.691 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-mgr-cephadm noarch 2:19.1.0-1260.g26c3fb8e.el9 ceph-noarch 146 k 2024-08-02T23:39:07.691 INFO:teuthology.orchestra.run.smithi073.stdout:Installing dependencies: 2024-08-02T23:39:07.691 INFO:teuthology.orchestra.run.smithi073.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-08-02T23:39:07.691 INFO:teuthology.orchestra.run.smithi073.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-08-02T23:39:07.691 INFO:teuthology.orchestra.run.smithi073.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-08-02T23:39:07.691 INFO:teuthology.orchestra.run.smithi073.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-08-02T23:39:07.691 INFO:teuthology.orchestra.run.smithi073.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-08-02T23:39:07.691 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-08-02T23:39:07.691 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction Summary 2024-08-02T23:39:07.691 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2024-08-02T23:39:07.691 INFO:teuthology.orchestra.run.smithi073.stdout:Install 6 Packages 2024-08-02T23:39:07.691 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-08-02T23:39:07.692 INFO:teuthology.orchestra.run.smithi073.stdout:Total download size: 7.0 M 2024-08-02T23:39:07.692 INFO:teuthology.orchestra.run.smithi073.stdout:Installed size: 33 M 2024-08-02T23:39:07.692 INFO:teuthology.orchestra.run.smithi073.stdout:Downloading Packages: 2024-08-02T23:39:08.024 INFO:teuthology.orchestra.run.smithi073.stdout:(1/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.9 MB/s | 248 kB 00:00 2024-08-02T23:39:08.054 INFO:teuthology.orchestra.run.smithi131.stdout:Last metadata expiration check: 0:02:16 ago on Fri 02 Aug 2024 11:36:52 PM UTC. 2024-08-02T23:39:08.099 INFO:teuthology.orchestra.run.smithi073.stdout:(2/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 7.2 MB/s | 548 kB 00:00 2024-08-02T23:39:08.124 INFO:teuthology.orchestra.run.smithi073.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 2.3 MB/s | 58 kB 00:00 2024-08-02T23:39:08.150 INFO:teuthology.orchestra.run.smithi073.stdout:(4/6): ceph-mgr-cephadm-19.1.0-1260.g26c3fb8e.e 578 kB/s | 146 kB 00:00 2024-08-02T23:39:08.175 INFO:teuthology.orchestra.run.smithi073.stdout:(5/6): python3-typing-extensions-4.12.2-2.el9.n 1.4 MB/s | 74 kB 00:00 2024-08-02T23:39:08.185 INFO:teuthology.orchestra.run.smithi131.stdout:Dependencies resolved. 2024-08-02T23:39:08.186 INFO:teuthology.orchestra.run.smithi131.stdout:================================================================================ 2024-08-02T23:39:08.186 INFO:teuthology.orchestra.run.smithi131.stdout: Package Arch Version Repository Size 2024-08-02T23:39:08.187 INFO:teuthology.orchestra.run.smithi131.stdout:================================================================================ 2024-08-02T23:39:08.187 INFO:teuthology.orchestra.run.smithi131.stdout:Installing: 2024-08-02T23:39:08.187 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-mgr-cephadm noarch 2:19.1.0-1260.g26c3fb8e.el9 ceph-noarch 146 k 2024-08-02T23:39:08.187 INFO:teuthology.orchestra.run.smithi131.stdout:Installing dependencies: 2024-08-02T23:39:08.187 INFO:teuthology.orchestra.run.smithi131.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-08-02T23:39:08.187 INFO:teuthology.orchestra.run.smithi131.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-08-02T23:39:08.187 INFO:teuthology.orchestra.run.smithi131.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-08-02T23:39:08.187 INFO:teuthology.orchestra.run.smithi131.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-08-02T23:39:08.187 INFO:teuthology.orchestra.run.smithi131.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-08-02T23:39:08.187 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-08-02T23:39:08.187 INFO:teuthology.orchestra.run.smithi131.stdout:Transaction Summary 2024-08-02T23:39:08.187 INFO:teuthology.orchestra.run.smithi131.stdout:================================================================================ 2024-08-02T23:39:08.187 INFO:teuthology.orchestra.run.smithi131.stdout:Install 6 Packages 2024-08-02T23:39:08.188 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-08-02T23:39:08.188 INFO:teuthology.orchestra.run.smithi131.stdout:Total download size: 7.0 M 2024-08-02T23:39:08.188 INFO:teuthology.orchestra.run.smithi131.stdout:Installed size: 33 M 2024-08-02T23:39:08.188 INFO:teuthology.orchestra.run.smithi131.stdout:Downloading Packages: 2024-08-02T23:39:08.545 INFO:teuthology.orchestra.run.smithi131.stdout:(1/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.5 MB/s | 248 kB 00:00 2024-08-02T23:39:08.612 INFO:teuthology.orchestra.run.smithi131.stdout:(2/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 8.1 MB/s | 548 kB 00:00 2024-08-02T23:39:08.638 INFO:teuthology.orchestra.run.smithi131.stdout:(3/6): ceph-mgr-cephadm-19.1.0-1260.g26c3fb8e.e 568 kB/s | 146 kB 00:00 2024-08-02T23:39:08.654 INFO:teuthology.orchestra.run.smithi131.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 1.3 MB/s | 58 kB 00:00 2024-08-02T23:39:08.713 INFO:teuthology.orchestra.run.smithi131.stdout:(5/6): python3-typing-extensions-4.12.2-2.el9.n 989 kB/s | 74 kB 00:00 2024-08-02T23:39:08.904 INFO:teuthology.orchestra.run.smithi131.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 15 MB/s | 6.0 MB 00:00 2024-08-02T23:39:08.904 INFO:teuthology.orchestra.run.smithi131.stdout:-------------------------------------------------------------------------------- 2024-08-02T23:39:08.904 INFO:teuthology.orchestra.run.smithi131.stdout:Total 12 MB/s | 7.0 MB 00:00 2024-08-02T23:39:08.904 INFO:teuthology.orchestra.run.smithi131.stdout:Running transaction check 2024-08-02T23:39:08.904 INFO:teuthology.orchestra.run.smithi131.stdout:Transaction check succeeded. 2024-08-02T23:39:08.905 INFO:teuthology.orchestra.run.smithi131.stdout:Running transaction test 2024-08-02T23:39:08.970 INFO:teuthology.orchestra.run.smithi131.stdout:Transaction test succeeded. 2024-08-02T23:39:08.970 INFO:teuthology.orchestra.run.smithi131.stdout:Running transaction 2024-08-02T23:39:09.278 INFO:teuthology.orchestra.run.smithi131.stdout: Preparing : 1/1 2024-08-02T23:39:09.462 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-08-02T23:39:09.467 INFO:teuthology.orchestra.run.smithi073.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 3.8 MB/s | 6.0 MB 00:01 2024-08-02T23:39:09.469 INFO:teuthology.orchestra.run.smithi073.stdout:-------------------------------------------------------------------------------- 2024-08-02T23:39:09.469 INFO:teuthology.orchestra.run.smithi073.stdout:Total 3.9 MB/s | 7.0 MB 00:01 2024-08-02T23:39:09.516 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-08-02T23:39:09.543 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction check 2024-08-02T23:39:09.557 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction check succeeded. 2024-08-02T23:39:09.557 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction test 2024-08-02T23:39:09.661 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction test succeeded. 2024-08-02T23:39:09.661 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction 2024-08-02T23:39:09.969 INFO:teuthology.orchestra.run.smithi073.stdout: Preparing : 1/1 2024-08-02T23:39:10.039 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-08-02T23:39:10.114 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-08-02T23:39:10.158 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-08-02T23:39:10.178 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-08-02T23:39:10.202 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-08-02T23:39:10.208 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : ceph-mgr-cephadm-2:19.1.0-1260.g26c3fb8e.el9.noarch 6/6 2024-08-02T23:39:10.715 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-08-02T23:39:10.807 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-08-02T23:39:10.876 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-08-02T23:39:10.906 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : ceph-mgr-cephadm-2:19.1.0-1260.g26c3fb8e.el9.noarch 6/6 2024-08-02T23:39:10.966 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.1.0-1260.g26c3fb8e.el9.noarch 6/6 2024-08-02T23:39:10.966 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : ceph-mgr-cephadm-2:19.1.0-1260.g26c3fb8e.el9.noarch 1/6 2024-08-02T23:39:10.966 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-08-02T23:39:10.966 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-08-02T23:39:10.967 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-08-02T23:39:10.967 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-08-02T23:39:11.226 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-08-02T23:39:11.227 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-08-02T23:39:11.227 INFO:teuthology.orchestra.run.smithi131.stdout:Installed: 2024-08-02T23:39:11.227 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-mgr-cephadm-2:19.1.0-1260.g26c3fb8e.el9.noarch 2024-08-02T23:39:11.227 INFO:teuthology.orchestra.run.smithi131.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-08-02T23:39:11.227 INFO:teuthology.orchestra.run.smithi131.stdout: python3-babel-2.9.1-2.el9.noarch 2024-08-02T23:39:11.227 INFO:teuthology.orchestra.run.smithi131.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-08-02T23:39:11.227 INFO:teuthology.orchestra.run.smithi131.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-08-02T23:39:11.227 INFO:teuthology.orchestra.run.smithi131.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-08-02T23:39:11.227 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-08-02T23:39:11.227 INFO:teuthology.orchestra.run.smithi131.stdout:Complete! 2024-08-02T23:39:11.466 DEBUG:teuthology.orchestra.run.smithi131:> sudo yum -y install ceph-fuse 2024-08-02T23:39:11.654 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.1.0-1260.g26c3fb8e.el9.noarch 6/6 2024-08-02T23:39:11.654 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : ceph-mgr-cephadm-2:19.1.0-1260.g26c3fb8e.el9.noarch 1/6 2024-08-02T23:39:11.654 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-08-02T23:39:11.654 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-08-02T23:39:11.654 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-08-02T23:39:11.654 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-08-02T23:39:11.915 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-08-02T23:39:11.916 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-08-02T23:39:11.916 INFO:teuthology.orchestra.run.smithi073.stdout:Installed: 2024-08-02T23:39:11.916 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-mgr-cephadm-2:19.1.0-1260.g26c3fb8e.el9.noarch 2024-08-02T23:39:11.916 INFO:teuthology.orchestra.run.smithi073.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-08-02T23:39:11.916 INFO:teuthology.orchestra.run.smithi073.stdout: python3-babel-2.9.1-2.el9.noarch 2024-08-02T23:39:11.916 INFO:teuthology.orchestra.run.smithi073.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-08-02T23:39:11.916 INFO:teuthology.orchestra.run.smithi073.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-08-02T23:39:11.916 INFO:teuthology.orchestra.run.smithi073.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-08-02T23:39:11.916 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-08-02T23:39:11.916 INFO:teuthology.orchestra.run.smithi073.stdout:Complete! 2024-08-02T23:39:12.016 INFO:teuthology.orchestra.run.smithi131.stdout:Last metadata expiration check: 0:02:20 ago on Fri 02 Aug 2024 11:36:52 PM UTC. 2024-08-02T23:39:12.146 INFO:teuthology.orchestra.run.smithi131.stdout:Dependencies resolved. 2024-08-02T23:39:12.147 INFO:teuthology.orchestra.run.smithi131.stdout:================================================================================ 2024-08-02T23:39:12.147 INFO:teuthology.orchestra.run.smithi131.stdout: Package Arch Version Repository Size 2024-08-02T23:39:12.147 INFO:teuthology.orchestra.run.smithi131.stdout:================================================================================ 2024-08-02T23:39:12.147 INFO:teuthology.orchestra.run.smithi131.stdout:Installing: 2024-08-02T23:39:12.147 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-fuse x86_64 2:19.1.0-1260.g26c3fb8e.el9 ceph 867 k 2024-08-02T23:39:12.147 INFO:teuthology.orchestra.run.smithi131.stdout:Installing dependencies: 2024-08-02T23:39:12.147 INFO:teuthology.orchestra.run.smithi131.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-08-02T23:39:12.147 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-08-02T23:39:12.147 INFO:teuthology.orchestra.run.smithi131.stdout:Transaction Summary 2024-08-02T23:39:12.147 INFO:teuthology.orchestra.run.smithi131.stdout:================================================================================ 2024-08-02T23:39:12.147 INFO:teuthology.orchestra.run.smithi131.stdout:Install 2 Packages 2024-08-02T23:39:12.147 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-08-02T23:39:12.148 INFO:teuthology.orchestra.run.smithi131.stdout:Total download size: 947 k 2024-08-02T23:39:12.148 INFO:teuthology.orchestra.run.smithi131.stdout:Installed size: 3.1 M 2024-08-02T23:39:12.148 INFO:teuthology.orchestra.run.smithi131.stdout:Downloading Packages: 2024-08-02T23:39:12.180 DEBUG:teuthology.orchestra.run.smithi073:> sudo yum -y install ceph-fuse 2024-08-02T23:39:12.580 INFO:teuthology.orchestra.run.smithi131.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 500 kB/s | 80 kB 00:00 2024-08-02T23:39:12.724 INFO:teuthology.orchestra.run.smithi073.stdout:Last metadata expiration check: 0:02:18 ago on Fri 02 Aug 2024 11:36:54 PM UTC. 2024-08-02T23:39:12.854 INFO:teuthology.orchestra.run.smithi073.stdout:Dependencies resolved. 2024-08-02T23:39:12.854 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2024-08-02T23:39:12.854 INFO:teuthology.orchestra.run.smithi073.stdout: Package Arch Version Repository Size 2024-08-02T23:39:12.854 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2024-08-02T23:39:12.854 INFO:teuthology.orchestra.run.smithi073.stdout:Installing: 2024-08-02T23:39:12.854 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-fuse x86_64 2:19.1.0-1260.g26c3fb8e.el9 ceph 867 k 2024-08-02T23:39:12.854 INFO:teuthology.orchestra.run.smithi073.stdout:Installing dependencies: 2024-08-02T23:39:12.854 INFO:teuthology.orchestra.run.smithi073.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-08-02T23:39:12.854 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-08-02T23:39:12.855 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction Summary 2024-08-02T23:39:12.855 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2024-08-02T23:39:12.855 INFO:teuthology.orchestra.run.smithi073.stdout:Install 2 Packages 2024-08-02T23:39:12.855 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-08-02T23:39:12.855 INFO:teuthology.orchestra.run.smithi073.stdout:Total download size: 947 k 2024-08-02T23:39:12.855 INFO:teuthology.orchestra.run.smithi073.stdout:Installed size: 3.1 M 2024-08-02T23:39:12.855 INFO:teuthology.orchestra.run.smithi073.stdout:Downloading Packages: 2024-08-02T23:39:13.113 INFO:teuthology.orchestra.run.smithi073.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 552 kB/s | 80 kB 00:00 2024-08-02T23:39:13.163 INFO:teuthology.orchestra.run.smithi131.stdout:(2/2): ceph-fuse-19.1.0-1260.g26c3fb8e.el9.x86_ 1.1 MB/s | 867 kB 00:00 2024-08-02T23:39:13.164 INFO:teuthology.orchestra.run.smithi131.stdout:-------------------------------------------------------------------------------- 2024-08-02T23:39:13.164 INFO:teuthology.orchestra.run.smithi131.stdout:Total 931 kB/s | 947 kB 00:01 2024-08-02T23:39:13.176 INFO:teuthology.orchestra.run.smithi131.stdout:Running transaction check 2024-08-02T23:39:13.199 INFO:teuthology.orchestra.run.smithi131.stdout:Transaction check succeeded. 2024-08-02T23:39:13.199 INFO:teuthology.orchestra.run.smithi131.stdout:Running transaction test 2024-08-02T23:39:13.278 INFO:teuthology.orchestra.run.smithi131.stdout:Transaction test succeeded. 2024-08-02T23:39:13.278 INFO:teuthology.orchestra.run.smithi131.stdout:Running transaction 2024-08-02T23:39:13.396 INFO:teuthology.orchestra.run.smithi073.stdout:(2/2): ceph-fuse-19.1.0-1260.g26c3fb8e.el9.x86_ 2.0 MB/s | 867 kB 00:00 2024-08-02T23:39:13.397 INFO:teuthology.orchestra.run.smithi073.stdout:-------------------------------------------------------------------------------- 2024-08-02T23:39:13.397 INFO:teuthology.orchestra.run.smithi073.stdout:Total 1.7 MB/s | 947 kB 00:00 2024-08-02T23:39:13.409 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction check 2024-08-02T23:39:13.431 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction check succeeded. 2024-08-02T23:39:13.431 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction test 2024-08-02T23:39:13.462 INFO:teuthology.orchestra.run.smithi131.stdout: Preparing : 1/1 2024-08-02T23:39:13.509 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction test succeeded. 2024-08-02T23:39:13.510 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction 2024-08-02T23:39:13.599 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-08-02T23:39:13.673 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : ceph-fuse-2:19.1.0-1260.g26c3fb8e.el9.x86_64 2/2 2024-08-02T23:39:13.696 INFO:teuthology.orchestra.run.smithi073.stdout: Preparing : 1/1 2024-08-02T23:39:13.842 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-08-02T23:39:13.906 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : ceph-fuse-2:19.1.0-1260.g26c3fb8e.el9.x86_64 2/2 2024-08-02T23:39:14.382 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: ceph-fuse-2:19.1.0-1260.g26c3fb8e.el9.x86_64 2/2 2024-08-02T23:39:14.382 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : ceph-fuse-2:19.1.0-1260.g26c3fb8e.el9.x86_64 1/2 2024-08-02T23:39:14.596 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: ceph-fuse-2:19.1.0-1260.g26c3fb8e.el9.x86_64 2/2 2024-08-02T23:39:14.596 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : ceph-fuse-2:19.1.0-1260.g26c3fb8e.el9.x86_64 1/2 2024-08-02T23:39:14.660 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-08-02T23:39:14.660 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-08-02T23:39:14.660 INFO:teuthology.orchestra.run.smithi131.stdout:Installed: 2024-08-02T23:39:14.660 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-fuse-2:19.1.0-1260.g26c3fb8e.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-08-02T23:39:14.660 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-08-02T23:39:14.660 INFO:teuthology.orchestra.run.smithi131.stdout:Complete! 2024-08-02T23:39:14.877 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-08-02T23:39:14.877 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-08-02T23:39:14.877 INFO:teuthology.orchestra.run.smithi073.stdout:Installed: 2024-08-02T23:39:14.877 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-fuse-2:19.1.0-1260.g26c3fb8e.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-08-02T23:39:14.877 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-08-02T23:39:14.877 INFO:teuthology.orchestra.run.smithi073.stdout:Complete! 2024-08-02T23:39:14.945 DEBUG:teuthology.orchestra.run.smithi131:> sudo yum -y install ceph-volume 2024-08-02T23:39:15.151 DEBUG:teuthology.orchestra.run.smithi073:> sudo yum -y install ceph-volume 2024-08-02T23:39:15.510 INFO:teuthology.orchestra.run.smithi131.stdout:Last metadata expiration check: 0:02:23 ago on Fri 02 Aug 2024 11:36:52 PM UTC. 2024-08-02T23:39:15.643 INFO:teuthology.orchestra.run.smithi131.stdout:Dependencies resolved. 2024-08-02T23:39:15.644 INFO:teuthology.orchestra.run.smithi131.stdout:================================================================================ 2024-08-02T23:39:15.644 INFO:teuthology.orchestra.run.smithi131.stdout: Package Arch Version Repository Size 2024-08-02T23:39:15.644 INFO:teuthology.orchestra.run.smithi131.stdout:================================================================================ 2024-08-02T23:39:15.644 INFO:teuthology.orchestra.run.smithi131.stdout:Installing: 2024-08-02T23:39:15.644 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-volume noarch 2:19.1.0-1260.g26c3fb8e.el9 ceph-noarch 274 k 2024-08-02T23:39:15.644 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-08-02T23:39:15.644 INFO:teuthology.orchestra.run.smithi131.stdout:Transaction Summary 2024-08-02T23:39:15.644 INFO:teuthology.orchestra.run.smithi131.stdout:================================================================================ 2024-08-02T23:39:15.644 INFO:teuthology.orchestra.run.smithi131.stdout:Install 1 Package 2024-08-02T23:39:15.644 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-08-02T23:39:15.644 INFO:teuthology.orchestra.run.smithi131.stdout:Total download size: 274 k 2024-08-02T23:39:15.644 INFO:teuthology.orchestra.run.smithi131.stdout:Installed size: 1.3 M 2024-08-02T23:39:15.645 INFO:teuthology.orchestra.run.smithi131.stdout:Downloading Packages: 2024-08-02T23:39:15.703 INFO:teuthology.orchestra.run.smithi073.stdout:Last metadata expiration check: 0:02:21 ago on Fri 02 Aug 2024 11:36:54 PM UTC. 2024-08-02T23:39:15.836 INFO:teuthology.orchestra.run.smithi073.stdout:Dependencies resolved. 2024-08-02T23:39:15.837 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2024-08-02T23:39:15.837 INFO:teuthology.orchestra.run.smithi073.stdout: Package Arch Version Repository Size 2024-08-02T23:39:15.837 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2024-08-02T23:39:15.837 INFO:teuthology.orchestra.run.smithi073.stdout:Installing: 2024-08-02T23:39:15.837 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-volume noarch 2:19.1.0-1260.g26c3fb8e.el9 ceph-noarch 274 k 2024-08-02T23:39:15.837 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-08-02T23:39:15.837 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction Summary 2024-08-02T23:39:15.837 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2024-08-02T23:39:15.838 INFO:teuthology.orchestra.run.smithi073.stdout:Install 1 Package 2024-08-02T23:39:15.838 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-08-02T23:39:15.838 INFO:teuthology.orchestra.run.smithi073.stdout:Total download size: 274 k 2024-08-02T23:39:15.838 INFO:teuthology.orchestra.run.smithi073.stdout:Installed size: 1.3 M 2024-08-02T23:39:15.838 INFO:teuthology.orchestra.run.smithi073.stdout:Downloading Packages: 2024-08-02T23:39:15.903 INFO:teuthology.orchestra.run.smithi131.stdout:ceph-volume-19.1.0-1260.g26c3fb8e.el9.noarch.rp 1.0 MB/s | 274 kB 00:00 2024-08-02T23:39:15.903 INFO:teuthology.orchestra.run.smithi131.stdout:-------------------------------------------------------------------------------- 2024-08-02T23:39:15.903 INFO:teuthology.orchestra.run.smithi131.stdout:Total 1.0 MB/s | 274 kB 00:00 2024-08-02T23:39:15.904 INFO:teuthology.orchestra.run.smithi131.stdout:Running transaction check 2024-08-02T23:39:15.912 INFO:teuthology.orchestra.run.smithi131.stdout:Transaction check succeeded. 2024-08-02T23:39:15.912 INFO:teuthology.orchestra.run.smithi131.stdout:Running transaction test 2024-08-02T23:39:15.940 INFO:teuthology.orchestra.run.smithi131.stdout:Transaction test succeeded. 2024-08-02T23:39:15.940 INFO:teuthology.orchestra.run.smithi131.stdout:Running transaction 2024-08-02T23:39:16.080 INFO:teuthology.orchestra.run.smithi073.stdout:ceph-volume-19.1.0-1260.g26c3fb8e.el9.noarch.rp 1.1 MB/s | 274 kB 00:00 2024-08-02T23:39:16.080 INFO:teuthology.orchestra.run.smithi073.stdout:-------------------------------------------------------------------------------- 2024-08-02T23:39:16.080 INFO:teuthology.orchestra.run.smithi073.stdout:Total 1.1 MB/s | 274 kB 00:00 2024-08-02T23:39:16.080 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction check 2024-08-02T23:39:16.089 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction check succeeded. 2024-08-02T23:39:16.089 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction test 2024-08-02T23:39:16.116 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction test succeeded. 2024-08-02T23:39:16.116 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction 2024-08-02T23:39:16.154 INFO:teuthology.orchestra.run.smithi131.stdout: Preparing : 1/1 2024-08-02T23:39:16.221 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : ceph-volume-2:19.1.0-1260.g26c3fb8e.el9.noarch 1/1 2024-08-02T23:39:16.237 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: ceph-volume-2:19.1.0-1260.g26c3fb8e.el9.noarch 1/1 2024-08-02T23:39:16.237 INFO:teuthology.orchestra.run.smithi131.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-02T23:39:16.237 INFO:teuthology.orchestra.run.smithi131.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-08-02T23:39:16.237 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-08-02T23:39:16.335 INFO:teuthology.orchestra.run.smithi073.stdout: Preparing : 1/1 2024-08-02T23:39:16.410 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : ceph-volume-2:19.1.0-1260.g26c3fb8e.el9.noarch 1/1 2024-08-02T23:39:16.426 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: ceph-volume-2:19.1.0-1260.g26c3fb8e.el9.noarch 1/1 2024-08-02T23:39:16.427 INFO:teuthology.orchestra.run.smithi073.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-02T23:39:16.427 INFO:teuthology.orchestra.run.smithi073.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-08-02T23:39:16.427 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-08-02T23:39:17.241 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : ceph-volume-2:19.1.0-1260.g26c3fb8e.el9.noarch 1/1 2024-08-02T23:39:17.241 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-08-02T23:39:17.242 INFO:teuthology.orchestra.run.smithi131.stdout:Installed: 2024-08-02T23:39:17.242 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-volume-2:19.1.0-1260.g26c3fb8e.el9.noarch 2024-08-02T23:39:17.242 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-08-02T23:39:17.242 INFO:teuthology.orchestra.run.smithi131.stdout:Complete! 2024-08-02T23:39:17.490 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : ceph-volume-2:19.1.0-1260.g26c3fb8e.el9.noarch 1/1 2024-08-02T23:39:17.490 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-08-02T23:39:17.490 INFO:teuthology.orchestra.run.smithi073.stdout:Installed: 2024-08-02T23:39:17.490 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-volume-2:19.1.0-1260.g26c3fb8e.el9.noarch 2024-08-02T23:39:17.490 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-08-02T23:39:17.490 INFO:teuthology.orchestra.run.smithi073.stdout:Complete! 2024-08-02T23:39:17.519 DEBUG:teuthology.orchestra.run.smithi131:> sudo yum -y install librados-devel 2024-08-02T23:39:17.799 DEBUG:teuthology.orchestra.run.smithi073:> sudo yum -y install librados-devel 2024-08-02T23:39:18.081 INFO:teuthology.orchestra.run.smithi131.stdout:Last metadata expiration check: 0:02:26 ago on Fri 02 Aug 2024 11:36:52 PM UTC. 2024-08-02T23:39:18.214 INFO:teuthology.orchestra.run.smithi131.stdout:Dependencies resolved. 2024-08-02T23:39:18.215 INFO:teuthology.orchestra.run.smithi131.stdout:================================================================================ 2024-08-02T23:39:18.215 INFO:teuthology.orchestra.run.smithi131.stdout: Package Arch Version Repo Size 2024-08-02T23:39:18.215 INFO:teuthology.orchestra.run.smithi131.stdout:================================================================================ 2024-08-02T23:39:18.215 INFO:teuthology.orchestra.run.smithi131.stdout:Installing: 2024-08-02T23:39:18.215 INFO:teuthology.orchestra.run.smithi131.stdout: librados-devel x86_64 2:19.1.0-1260.g26c3fb8e.el9 ceph 127 k 2024-08-02T23:39:18.215 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-08-02T23:39:18.215 INFO:teuthology.orchestra.run.smithi131.stdout:Transaction Summary 2024-08-02T23:39:18.215 INFO:teuthology.orchestra.run.smithi131.stdout:================================================================================ 2024-08-02T23:39:18.215 INFO:teuthology.orchestra.run.smithi131.stdout:Install 1 Package 2024-08-02T23:39:18.215 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-08-02T23:39:18.215 INFO:teuthology.orchestra.run.smithi131.stdout:Total download size: 127 k 2024-08-02T23:39:18.215 INFO:teuthology.orchestra.run.smithi131.stdout:Installed size: 457 k 2024-08-02T23:39:18.215 INFO:teuthology.orchestra.run.smithi131.stdout:Downloading Packages: 2024-08-02T23:39:18.362 INFO:teuthology.orchestra.run.smithi073.stdout:Last metadata expiration check: 0:02:24 ago on Fri 02 Aug 2024 11:36:54 PM UTC. 2024-08-02T23:39:18.429 INFO:teuthology.orchestra.run.smithi131.stdout:librados-devel-19.1.0-1260.g26c3fb8e.el9.x86_64 596 kB/s | 127 kB 00:00 2024-08-02T23:39:18.429 INFO:teuthology.orchestra.run.smithi131.stdout:-------------------------------------------------------------------------------- 2024-08-02T23:39:18.429 INFO:teuthology.orchestra.run.smithi131.stdout:Total 592 kB/s | 127 kB 00:00 2024-08-02T23:39:18.429 INFO:teuthology.orchestra.run.smithi131.stdout:Running transaction check 2024-08-02T23:39:18.435 INFO:teuthology.orchestra.run.smithi131.stdout:Transaction check succeeded. 2024-08-02T23:39:18.435 INFO:teuthology.orchestra.run.smithi131.stdout:Running transaction test 2024-08-02T23:39:18.493 INFO:teuthology.orchestra.run.smithi073.stdout:Dependencies resolved. 2024-08-02T23:39:18.494 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2024-08-02T23:39:18.494 INFO:teuthology.orchestra.run.smithi073.stdout: Package Arch Version Repo Size 2024-08-02T23:39:18.494 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2024-08-02T23:39:18.494 INFO:teuthology.orchestra.run.smithi073.stdout:Installing: 2024-08-02T23:39:18.494 INFO:teuthology.orchestra.run.smithi073.stdout: librados-devel x86_64 2:19.1.0-1260.g26c3fb8e.el9 ceph 127 k 2024-08-02T23:39:18.494 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-08-02T23:39:18.494 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction Summary 2024-08-02T23:39:18.494 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2024-08-02T23:39:18.494 INFO:teuthology.orchestra.run.smithi073.stdout:Install 1 Package 2024-08-02T23:39:18.494 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-08-02T23:39:18.495 INFO:teuthology.orchestra.run.smithi073.stdout:Total download size: 127 k 2024-08-02T23:39:18.495 INFO:teuthology.orchestra.run.smithi073.stdout:Installed size: 457 k 2024-08-02T23:39:18.495 INFO:teuthology.orchestra.run.smithi073.stdout:Downloading Packages: 2024-08-02T23:39:18.498 INFO:teuthology.orchestra.run.smithi131.stdout:Transaction test succeeded. 2024-08-02T23:39:18.498 INFO:teuthology.orchestra.run.smithi131.stdout:Running transaction 2024-08-02T23:39:18.646 INFO:teuthology.orchestra.run.smithi131.stdout: Preparing : 1/1 2024-08-02T23:39:18.697 INFO:teuthology.orchestra.run.smithi073.stdout:librados-devel-19.1.0-1260.g26c3fb8e.el9.x86_64 626 kB/s | 127 kB 00:00 2024-08-02T23:39:18.698 INFO:teuthology.orchestra.run.smithi073.stdout:-------------------------------------------------------------------------------- 2024-08-02T23:39:18.698 INFO:teuthology.orchestra.run.smithi073.stdout:Total 622 kB/s | 127 kB 00:00 2024-08-02T23:39:18.698 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction check 2024-08-02T23:39:18.704 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction check succeeded. 2024-08-02T23:39:18.704 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction test 2024-08-02T23:39:18.745 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : librados-devel-2:19.1.0-1260.g26c3fb8e.el9.x86_64 1/1 2024-08-02T23:39:18.767 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction test succeeded. 2024-08-02T23:39:18.767 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction 2024-08-02T23:39:18.912 INFO:teuthology.orchestra.run.smithi073.stdout: Preparing : 1/1 2024-08-02T23:39:19.012 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : librados-devel-2:19.1.0-1260.g26c3fb8e.el9.x86_64 1/1 2024-08-02T23:39:19.203 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: librados-devel-2:19.1.0-1260.g26c3fb8e.el9.x86_64 1/1 2024-08-02T23:39:19.467 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : librados-devel-2:19.1.0-1260.g26c3fb8e.el9.x86_64 1/1 2024-08-02T23:39:19.467 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-08-02T23:39:19.467 INFO:teuthology.orchestra.run.smithi131.stdout:Installed: 2024-08-02T23:39:19.467 INFO:teuthology.orchestra.run.smithi131.stdout: librados-devel-2:19.1.0-1260.g26c3fb8e.el9.x86_64 2024-08-02T23:39:19.467 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-08-02T23:39:19.467 INFO:teuthology.orchestra.run.smithi131.stdout:Complete! 2024-08-02T23:39:19.472 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: librados-devel-2:19.1.0-1260.g26c3fb8e.el9.x86_64 1/1 2024-08-02T23:39:19.715 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : librados-devel-2:19.1.0-1260.g26c3fb8e.el9.x86_64 1/1 2024-08-02T23:39:19.716 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-08-02T23:39:19.716 INFO:teuthology.orchestra.run.smithi073.stdout:Installed: 2024-08-02T23:39:19.716 INFO:teuthology.orchestra.run.smithi073.stdout: librados-devel-2:19.1.0-1260.g26c3fb8e.el9.x86_64 2024-08-02T23:39:19.716 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-08-02T23:39:19.716 INFO:teuthology.orchestra.run.smithi073.stdout:Complete! 2024-08-02T23:39:19.723 DEBUG:teuthology.orchestra.run.smithi131:> sudo yum -y install libcephfs2 2024-08-02T23:39:20.000 DEBUG:teuthology.orchestra.run.smithi073:> sudo yum -y install libcephfs2 2024-08-02T23:39:20.281 INFO:teuthology.orchestra.run.smithi131.stdout:Last metadata expiration check: 0:02:28 ago on Fri 02 Aug 2024 11:36:52 PM UTC. 2024-08-02T23:39:20.368 INFO:teuthology.orchestra.run.smithi131.stdout:Package libcephfs2-2:19.1.0-1260.g26c3fb8e.el9.x86_64 is already installed. 2024-08-02T23:39:20.414 INFO:teuthology.orchestra.run.smithi131.stdout:Dependencies resolved. 2024-08-02T23:39:20.415 INFO:teuthology.orchestra.run.smithi131.stdout:Nothing to do. 2024-08-02T23:39:20.415 INFO:teuthology.orchestra.run.smithi131.stdout:Complete! 2024-08-02T23:39:20.473 DEBUG:teuthology.orchestra.run.smithi131:> sudo yum -y install libcephfs-devel 2024-08-02T23:39:20.559 INFO:teuthology.orchestra.run.smithi073.stdout:Last metadata expiration check: 0:02:26 ago on Fri 02 Aug 2024 11:36:54 PM UTC. 2024-08-02T23:39:20.647 INFO:teuthology.orchestra.run.smithi073.stdout:Package libcephfs2-2:19.1.0-1260.g26c3fb8e.el9.x86_64 is already installed. 2024-08-02T23:39:20.691 INFO:teuthology.orchestra.run.smithi073.stdout:Dependencies resolved. 2024-08-02T23:39:20.692 INFO:teuthology.orchestra.run.smithi073.stdout:Nothing to do. 2024-08-02T23:39:20.692 INFO:teuthology.orchestra.run.smithi073.stdout:Complete! 2024-08-02T23:39:20.758 DEBUG:teuthology.orchestra.run.smithi073:> sudo yum -y install libcephfs-devel 2024-08-02T23:39:21.018 INFO:teuthology.orchestra.run.smithi131.stdout:Last metadata expiration check: 0:02:29 ago on Fri 02 Aug 2024 11:36:52 PM UTC. 2024-08-02T23:39:21.145 INFO:teuthology.orchestra.run.smithi131.stdout:Dependencies resolved. 2024-08-02T23:39:21.146 INFO:teuthology.orchestra.run.smithi131.stdout:================================================================================ 2024-08-02T23:39:21.146 INFO:teuthology.orchestra.run.smithi131.stdout: Package Arch Version Repo Size 2024-08-02T23:39:21.146 INFO:teuthology.orchestra.run.smithi131.stdout:================================================================================ 2024-08-02T23:39:21.146 INFO:teuthology.orchestra.run.smithi131.stdout:Installing: 2024-08-02T23:39:21.146 INFO:teuthology.orchestra.run.smithi131.stdout: libcephfs-devel x86_64 2:19.1.0-1260.g26c3fb8e.el9 ceph 32 k 2024-08-02T23:39:21.146 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-08-02T23:39:21.146 INFO:teuthology.orchestra.run.smithi131.stdout:Transaction Summary 2024-08-02T23:39:21.146 INFO:teuthology.orchestra.run.smithi131.stdout:================================================================================ 2024-08-02T23:39:21.146 INFO:teuthology.orchestra.run.smithi131.stdout:Install 1 Package 2024-08-02T23:39:21.146 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-08-02T23:39:21.146 INFO:teuthology.orchestra.run.smithi131.stdout:Total download size: 32 k 2024-08-02T23:39:21.147 INFO:teuthology.orchestra.run.smithi131.stdout:Installed size: 142 k 2024-08-02T23:39:21.147 INFO:teuthology.orchestra.run.smithi131.stdout:Downloading Packages: 2024-08-02T23:39:21.302 INFO:teuthology.orchestra.run.smithi073.stdout:Last metadata expiration check: 0:02:27 ago on Fri 02 Aug 2024 11:36:54 PM UTC. 2024-08-02T23:39:21.314 INFO:teuthology.orchestra.run.smithi131.stdout:libcephfs-devel-19.1.0-1260.g26c3fb8e.el9.x86_6 190 kB/s | 32 kB 00:00 2024-08-02T23:39:21.315 INFO:teuthology.orchestra.run.smithi131.stdout:-------------------------------------------------------------------------------- 2024-08-02T23:39:21.315 INFO:teuthology.orchestra.run.smithi131.stdout:Total 188 kB/s | 32 kB 00:00 2024-08-02T23:39:21.316 INFO:teuthology.orchestra.run.smithi131.stdout:Running transaction check 2024-08-02T23:39:21.318 INFO:teuthology.orchestra.run.smithi131.stdout:Transaction check succeeded. 2024-08-02T23:39:21.318 INFO:teuthology.orchestra.run.smithi131.stdout:Running transaction test 2024-08-02T23:39:21.333 INFO:teuthology.orchestra.run.smithi131.stdout:Transaction test succeeded. 2024-08-02T23:39:21.333 INFO:teuthology.orchestra.run.smithi131.stdout:Running transaction 2024-08-02T23:39:21.416 INFO:teuthology.orchestra.run.smithi131.stdout: Preparing : 1/1 2024-08-02T23:39:21.428 INFO:teuthology.orchestra.run.smithi073.stdout:Dependencies resolved. 2024-08-02T23:39:21.429 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2024-08-02T23:39:21.429 INFO:teuthology.orchestra.run.smithi073.stdout: Package Arch Version Repo Size 2024-08-02T23:39:21.429 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2024-08-02T23:39:21.429 INFO:teuthology.orchestra.run.smithi073.stdout:Installing: 2024-08-02T23:39:21.429 INFO:teuthology.orchestra.run.smithi073.stdout: libcephfs-devel x86_64 2:19.1.0-1260.g26c3fb8e.el9 ceph 32 k 2024-08-02T23:39:21.429 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-08-02T23:39:21.429 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction Summary 2024-08-02T23:39:21.429 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2024-08-02T23:39:21.429 INFO:teuthology.orchestra.run.smithi073.stdout:Install 1 Package 2024-08-02T23:39:21.429 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-08-02T23:39:21.430 INFO:teuthology.orchestra.run.smithi073.stdout:Total download size: 32 k 2024-08-02T23:39:21.430 INFO:teuthology.orchestra.run.smithi073.stdout:Installed size: 142 k 2024-08-02T23:39:21.430 INFO:teuthology.orchestra.run.smithi073.stdout:Downloading Packages: 2024-08-02T23:39:21.518 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : libcephfs-devel-2:19.1.0-1260.g26c3fb8e.el9.x86_64 1/1 2024-08-02T23:39:21.597 INFO:teuthology.orchestra.run.smithi073.stdout:libcephfs-devel-19.1.0-1260.g26c3fb8e.el9.x86_6 190 kB/s | 32 kB 00:00 2024-08-02T23:39:21.597 INFO:teuthology.orchestra.run.smithi073.stdout:-------------------------------------------------------------------------------- 2024-08-02T23:39:21.598 INFO:teuthology.orchestra.run.smithi073.stdout:Total 189 kB/s | 32 kB 00:00 2024-08-02T23:39:21.598 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction check 2024-08-02T23:39:21.601 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction check succeeded. 2024-08-02T23:39:21.601 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction test 2024-08-02T23:39:21.619 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction test succeeded. 2024-08-02T23:39:21.619 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction 2024-08-02T23:39:21.701 INFO:teuthology.orchestra.run.smithi073.stdout: Preparing : 1/1 2024-08-02T23:39:21.812 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : libcephfs-devel-2:19.1.0-1260.g26c3fb8e.el9.x86_64 1/1 2024-08-02T23:39:21.889 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: libcephfs-devel-2:19.1.0-1260.g26c3fb8e.el9.x86_64 1/1 2024-08-02T23:39:22.132 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : libcephfs-devel-2:19.1.0-1260.g26c3fb8e.el9.x86_64 1/1 2024-08-02T23:39:22.133 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-08-02T23:39:22.133 INFO:teuthology.orchestra.run.smithi131.stdout:Installed: 2024-08-02T23:39:22.133 INFO:teuthology.orchestra.run.smithi131.stdout: libcephfs-devel-2:19.1.0-1260.g26c3fb8e.el9.x86_64 2024-08-02T23:39:22.133 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-08-02T23:39:22.133 INFO:teuthology.orchestra.run.smithi131.stdout:Complete! 2024-08-02T23:39:22.208 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: libcephfs-devel-2:19.1.0-1260.g26c3fb8e.el9.x86_64 1/1 2024-08-02T23:39:22.340 DEBUG:teuthology.orchestra.run.smithi131:> sudo yum -y install librados2 2024-08-02T23:39:22.423 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : libcephfs-devel-2:19.1.0-1260.g26c3fb8e.el9.x86_64 1/1 2024-08-02T23:39:22.423 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-08-02T23:39:22.423 INFO:teuthology.orchestra.run.smithi073.stdout:Installed: 2024-08-02T23:39:22.423 INFO:teuthology.orchestra.run.smithi073.stdout: libcephfs-devel-2:19.1.0-1260.g26c3fb8e.el9.x86_64 2024-08-02T23:39:22.424 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-08-02T23:39:22.424 INFO:teuthology.orchestra.run.smithi073.stdout:Complete! 2024-08-02T23:39:22.597 DEBUG:teuthology.orchestra.run.smithi073:> sudo yum -y install librados2 2024-08-02T23:39:22.887 INFO:teuthology.orchestra.run.smithi131.stdout:Last metadata expiration check: 0:02:30 ago on Fri 02 Aug 2024 11:36:52 PM UTC. 2024-08-02T23:39:22.971 INFO:teuthology.orchestra.run.smithi131.stdout:Package librados2-2:19.1.0-1260.g26c3fb8e.el9.x86_64 is already installed. 2024-08-02T23:39:23.015 INFO:teuthology.orchestra.run.smithi131.stdout:Dependencies resolved. 2024-08-02T23:39:23.015 INFO:teuthology.orchestra.run.smithi131.stdout:Nothing to do. 2024-08-02T23:39:23.016 INFO:teuthology.orchestra.run.smithi131.stdout:Complete! 2024-08-02T23:39:23.080 DEBUG:teuthology.orchestra.run.smithi131:> sudo yum -y install librbd1 2024-08-02T23:39:23.146 INFO:teuthology.orchestra.run.smithi073.stdout:Last metadata expiration check: 0:02:29 ago on Fri 02 Aug 2024 11:36:54 PM UTC. 2024-08-02T23:39:23.232 INFO:teuthology.orchestra.run.smithi073.stdout:Package librados2-2:19.1.0-1260.g26c3fb8e.el9.x86_64 is already installed. 2024-08-02T23:39:23.273 INFO:teuthology.orchestra.run.smithi073.stdout:Dependencies resolved. 2024-08-02T23:39:23.274 INFO:teuthology.orchestra.run.smithi073.stdout:Nothing to do. 2024-08-02T23:39:23.274 INFO:teuthology.orchestra.run.smithi073.stdout:Complete! 2024-08-02T23:39:23.346 DEBUG:teuthology.orchestra.run.smithi073:> sudo yum -y install librbd1 2024-08-02T23:39:23.635 INFO:teuthology.orchestra.run.smithi131.stdout:Last metadata expiration check: 0:02:31 ago on Fri 02 Aug 2024 11:36:52 PM UTC. 2024-08-02T23:39:23.721 INFO:teuthology.orchestra.run.smithi131.stdout:Package librbd1-2:19.1.0-1260.g26c3fb8e.el9.x86_64 is already installed. 2024-08-02T23:39:23.763 INFO:teuthology.orchestra.run.smithi131.stdout:Dependencies resolved. 2024-08-02T23:39:23.763 INFO:teuthology.orchestra.run.smithi131.stdout:Nothing to do. 2024-08-02T23:39:23.764 INFO:teuthology.orchestra.run.smithi131.stdout:Complete! 2024-08-02T23:39:23.829 DEBUG:teuthology.orchestra.run.smithi131:> sudo yum -y install python3-rados 2024-08-02T23:39:23.898 INFO:teuthology.orchestra.run.smithi073.stdout:Last metadata expiration check: 0:02:29 ago on Fri 02 Aug 2024 11:36:54 PM UTC. 2024-08-02T23:39:23.983 INFO:teuthology.orchestra.run.smithi073.stdout:Package librbd1-2:19.1.0-1260.g26c3fb8e.el9.x86_64 is already installed. 2024-08-02T23:39:24.025 INFO:teuthology.orchestra.run.smithi073.stdout:Dependencies resolved. 2024-08-02T23:39:24.026 INFO:teuthology.orchestra.run.smithi073.stdout:Nothing to do. 2024-08-02T23:39:24.026 INFO:teuthology.orchestra.run.smithi073.stdout:Complete! 2024-08-02T23:39:24.091 DEBUG:teuthology.orchestra.run.smithi073:> sudo yum -y install python3-rados 2024-08-02T23:39:24.375 INFO:teuthology.orchestra.run.smithi131.stdout:Last metadata expiration check: 0:02:32 ago on Fri 02 Aug 2024 11:36:52 PM UTC. 2024-08-02T23:39:24.460 INFO:teuthology.orchestra.run.smithi131.stdout:Package python3-rados-2:19.1.0-1260.g26c3fb8e.el9.x86_64 is already installed. 2024-08-02T23:39:24.502 INFO:teuthology.orchestra.run.smithi131.stdout:Dependencies resolved. 2024-08-02T23:39:24.503 INFO:teuthology.orchestra.run.smithi131.stdout:Nothing to do. 2024-08-02T23:39:24.503 INFO:teuthology.orchestra.run.smithi131.stdout:Complete! 2024-08-02T23:39:24.574 DEBUG:teuthology.orchestra.run.smithi131:> sudo yum -y install python3-rgw 2024-08-02T23:39:24.631 INFO:teuthology.orchestra.run.smithi073.stdout:Last metadata expiration check: 0:02:30 ago on Fri 02 Aug 2024 11:36:54 PM UTC. 2024-08-02T23:39:24.716 INFO:teuthology.orchestra.run.smithi073.stdout:Package python3-rados-2:19.1.0-1260.g26c3fb8e.el9.x86_64 is already installed. 2024-08-02T23:39:24.759 INFO:teuthology.orchestra.run.smithi073.stdout:Dependencies resolved. 2024-08-02T23:39:24.760 INFO:teuthology.orchestra.run.smithi073.stdout:Nothing to do. 2024-08-02T23:39:24.760 INFO:teuthology.orchestra.run.smithi073.stdout:Complete! 2024-08-02T23:39:24.819 DEBUG:teuthology.orchestra.run.smithi073:> sudo yum -y install python3-rgw 2024-08-02T23:39:25.122 INFO:teuthology.orchestra.run.smithi131.stdout:Last metadata expiration check: 0:02:33 ago on Fri 02 Aug 2024 11:36:52 PM UTC. 2024-08-02T23:39:25.207 INFO:teuthology.orchestra.run.smithi131.stdout:Package python3-rgw-2:19.1.0-1260.g26c3fb8e.el9.x86_64 is already installed. 2024-08-02T23:39:25.250 INFO:teuthology.orchestra.run.smithi131.stdout:Dependencies resolved. 2024-08-02T23:39:25.251 INFO:teuthology.orchestra.run.smithi131.stdout:Nothing to do. 2024-08-02T23:39:25.251 INFO:teuthology.orchestra.run.smithi131.stdout:Complete! 2024-08-02T23:39:25.325 DEBUG:teuthology.orchestra.run.smithi131:> sudo yum -y install python3-cephfs 2024-08-02T23:39:25.361 INFO:teuthology.orchestra.run.smithi073.stdout:Last metadata expiration check: 0:02:31 ago on Fri 02 Aug 2024 11:36:54 PM UTC. 2024-08-02T23:39:25.446 INFO:teuthology.orchestra.run.smithi073.stdout:Package python3-rgw-2:19.1.0-1260.g26c3fb8e.el9.x86_64 is already installed. 2024-08-02T23:39:25.489 INFO:teuthology.orchestra.run.smithi073.stdout:Dependencies resolved. 2024-08-02T23:39:25.490 INFO:teuthology.orchestra.run.smithi073.stdout:Nothing to do. 2024-08-02T23:39:25.491 INFO:teuthology.orchestra.run.smithi073.stdout:Complete! 2024-08-02T23:39:25.555 DEBUG:teuthology.orchestra.run.smithi073:> sudo yum -y install python3-cephfs 2024-08-02T23:39:25.868 INFO:teuthology.orchestra.run.smithi131.stdout:Last metadata expiration check: 0:02:33 ago on Fri 02 Aug 2024 11:36:52 PM UTC. 2024-08-02T23:39:25.953 INFO:teuthology.orchestra.run.smithi131.stdout:Package python3-cephfs-2:19.1.0-1260.g26c3fb8e.el9.x86_64 is already installed. 2024-08-02T23:39:25.995 INFO:teuthology.orchestra.run.smithi131.stdout:Dependencies resolved. 2024-08-02T23:39:25.996 INFO:teuthology.orchestra.run.smithi131.stdout:Nothing to do. 2024-08-02T23:39:25.996 INFO:teuthology.orchestra.run.smithi131.stdout:Complete! 2024-08-02T23:39:26.058 DEBUG:teuthology.orchestra.run.smithi131:> sudo yum -y install python3-rbd 2024-08-02T23:39:26.096 INFO:teuthology.orchestra.run.smithi073.stdout:Last metadata expiration check: 0:02:32 ago on Fri 02 Aug 2024 11:36:54 PM UTC. 2024-08-02T23:39:26.181 INFO:teuthology.orchestra.run.smithi073.stdout:Package python3-cephfs-2:19.1.0-1260.g26c3fb8e.el9.x86_64 is already installed. 2024-08-02T23:39:26.223 INFO:teuthology.orchestra.run.smithi073.stdout:Dependencies resolved. 2024-08-02T23:39:26.225 INFO:teuthology.orchestra.run.smithi073.stdout:Nothing to do. 2024-08-02T23:39:26.225 INFO:teuthology.orchestra.run.smithi073.stdout:Complete! 2024-08-02T23:39:26.286 DEBUG:teuthology.orchestra.run.smithi073:> sudo yum -y install python3-rbd 2024-08-02T23:39:26.608 INFO:teuthology.orchestra.run.smithi131.stdout:Last metadata expiration check: 0:02:34 ago on Fri 02 Aug 2024 11:36:52 PM UTC. 2024-08-02T23:39:26.693 INFO:teuthology.orchestra.run.smithi131.stdout:Package python3-rbd-2:19.1.0-1260.g26c3fb8e.el9.x86_64 is already installed. 2024-08-02T23:39:26.736 INFO:teuthology.orchestra.run.smithi131.stdout:Dependencies resolved. 2024-08-02T23:39:26.737 INFO:teuthology.orchestra.run.smithi131.stdout:Nothing to do. 2024-08-02T23:39:26.737 INFO:teuthology.orchestra.run.smithi131.stdout:Complete! 2024-08-02T23:39:26.800 DEBUG:teuthology.orchestra.run.smithi131:> sudo yum -y install rbd-fuse 2024-08-02T23:39:26.831 INFO:teuthology.orchestra.run.smithi073.stdout:Last metadata expiration check: 0:02:32 ago on Fri 02 Aug 2024 11:36:54 PM UTC. 2024-08-02T23:39:26.917 INFO:teuthology.orchestra.run.smithi073.stdout:Package python3-rbd-2:19.1.0-1260.g26c3fb8e.el9.x86_64 is already installed. 2024-08-02T23:39:26.959 INFO:teuthology.orchestra.run.smithi073.stdout:Dependencies resolved. 2024-08-02T23:39:26.960 INFO:teuthology.orchestra.run.smithi073.stdout:Nothing to do. 2024-08-02T23:39:26.961 INFO:teuthology.orchestra.run.smithi073.stdout:Complete! 2024-08-02T23:39:27.023 DEBUG:teuthology.orchestra.run.smithi073:> sudo yum -y install rbd-fuse 2024-08-02T23:39:27.353 INFO:teuthology.orchestra.run.smithi131.stdout:Last metadata expiration check: 0:02:35 ago on Fri 02 Aug 2024 11:36:52 PM UTC. 2024-08-02T23:39:27.482 INFO:teuthology.orchestra.run.smithi131.stdout:Dependencies resolved. 2024-08-02T23:39:27.483 INFO:teuthology.orchestra.run.smithi131.stdout:================================================================================ 2024-08-02T23:39:27.483 INFO:teuthology.orchestra.run.smithi131.stdout: Package Architecture Version Repository Size 2024-08-02T23:39:27.483 INFO:teuthology.orchestra.run.smithi131.stdout:================================================================================ 2024-08-02T23:39:27.483 INFO:teuthology.orchestra.run.smithi131.stdout:Installing: 2024-08-02T23:39:27.483 INFO:teuthology.orchestra.run.smithi131.stdout: rbd-fuse x86_64 2:19.1.0-1260.g26c3fb8e.el9 ceph 86 k 2024-08-02T23:39:27.483 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-08-02T23:39:27.483 INFO:teuthology.orchestra.run.smithi131.stdout:Transaction Summary 2024-08-02T23:39:27.483 INFO:teuthology.orchestra.run.smithi131.stdout:================================================================================ 2024-08-02T23:39:27.484 INFO:teuthology.orchestra.run.smithi131.stdout:Install 1 Package 2024-08-02T23:39:27.484 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-08-02T23:39:27.484 INFO:teuthology.orchestra.run.smithi131.stdout:Total download size: 86 k 2024-08-02T23:39:27.484 INFO:teuthology.orchestra.run.smithi131.stdout:Installed size: 256 k 2024-08-02T23:39:27.484 INFO:teuthology.orchestra.run.smithi131.stdout:Downloading Packages: 2024-08-02T23:39:27.564 INFO:teuthology.orchestra.run.smithi073.stdout:Last metadata expiration check: 0:02:33 ago on Fri 02 Aug 2024 11:36:54 PM UTC. 2024-08-02T23:39:27.676 INFO:teuthology.orchestra.run.smithi131.stdout:rbd-fuse-19.1.0-1260.g26c3fb8e.el9.x86_64.rpm 445 kB/s | 86 kB 00:00 2024-08-02T23:39:27.677 INFO:teuthology.orchestra.run.smithi131.stdout:-------------------------------------------------------------------------------- 2024-08-02T23:39:27.677 INFO:teuthology.orchestra.run.smithi131.stdout:Total 442 kB/s | 86 kB 00:00 2024-08-02T23:39:27.677 INFO:teuthology.orchestra.run.smithi131.stdout:Running transaction check 2024-08-02T23:39:27.684 INFO:teuthology.orchestra.run.smithi131.stdout:Transaction check succeeded. 2024-08-02T23:39:27.684 INFO:teuthology.orchestra.run.smithi131.stdout:Running transaction test 2024-08-02T23:39:27.693 INFO:teuthology.orchestra.run.smithi073.stdout:Dependencies resolved. 2024-08-02T23:39:27.694 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2024-08-02T23:39:27.694 INFO:teuthology.orchestra.run.smithi073.stdout: Package Architecture Version Repository Size 2024-08-02T23:39:27.694 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2024-08-02T23:39:27.694 INFO:teuthology.orchestra.run.smithi073.stdout:Installing: 2024-08-02T23:39:27.694 INFO:teuthology.orchestra.run.smithi073.stdout: rbd-fuse x86_64 2:19.1.0-1260.g26c3fb8e.el9 ceph 86 k 2024-08-02T23:39:27.694 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-08-02T23:39:27.694 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction Summary 2024-08-02T23:39:27.694 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2024-08-02T23:39:27.694 INFO:teuthology.orchestra.run.smithi073.stdout:Install 1 Package 2024-08-02T23:39:27.694 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-08-02T23:39:27.695 INFO:teuthology.orchestra.run.smithi073.stdout:Total download size: 86 k 2024-08-02T23:39:27.695 INFO:teuthology.orchestra.run.smithi073.stdout:Installed size: 256 k 2024-08-02T23:39:27.695 INFO:teuthology.orchestra.run.smithi073.stdout:Downloading Packages: 2024-08-02T23:39:27.752 INFO:teuthology.orchestra.run.smithi131.stdout:Transaction test succeeded. 2024-08-02T23:39:27.752 INFO:teuthology.orchestra.run.smithi131.stdout:Running transaction 2024-08-02T23:39:27.900 INFO:teuthology.orchestra.run.smithi131.stdout: Preparing : 1/1 2024-08-02T23:39:27.920 INFO:teuthology.orchestra.run.smithi073.stdout:rbd-fuse-19.1.0-1260.g26c3fb8e.el9.x86_64.rpm 380 kB/s | 86 kB 00:00 2024-08-02T23:39:27.921 INFO:teuthology.orchestra.run.smithi073.stdout:-------------------------------------------------------------------------------- 2024-08-02T23:39:27.921 INFO:teuthology.orchestra.run.smithi073.stdout:Total 378 kB/s | 86 kB 00:00 2024-08-02T23:39:27.921 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction check 2024-08-02T23:39:27.928 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction check succeeded. 2024-08-02T23:39:27.928 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction test 2024-08-02T23:39:27.996 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction test succeeded. 2024-08-02T23:39:27.996 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction 2024-08-02T23:39:27.999 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : rbd-fuse-2:19.1.0-1260.g26c3fb8e.el9.x86_64 1/1 2024-08-02T23:39:28.145 INFO:teuthology.orchestra.run.smithi073.stdout: Preparing : 1/1 2024-08-02T23:39:28.243 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : rbd-fuse-2:19.1.0-1260.g26c3fb8e.el9.x86_64 1/1 2024-08-02T23:39:28.370 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: rbd-fuse-2:19.1.0-1260.g26c3fb8e.el9.x86_64 1/1 2024-08-02T23:39:28.641 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : rbd-fuse-2:19.1.0-1260.g26c3fb8e.el9.x86_64 1/1 2024-08-02T23:39:28.641 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-08-02T23:39:28.641 INFO:teuthology.orchestra.run.smithi131.stdout:Installed: 2024-08-02T23:39:28.641 INFO:teuthology.orchestra.run.smithi131.stdout: rbd-fuse-2:19.1.0-1260.g26c3fb8e.el9.x86_64 2024-08-02T23:39:28.641 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-08-02T23:39:28.641 INFO:teuthology.orchestra.run.smithi131.stdout:Complete! 2024-08-02T23:39:28.732 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: rbd-fuse-2:19.1.0-1260.g26c3fb8e.el9.x86_64 1/1 2024-08-02T23:39:28.872 DEBUG:teuthology.orchestra.run.smithi131:> sudo yum -y install rbd-mirror 2024-08-02T23:39:28.976 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : rbd-fuse-2:19.1.0-1260.g26c3fb8e.el9.x86_64 1/1 2024-08-02T23:39:28.977 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-08-02T23:39:28.977 INFO:teuthology.orchestra.run.smithi073.stdout:Installed: 2024-08-02T23:39:28.977 INFO:teuthology.orchestra.run.smithi073.stdout: rbd-fuse-2:19.1.0-1260.g26c3fb8e.el9.x86_64 2024-08-02T23:39:28.977 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-08-02T23:39:28.977 INFO:teuthology.orchestra.run.smithi073.stdout:Complete! 2024-08-02T23:39:29.215 DEBUG:teuthology.orchestra.run.smithi073:> sudo yum -y install rbd-mirror 2024-08-02T23:39:29.430 INFO:teuthology.orchestra.run.smithi131.stdout:Last metadata expiration check: 0:02:37 ago on Fri 02 Aug 2024 11:36:52 PM UTC. 2024-08-02T23:39:29.561 INFO:teuthology.orchestra.run.smithi131.stdout:Dependencies resolved. 2024-08-02T23:39:29.562 INFO:teuthology.orchestra.run.smithi131.stdout:================================================================================ 2024-08-02T23:39:29.562 INFO:teuthology.orchestra.run.smithi131.stdout: Package Arch Version Repo Size 2024-08-02T23:39:29.562 INFO:teuthology.orchestra.run.smithi131.stdout:================================================================================ 2024-08-02T23:39:29.562 INFO:teuthology.orchestra.run.smithi131.stdout:Installing: 2024-08-02T23:39:29.563 INFO:teuthology.orchestra.run.smithi131.stdout: rbd-mirror x86_64 2:19.1.0-1260.g26c3fb8e.el9 ceph 3.1 M 2024-08-02T23:39:29.563 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-08-02T23:39:29.563 INFO:teuthology.orchestra.run.smithi131.stdout:Transaction Summary 2024-08-02T23:39:29.563 INFO:teuthology.orchestra.run.smithi131.stdout:================================================================================ 2024-08-02T23:39:29.563 INFO:teuthology.orchestra.run.smithi131.stdout:Install 1 Package 2024-08-02T23:39:29.563 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-08-02T23:39:29.563 INFO:teuthology.orchestra.run.smithi131.stdout:Total download size: 3.1 M 2024-08-02T23:39:29.563 INFO:teuthology.orchestra.run.smithi131.stdout:Installed size: 14 M 2024-08-02T23:39:29.563 INFO:teuthology.orchestra.run.smithi131.stdout:Downloading Packages: 2024-08-02T23:39:29.791 INFO:teuthology.orchestra.run.smithi073.stdout:Last metadata expiration check: 0:02:35 ago on Fri 02 Aug 2024 11:36:54 PM UTC. 2024-08-02T23:39:29.889 INFO:teuthology.orchestra.run.smithi131.stdout:rbd-mirror-19.1.0-1260.g26c3fb8e.el9.x86_64.rpm 9.4 MB/s | 3.1 MB 00:00 2024-08-02T23:39:29.890 INFO:teuthology.orchestra.run.smithi131.stdout:-------------------------------------------------------------------------------- 2024-08-02T23:39:29.890 INFO:teuthology.orchestra.run.smithi131.stdout:Total 9.3 MB/s | 3.1 MB 00:00 2024-08-02T23:39:29.891 INFO:teuthology.orchestra.run.smithi131.stdout:Running transaction check 2024-08-02T23:39:29.899 INFO:teuthology.orchestra.run.smithi131.stdout:Transaction check succeeded. 2024-08-02T23:39:29.899 INFO:teuthology.orchestra.run.smithi131.stdout:Running transaction test 2024-08-02T23:39:29.924 INFO:teuthology.orchestra.run.smithi073.stdout:Dependencies resolved. 2024-08-02T23:39:29.925 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2024-08-02T23:39:29.925 INFO:teuthology.orchestra.run.smithi073.stdout: Package Arch Version Repo Size 2024-08-02T23:39:29.925 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2024-08-02T23:39:29.925 INFO:teuthology.orchestra.run.smithi073.stdout:Installing: 2024-08-02T23:39:29.925 INFO:teuthology.orchestra.run.smithi073.stdout: rbd-mirror x86_64 2:19.1.0-1260.g26c3fb8e.el9 ceph 3.1 M 2024-08-02T23:39:29.925 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-08-02T23:39:29.925 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction Summary 2024-08-02T23:39:29.925 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2024-08-02T23:39:29.925 INFO:teuthology.orchestra.run.smithi073.stdout:Install 1 Package 2024-08-02T23:39:29.925 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-08-02T23:39:29.926 INFO:teuthology.orchestra.run.smithi073.stdout:Total download size: 3.1 M 2024-08-02T23:39:29.926 INFO:teuthology.orchestra.run.smithi073.stdout:Installed size: 14 M 2024-08-02T23:39:29.926 INFO:teuthology.orchestra.run.smithi073.stdout:Downloading Packages: 2024-08-02T23:39:29.980 INFO:teuthology.orchestra.run.smithi131.stdout:Transaction test succeeded. 2024-08-02T23:39:29.980 INFO:teuthology.orchestra.run.smithi131.stdout:Running transaction 2024-08-02T23:39:30.292 INFO:teuthology.orchestra.run.smithi073.stdout:rbd-mirror-19.1.0-1260.g26c3fb8e.el9.x86_64.rpm 8.4 MB/s | 3.1 MB 00:00 2024-08-02T23:39:30.292 INFO:teuthology.orchestra.run.smithi073.stdout:-------------------------------------------------------------------------------- 2024-08-02T23:39:30.293 INFO:teuthology.orchestra.run.smithi073.stdout:Total 8.3 MB/s | 3.1 MB 00:00 2024-08-02T23:39:30.293 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction check 2024-08-02T23:39:30.302 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction check succeeded. 2024-08-02T23:39:30.302 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction test 2024-08-02T23:39:30.383 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction test succeeded. 2024-08-02T23:39:30.384 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction 2024-08-02T23:39:30.427 INFO:teuthology.orchestra.run.smithi131.stdout: Preparing : 1/1 2024-08-02T23:39:30.498 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : rbd-mirror-2:19.1.0-1260.g26c3fb8e.el9.x86_64 1/1 2024-08-02T23:39:30.525 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: rbd-mirror-2:19.1.0-1260.g26c3fb8e.el9.x86_64 1/1 2024-08-02T23:39:30.525 INFO:teuthology.orchestra.run.smithi131.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-02T23:39:30.525 INFO:teuthology.orchestra.run.smithi131.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-08-02T23:39:30.525 INFO:teuthology.orchestra.run.smithi131.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-08-02T23:39:30.526 INFO:teuthology.orchestra.run.smithi131.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-08-02T23:39:30.526 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-08-02T23:39:30.828 INFO:teuthology.orchestra.run.smithi073.stdout: Preparing : 1/1 2024-08-02T23:39:30.884 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : rbd-mirror-2:19.1.0-1260.g26c3fb8e.el9.x86_64 1/1 2024-08-02T23:39:30.910 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: rbd-mirror-2:19.1.0-1260.g26c3fb8e.el9.x86_64 1/1 2024-08-02T23:39:30.910 INFO:teuthology.orchestra.run.smithi073.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-02T23:39:30.910 INFO:teuthology.orchestra.run.smithi073.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-08-02T23:39:30.911 INFO:teuthology.orchestra.run.smithi073.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-08-02T23:39:30.911 INFO:teuthology.orchestra.run.smithi073.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-08-02T23:39:30.911 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-08-02T23:39:31.682 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : rbd-mirror-2:19.1.0-1260.g26c3fb8e.el9.x86_64 1/1 2024-08-02T23:39:31.682 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-08-02T23:39:31.682 INFO:teuthology.orchestra.run.smithi131.stdout:Installed: 2024-08-02T23:39:31.682 INFO:teuthology.orchestra.run.smithi131.stdout: rbd-mirror-2:19.1.0-1260.g26c3fb8e.el9.x86_64 2024-08-02T23:39:31.682 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-08-02T23:39:31.682 INFO:teuthology.orchestra.run.smithi131.stdout:Complete! 2024-08-02T23:39:31.930 DEBUG:teuthology.orchestra.run.smithi131:> sudo yum -y install rbd-nbd 2024-08-02T23:39:32.025 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : rbd-mirror-2:19.1.0-1260.g26c3fb8e.el9.x86_64 1/1 2024-08-02T23:39:32.026 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-08-02T23:39:32.026 INFO:teuthology.orchestra.run.smithi073.stdout:Installed: 2024-08-02T23:39:32.026 INFO:teuthology.orchestra.run.smithi073.stdout: rbd-mirror-2:19.1.0-1260.g26c3fb8e.el9.x86_64 2024-08-02T23:39:32.026 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-08-02T23:39:32.026 INFO:teuthology.orchestra.run.smithi073.stdout:Complete! 2024-08-02T23:39:32.258 DEBUG:teuthology.orchestra.run.smithi073:> sudo yum -y install rbd-nbd 2024-08-02T23:39:32.494 INFO:teuthology.orchestra.run.smithi131.stdout:Last metadata expiration check: 0:02:40 ago on Fri 02 Aug 2024 11:36:52 PM UTC. 2024-08-02T23:39:32.624 INFO:teuthology.orchestra.run.smithi131.stdout:Dependencies resolved. 2024-08-02T23:39:32.624 INFO:teuthology.orchestra.run.smithi131.stdout:================================================================================ 2024-08-02T23:39:32.624 INFO:teuthology.orchestra.run.smithi131.stdout: Package Architecture Version Repository Size 2024-08-02T23:39:32.625 INFO:teuthology.orchestra.run.smithi131.stdout:================================================================================ 2024-08-02T23:39:32.625 INFO:teuthology.orchestra.run.smithi131.stdout:Installing: 2024-08-02T23:39:32.625 INFO:teuthology.orchestra.run.smithi131.stdout: rbd-nbd x86_64 2:19.1.0-1260.g26c3fb8e.el9 ceph 169 k 2024-08-02T23:39:32.625 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-08-02T23:39:32.625 INFO:teuthology.orchestra.run.smithi131.stdout:Transaction Summary 2024-08-02T23:39:32.625 INFO:teuthology.orchestra.run.smithi131.stdout:================================================================================ 2024-08-02T23:39:32.625 INFO:teuthology.orchestra.run.smithi131.stdout:Install 1 Package 2024-08-02T23:39:32.625 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-08-02T23:39:32.625 INFO:teuthology.orchestra.run.smithi131.stdout:Total download size: 169 k 2024-08-02T23:39:32.625 INFO:teuthology.orchestra.run.smithi131.stdout:Installed size: 540 k 2024-08-02T23:39:32.625 INFO:teuthology.orchestra.run.smithi131.stdout:Downloading Packages: 2024-08-02T23:39:32.829 INFO:teuthology.orchestra.run.smithi073.stdout:Last metadata expiration check: 0:02:38 ago on Fri 02 Aug 2024 11:36:54 PM UTC. 2024-08-02T23:39:32.854 INFO:teuthology.orchestra.run.smithi131.stdout:rbd-nbd-19.1.0-1260.g26c3fb8e.el9.x86_64.rpm 743 kB/s | 169 kB 00:00 2024-08-02T23:39:32.854 INFO:teuthology.orchestra.run.smithi131.stdout:-------------------------------------------------------------------------------- 2024-08-02T23:39:32.855 INFO:teuthology.orchestra.run.smithi131.stdout:Total 737 kB/s | 169 kB 00:00 2024-08-02T23:39:32.855 INFO:teuthology.orchestra.run.smithi131.stdout:Running transaction check 2024-08-02T23:39:32.862 INFO:teuthology.orchestra.run.smithi131.stdout:Transaction check succeeded. 2024-08-02T23:39:32.862 INFO:teuthology.orchestra.run.smithi131.stdout:Running transaction test 2024-08-02T23:39:32.927 INFO:teuthology.orchestra.run.smithi131.stdout:Transaction test succeeded. 2024-08-02T23:39:32.927 INFO:teuthology.orchestra.run.smithi131.stdout:Running transaction 2024-08-02T23:39:32.958 INFO:teuthology.orchestra.run.smithi073.stdout:Dependencies resolved. 2024-08-02T23:39:32.958 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2024-08-02T23:39:32.958 INFO:teuthology.orchestra.run.smithi073.stdout: Package Architecture Version Repository Size 2024-08-02T23:39:32.959 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2024-08-02T23:39:32.959 INFO:teuthology.orchestra.run.smithi073.stdout:Installing: 2024-08-02T23:39:32.959 INFO:teuthology.orchestra.run.smithi073.stdout: rbd-nbd x86_64 2:19.1.0-1260.g26c3fb8e.el9 ceph 169 k 2024-08-02T23:39:32.959 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-08-02T23:39:32.959 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction Summary 2024-08-02T23:39:32.959 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2024-08-02T23:39:32.959 INFO:teuthology.orchestra.run.smithi073.stdout:Install 1 Package 2024-08-02T23:39:32.959 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-08-02T23:39:32.960 INFO:teuthology.orchestra.run.smithi073.stdout:Total download size: 169 k 2024-08-02T23:39:32.960 INFO:teuthology.orchestra.run.smithi073.stdout:Installed size: 540 k 2024-08-02T23:39:32.960 INFO:teuthology.orchestra.run.smithi073.stdout:Downloading Packages: 2024-08-02T23:39:33.080 INFO:teuthology.orchestra.run.smithi131.stdout: Preparing : 1/1 2024-08-02T23:39:33.186 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : rbd-nbd-2:19.1.0-1260.g26c3fb8e.el9.x86_64 1/1 2024-08-02T23:39:33.201 INFO:teuthology.orchestra.run.smithi073.stdout:rbd-nbd-19.1.0-1260.g26c3fb8e.el9.x86_64.rpm 703 kB/s | 169 kB 00:00 2024-08-02T23:39:33.201 INFO:teuthology.orchestra.run.smithi073.stdout:-------------------------------------------------------------------------------- 2024-08-02T23:39:33.202 INFO:teuthology.orchestra.run.smithi073.stdout:Total 699 kB/s | 169 kB 00:00 2024-08-02T23:39:33.202 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction check 2024-08-02T23:39:33.210 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction check succeeded. 2024-08-02T23:39:33.210 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction test 2024-08-02T23:39:33.273 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction test succeeded. 2024-08-02T23:39:33.273 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction 2024-08-02T23:39:33.422 INFO:teuthology.orchestra.run.smithi073.stdout: Preparing : 1/1 2024-08-02T23:39:33.530 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : rbd-nbd-2:19.1.0-1260.g26c3fb8e.el9.x86_64 1/1 2024-08-02T23:39:33.599 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: rbd-nbd-2:19.1.0-1260.g26c3fb8e.el9.x86_64 1/1 2024-08-02T23:39:33.840 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : rbd-nbd-2:19.1.0-1260.g26c3fb8e.el9.x86_64 1/1 2024-08-02T23:39:33.841 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-08-02T23:39:33.841 INFO:teuthology.orchestra.run.smithi131.stdout:Installed: 2024-08-02T23:39:33.841 INFO:teuthology.orchestra.run.smithi131.stdout: rbd-nbd-2:19.1.0-1260.g26c3fb8e.el9.x86_64 2024-08-02T23:39:33.841 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-08-02T23:39:33.841 INFO:teuthology.orchestra.run.smithi131.stdout:Complete! 2024-08-02T23:39:33.932 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: rbd-nbd-2:19.1.0-1260.g26c3fb8e.el9.x86_64 1/1 2024-08-02T23:39:34.106 DEBUG:teuthology.parallel:result is None 2024-08-02T23:39:34.195 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : rbd-nbd-2:19.1.0-1260.g26c3fb8e.el9.x86_64 1/1 2024-08-02T23:39:34.195 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-08-02T23:39:34.196 INFO:teuthology.orchestra.run.smithi073.stdout:Installed: 2024-08-02T23:39:34.196 INFO:teuthology.orchestra.run.smithi073.stdout: rbd-nbd-2:19.1.0-1260.g26c3fb8e.el9.x86_64 2024-08-02T23:39:34.196 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-08-02T23:39:34.196 INFO:teuthology.orchestra.run.smithi073.stdout:Complete! 2024-08-02T23:39:34.454 DEBUG:teuthology.parallel:result is None 2024-08-02T23:39:34.455 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=26c3fb8e197dcf7a49a54d1f4c8a7362ee35a8ea 2024-08-02T23:39:34.600 DEBUG:teuthology.orchestra.run.smithi073:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-08-02T23:39:34.621 INFO:teuthology.orchestra.run.smithi073.stdout:19.1.0-1260.g26c3fb8e.el9 2024-08-02T23:39:34.622 INFO:teuthology.packaging:The installed version of ceph is 19.1.0-1260.g26c3fb8e.el9 2024-08-02T23:39:34.622 INFO:teuthology.task.install:The correct ceph version 19.1.0-1260.g26c3fb8e is installed. 2024-08-02T23:39:34.624 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=26c3fb8e197dcf7a49a54d1f4c8a7362ee35a8ea 2024-08-02T23:39:34.764 DEBUG:teuthology.orchestra.run.smithi131:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-08-02T23:39:34.789 INFO:teuthology.orchestra.run.smithi131.stdout:19.1.0-1260.g26c3fb8e.el9 2024-08-02T23:39:34.789 INFO:teuthology.packaging:The installed version of ceph is 19.1.0-1260.g26c3fb8e.el9 2024-08-02T23:39:34.789 INFO:teuthology.task.install:The correct ceph version 19.1.0-1260.g26c3fb8e is installed. 2024-08-02T23:39:34.791 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-08-02T23:39:34.792 DEBUG:teuthology.orchestra.run.smithi073:> set -ex 2024-08-02T23:39:34.792 DEBUG:teuthology.orchestra.run.smithi073:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-08-02T23:39:34.817 DEBUG:teuthology.orchestra.run.smithi131:> set -ex 2024-08-02T23:39:34.817 DEBUG:teuthology.orchestra.run.smithi131:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-08-02T23:39:34.860 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-08-02T23:39:34.860 DEBUG:teuthology.orchestra.run.smithi073:> set -ex 2024-08-02T23:39:34.861 DEBUG:teuthology.orchestra.run.smithi073:> sudo dd of=/usr/bin/daemon-helper 2024-08-02T23:39:34.884 DEBUG:teuthology.orchestra.run.smithi073:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-08-02T23:39:34.948 DEBUG:teuthology.orchestra.run.smithi131:> set -ex 2024-08-02T23:39:34.949 DEBUG:teuthology.orchestra.run.smithi131:> sudo dd of=/usr/bin/daemon-helper 2024-08-02T23:39:34.978 DEBUG:teuthology.orchestra.run.smithi131:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-08-02T23:39:35.047 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-08-02T23:39:35.048 DEBUG:teuthology.orchestra.run.smithi073:> set -ex 2024-08-02T23:39:35.048 DEBUG:teuthology.orchestra.run.smithi073:> sudo dd of=/usr/bin/adjust-ulimits 2024-08-02T23:39:35.073 DEBUG:teuthology.orchestra.run.smithi073:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-08-02T23:39:35.139 DEBUG:teuthology.orchestra.run.smithi131:> set -ex 2024-08-02T23:39:35.139 DEBUG:teuthology.orchestra.run.smithi131:> sudo dd of=/usr/bin/adjust-ulimits 2024-08-02T23:39:35.167 DEBUG:teuthology.orchestra.run.smithi131:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-08-02T23:39:35.236 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-08-02T23:39:35.237 DEBUG:teuthology.orchestra.run.smithi073:> set -ex 2024-08-02T23:39:35.237 DEBUG:teuthology.orchestra.run.smithi073:> sudo dd of=/usr/bin/stdin-killer 2024-08-02T23:39:35.262 DEBUG:teuthology.orchestra.run.smithi073:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-08-02T23:39:35.327 DEBUG:teuthology.orchestra.run.smithi131:> set -ex 2024-08-02T23:39:35.327 DEBUG:teuthology.orchestra.run.smithi131:> sudo dd of=/usr/bin/stdin-killer 2024-08-02T23:39:35.356 DEBUG:teuthology.orchestra.run.smithi131:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-08-02T23:39:35.425 INFO:teuthology.run_tasks:Running task ceph... 2024-08-02T23:39:35.503 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-08-02T23:39:35.504 DEBUG:teuthology.orchestra.run.smithi073:> sudo chmod 777 /var/log/ceph 2024-08-02T23:39:35.506 DEBUG:teuthology.orchestra.run.smithi131:> sudo chmod 777 /var/log/ceph 2024-08-02T23:39:35.535 INFO:tasks.ceph:Disabling ceph logrotate... 2024-08-02T23:39:35.535 DEBUG:teuthology.orchestra.run.smithi073:> sudo rm -f -- /etc/logrotate.d/ceph 2024-08-02T23:39:35.571 DEBUG:teuthology.orchestra.run.smithi131:> sudo rm -f -- /etc/logrotate.d/ceph 2024-08-02T23:39:35.603 INFO:tasks.ceph:Creating extra log directories... 2024-08-02T23:39:35.603 DEBUG:teuthology.orchestra.run.smithi073:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-08-02T23:39:35.636 DEBUG:teuthology.orchestra.run.smithi131:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-08-02T23:39:35.678 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-08-02T23:39:35.679 INFO:tasks.ceph:config {'conf': {'mon': {'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10, 'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon warn on pool no app': False, 'osd pool default crimson': True}, 'osd': {'debug monc': 20, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug ms': 1, 'debug osd': 20, 'osd blocked scrub grace period': 3600, 'osd class default list': '*', 'osd class load list': '*', 'osd objectstore': 'seastore', 'osd shutdown pgref assert': True}, 'client': {'debug ms': 1}, 'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1, 'mgr stats period': 30}, 'osd.0': {'crimson seastar cpu cores': '0-2'}, 'osd.1': {'crimson seastar cpu cores': '3-5'}, 'osd.2': {'crimson seastar cpu cores': '0-2'}, 'osd.3': {'crimson seastar cpu cores': '3-5'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'reached quota', 'but it is still running', 'overall HEALTH_', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-08-02T23:39:35.679 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yuriw-2024-08-02_15:39:04-crimson-rados-squid-release-distro-default-smithi/7833276', 'branch': 'squid-release', 'description': 'crimson-rados/basic/{clusters/fixed-2 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/seastore tasks/rados_api_tests}', 'email': 'yweinste@redhat.com', 'first_in_suite': False, 'job_id': '7833276', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'yuriw-2024-08-02_15:39:04-crimson-rados-squid-release-distro-default-smithi', 'no_nested_subset': False, 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'squid-release'}, 'ceph': {'conf': {'client': {'debug ms': 1}, 'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1, 'mgr stats period': 30}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon warn on pool no app': False, 'osd pool default crimson': True}, 'osd': {'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug ms': 1, 'debug osd': 20, 'osd blocked scrub grace period': 3600, 'osd class default list': '*', 'osd class load list': '*', 'osd objectstore': 'seastore', 'osd shutdown pgref assert': True}, 'osd.0': {'crimson seastar cpu cores': '0-2'}, 'osd.1': {'crimson seastar cpu cores': '3-5'}, 'osd.2': {'crimson seastar cpu cores': '0-2'}, 'osd.3': {'crimson seastar cpu cores': '3-5'}}, 'flavor': 'crimson', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'reached quota', 'but it is still running', 'overall HEALTH_', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)'], 'sha1': '26c3fb8e197dcf7a49a54d1f4c8a7362ee35a8ea'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'crimson', 'sha1': '26c3fb8e197dcf7a49a54d1f4c8a7362ee35a8ea'}}, 'selinux': {'allowlist': ['scontext=system_u:system_r:getty_t:s0']}, 'workunit': {'branch': 'squid-release', 'env': {'CRIMSON_COMPAT': '1'}, 'sha1': '26c3fb8e197dcf7a49a54d1f4c8a7362ee35a8ea'}}, 'owner': 'scheduled_yuriw@teuthology', 'priority': 99, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['mon.a', 'osd.0', 'osd.1', 'client.0', 'node-exporter.a'], ['mgr.x', 'osd.2', 'osd.3', 'client.1', 'prometheus.a', 'node-exporter.b']], 'seed': 5670, 'sha1': '26c3fb8e197dcf7a49a54d1f4c8a7362ee35a8ea', 'sleep_before_teardown': 0, 'subset': '111/120000', 'suite': 'crimson-rados', 'suite_branch': 'squid-release', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_26c3fb8e197dcf7a49a54d1f4c8a7362ee35a8ea/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': '26c3fb8e197dcf7a49a54d1f4c8a7362ee35a8ea', 'targets': {'smithi073.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIKvQZFa6VwuEe1OB7leedPBiptAZaVBbyHRtF8y8R1eMRCe1ZAwnuytmeA9MKuyXX5kWXmo/XdEtGVc4T1M8aI=', 'smithi131.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPRu4Go9/I1OWxz4uAF5w3bG6Emuz4/nUMZtNrjl1gfafid+PoEwVI1qZgMZzgaBeDrJp9+b1Cc0cLrqNEDC5q8='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'conf': {'mon': {'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10, 'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon warn on pool no app': False, 'osd pool default crimson': True}, 'osd': {'debug monc': 20, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug ms': 1, 'debug osd': 20, 'osd blocked scrub grace period': 3600, 'osd class default list': '*', 'osd class load list': '*', 'osd objectstore': 'seastore', 'osd shutdown pgref assert': True}, 'client': {'debug ms': 1}, 'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1, 'mgr stats period': 30}, 'osd.0': {'crimson seastar cpu cores': '0-2'}, 'osd.1': {'crimson seastar cpu cores': '3-5'}, 'osd.2': {'crimson seastar cpu cores': '0-2'}, 'osd.3': {'crimson seastar cpu cores': '3-5'}}, 'flavor': 'crimson', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'reached quota', 'but it is still running', 'overall HEALTH_', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)'], 'sha1': '26c3fb8e197dcf7a49a54d1f4c8a7362ee35a8ea', 'cluster': 'ceph'}}, {'workunit': {'clients': {'client.0': ['rados/test.sh --crimson', 'rados/test_pool_quota.sh']}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '53ce1462e129f6eb4071986336534c740fdebd31', 'timestamp': '2024-08-02_15:39:04', 'tube': 'smithi', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.4074525'} 2024-08-02T23:39:35.679 DEBUG:teuthology.orchestra.run.smithi073:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-08-02T23:39:35.701 DEBUG:teuthology.orchestra.run.smithi131:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-08-02T23:39:35.728 DEBUG:teuthology.orchestra.run.smithi073:> sudo install -d -m0777 -- /var/run/ceph 2024-08-02T23:39:35.756 DEBUG:teuthology.orchestra.run.smithi131:> sudo install -d -m0777 -- /var/run/ceph 2024-08-02T23:39:35.795 DEBUG:teuthology.orchestra.run.smithi073:> set -ex 2024-08-02T23:39:35.795 DEBUG:teuthology.orchestra.run.smithi073:> dd if=/scratch_devs of=/dev/stdout 2024-08-02T23:39:35.834 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-08-02T23:39:35.835 DEBUG:teuthology.orchestra.run.smithi073:> stat /dev/vg_nvme/lv_1 2024-08-02T23:39:35.890 INFO:teuthology.orchestra.run.smithi073.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-08-02T23:39:35.890 INFO:teuthology.orchestra.run.smithi073.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-08-02T23:39:35.890 INFO:teuthology.orchestra.run.smithi073.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-08-02T23:39:35.890 INFO:teuthology.orchestra.run.smithi073.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-08-02T23:39:35.890 INFO:teuthology.orchestra.run.smithi073.stdout:Context: system_u:object_r:device_t:s0 2024-08-02T23:39:35.890 INFO:teuthology.orchestra.run.smithi073.stdout:Access: 2024-08-02 23:39:31.167020479 +0000 2024-08-02T23:39:35.890 INFO:teuthology.orchestra.run.smithi073.stdout:Modify: 2024-08-02 23:35:37.805667468 +0000 2024-08-02T23:39:35.891 INFO:teuthology.orchestra.run.smithi073.stdout:Change: 2024-08-02 23:35:37.805667468 +0000 2024-08-02T23:39:35.891 INFO:teuthology.orchestra.run.smithi073.stdout: Birth: 2024-08-02 23:35:37.805667468 +0000 2024-08-02T23:39:35.891 DEBUG:teuthology.orchestra.run.smithi073:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-08-02T23:39:35.954 INFO:teuthology.orchestra.run.smithi073.stderr:1+0 records in 2024-08-02T23:39:35.954 INFO:teuthology.orchestra.run.smithi073.stderr:1+0 records out 2024-08-02T23:39:35.954 INFO:teuthology.orchestra.run.smithi073.stderr:512 bytes copied, 0.000219657 s, 2.3 MB/s 2024-08-02T23:39:35.955 DEBUG:teuthology.orchestra.run.smithi073:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-08-02T23:39:36.012 DEBUG:teuthology.orchestra.run.smithi073:> stat /dev/vg_nvme/lv_2 2024-08-02T23:39:36.068 INFO:teuthology.orchestra.run.smithi073.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-08-02T23:39:36.068 INFO:teuthology.orchestra.run.smithi073.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-08-02T23:39:36.068 INFO:teuthology.orchestra.run.smithi073.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-08-02T23:39:36.068 INFO:teuthology.orchestra.run.smithi073.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-08-02T23:39:36.068 INFO:teuthology.orchestra.run.smithi073.stdout:Context: system_u:object_r:device_t:s0 2024-08-02T23:39:36.068 INFO:teuthology.orchestra.run.smithi073.stdout:Access: 2024-08-02 23:39:31.167020479 +0000 2024-08-02T23:39:36.068 INFO:teuthology.orchestra.run.smithi073.stdout:Modify: 2024-08-02 23:35:38.139659357 +0000 2024-08-02T23:39:36.068 INFO:teuthology.orchestra.run.smithi073.stdout:Change: 2024-08-02 23:35:38.139659357 +0000 2024-08-02T23:39:36.068 INFO:teuthology.orchestra.run.smithi073.stdout: Birth: 2024-08-02 23:35:38.139659357 +0000 2024-08-02T23:39:36.069 DEBUG:teuthology.orchestra.run.smithi073:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-08-02T23:39:36.130 INFO:teuthology.orchestra.run.smithi073.stderr:1+0 records in 2024-08-02T23:39:36.130 INFO:teuthology.orchestra.run.smithi073.stderr:1+0 records out 2024-08-02T23:39:36.130 INFO:teuthology.orchestra.run.smithi073.stderr:512 bytes copied, 0.000194833 s, 2.6 MB/s 2024-08-02T23:39:36.131 DEBUG:teuthology.orchestra.run.smithi073:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-08-02T23:39:36.187 DEBUG:teuthology.orchestra.run.smithi073:> stat /dev/vg_nvme/lv_3 2024-08-02T23:39:36.242 INFO:teuthology.orchestra.run.smithi073.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-08-02T23:39:36.243 INFO:teuthology.orchestra.run.smithi073.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-08-02T23:39:36.243 INFO:teuthology.orchestra.run.smithi073.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-08-02T23:39:36.243 INFO:teuthology.orchestra.run.smithi073.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-08-02T23:39:36.243 INFO:teuthology.orchestra.run.smithi073.stdout:Context: system_u:object_r:device_t:s0 2024-08-02T23:39:36.243 INFO:teuthology.orchestra.run.smithi073.stdout:Access: 2024-08-02 23:39:31.167020479 +0000 2024-08-02T23:39:36.243 INFO:teuthology.orchestra.run.smithi073.stdout:Modify: 2024-08-02 23:35:38.480651076 +0000 2024-08-02T23:39:36.243 INFO:teuthology.orchestra.run.smithi073.stdout:Change: 2024-08-02 23:35:38.480651076 +0000 2024-08-02T23:39:36.243 INFO:teuthology.orchestra.run.smithi073.stdout: Birth: 2024-08-02 23:35:38.480651076 +0000 2024-08-02T23:39:36.243 DEBUG:teuthology.orchestra.run.smithi073:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-08-02T23:39:36.305 INFO:teuthology.orchestra.run.smithi073.stderr:1+0 records in 2024-08-02T23:39:36.305 INFO:teuthology.orchestra.run.smithi073.stderr:1+0 records out 2024-08-02T23:39:36.305 INFO:teuthology.orchestra.run.smithi073.stderr:512 bytes copied, 0.000160673 s, 3.2 MB/s 2024-08-02T23:39:36.307 DEBUG:teuthology.orchestra.run.smithi073:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-08-02T23:39:36.363 DEBUG:teuthology.orchestra.run.smithi073:> stat /dev/vg_nvme/lv_4 2024-08-02T23:39:36.419 INFO:teuthology.orchestra.run.smithi073.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-08-02T23:39:36.419 INFO:teuthology.orchestra.run.smithi073.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-08-02T23:39:36.419 INFO:teuthology.orchestra.run.smithi073.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-08-02T23:39:36.419 INFO:teuthology.orchestra.run.smithi073.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-08-02T23:39:36.419 INFO:teuthology.orchestra.run.smithi073.stdout:Context: system_u:object_r:device_t:s0 2024-08-02T23:39:36.420 INFO:teuthology.orchestra.run.smithi073.stdout:Access: 2024-08-02 23:39:31.168020455 +0000 2024-08-02T23:39:36.420 INFO:teuthology.orchestra.run.smithi073.stdout:Modify: 2024-08-02 23:35:38.804643207 +0000 2024-08-02T23:39:36.420 INFO:teuthology.orchestra.run.smithi073.stdout:Change: 2024-08-02 23:35:38.804643207 +0000 2024-08-02T23:39:36.420 INFO:teuthology.orchestra.run.smithi073.stdout: Birth: 2024-08-02 23:35:38.804643207 +0000 2024-08-02T23:39:36.420 DEBUG:teuthology.orchestra.run.smithi073:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-08-02T23:39:36.482 INFO:teuthology.orchestra.run.smithi073.stderr:1+0 records in 2024-08-02T23:39:36.482 INFO:teuthology.orchestra.run.smithi073.stderr:1+0 records out 2024-08-02T23:39:36.483 INFO:teuthology.orchestra.run.smithi073.stderr:512 bytes copied, 0.000190947 s, 2.7 MB/s 2024-08-02T23:39:36.484 DEBUG:teuthology.orchestra.run.smithi073:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-08-02T23:39:36.540 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2'} 2024-08-02T23:39:36.540 DEBUG:teuthology.orchestra.run.smithi131:> set -ex 2024-08-02T23:39:36.540 DEBUG:teuthology.orchestra.run.smithi131:> dd if=/scratch_devs of=/dev/stdout 2024-08-02T23:39:36.559 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-08-02T23:39:36.559 DEBUG:teuthology.orchestra.run.smithi131:> stat /dev/vg_nvme/lv_1 2024-08-02T23:39:36.618 INFO:teuthology.orchestra.run.smithi131.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-08-02T23:39:36.618 INFO:teuthology.orchestra.run.smithi131.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-08-02T23:39:36.618 INFO:teuthology.orchestra.run.smithi131.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-08-02T23:39:36.618 INFO:teuthology.orchestra.run.smithi131.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-08-02T23:39:36.618 INFO:teuthology.orchestra.run.smithi131.stdout:Context: system_u:object_r:device_t:s0 2024-08-02T23:39:36.618 INFO:teuthology.orchestra.run.smithi131.stdout:Access: 2024-08-02 23:39:30.769409084 +0000 2024-08-02T23:39:36.618 INFO:teuthology.orchestra.run.smithi131.stdout:Modify: 2024-08-02 23:35:37.815477505 +0000 2024-08-02T23:39:36.618 INFO:teuthology.orchestra.run.smithi131.stdout:Change: 2024-08-02 23:35:37.815477505 +0000 2024-08-02T23:39:36.618 INFO:teuthology.orchestra.run.smithi131.stdout: Birth: 2024-08-02 23:35:37.815477505 +0000 2024-08-02T23:39:36.619 DEBUG:teuthology.orchestra.run.smithi131:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-08-02T23:39:36.685 INFO:teuthology.orchestra.run.smithi131.stderr:1+0 records in 2024-08-02T23:39:36.685 INFO:teuthology.orchestra.run.smithi131.stderr:1+0 records out 2024-08-02T23:39:36.685 INFO:teuthology.orchestra.run.smithi131.stderr:512 bytes copied, 0.000223344 s, 2.3 MB/s 2024-08-02T23:39:36.687 DEBUG:teuthology.orchestra.run.smithi131:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-08-02T23:39:36.746 DEBUG:teuthology.orchestra.run.smithi131:> stat /dev/vg_nvme/lv_2 2024-08-02T23:39:36.806 INFO:teuthology.orchestra.run.smithi131.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-08-02T23:39:36.806 INFO:teuthology.orchestra.run.smithi131.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-08-02T23:39:36.806 INFO:teuthology.orchestra.run.smithi131.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-08-02T23:39:36.806 INFO:teuthology.orchestra.run.smithi131.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-08-02T23:39:36.806 INFO:teuthology.orchestra.run.smithi131.stdout:Context: system_u:object_r:device_t:s0 2024-08-02T23:39:36.806 INFO:teuthology.orchestra.run.smithi131.stdout:Access: 2024-08-02 23:39:30.769409084 +0000 2024-08-02T23:39:36.806 INFO:teuthology.orchestra.run.smithi131.stdout:Modify: 2024-08-02 23:35:38.154470462 +0000 2024-08-02T23:39:36.806 INFO:teuthology.orchestra.run.smithi131.stdout:Change: 2024-08-02 23:35:38.154470462 +0000 2024-08-02T23:39:36.807 INFO:teuthology.orchestra.run.smithi131.stdout: Birth: 2024-08-02 23:35:38.154470462 +0000 2024-08-02T23:39:36.807 DEBUG:teuthology.orchestra.run.smithi131:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-08-02T23:39:36.874 INFO:teuthology.orchestra.run.smithi131.stderr:1+0 records in 2024-08-02T23:39:36.874 INFO:teuthology.orchestra.run.smithi131.stderr:1+0 records out 2024-08-02T23:39:36.874 INFO:teuthology.orchestra.run.smithi131.stderr:512 bytes copied, 0.000236371 s, 2.2 MB/s 2024-08-02T23:39:36.875 DEBUG:teuthology.orchestra.run.smithi131:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-08-02T23:39:36.934 DEBUG:teuthology.orchestra.run.smithi131:> stat /dev/vg_nvme/lv_3 2024-08-02T23:39:36.991 INFO:teuthology.orchestra.run.smithi131.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-08-02T23:39:36.991 INFO:teuthology.orchestra.run.smithi131.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-08-02T23:39:36.991 INFO:teuthology.orchestra.run.smithi131.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-08-02T23:39:36.992 INFO:teuthology.orchestra.run.smithi131.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-08-02T23:39:36.992 INFO:teuthology.orchestra.run.smithi131.stdout:Context: system_u:object_r:device_t:s0 2024-08-02T23:39:36.992 INFO:teuthology.orchestra.run.smithi131.stdout:Access: 2024-08-02 23:39:30.769409084 +0000 2024-08-02T23:39:36.992 INFO:teuthology.orchestra.run.smithi131.stdout:Modify: 2024-08-02 23:35:38.462464063 +0000 2024-08-02T23:39:36.992 INFO:teuthology.orchestra.run.smithi131.stdout:Change: 2024-08-02 23:35:38.462464063 +0000 2024-08-02T23:39:36.992 INFO:teuthology.orchestra.run.smithi131.stdout: Birth: 2024-08-02 23:35:38.462464063 +0000 2024-08-02T23:39:36.992 DEBUG:teuthology.orchestra.run.smithi131:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-08-02T23:39:37.057 INFO:teuthology.orchestra.run.smithi131.stderr:1+0 records in 2024-08-02T23:39:37.057 INFO:teuthology.orchestra.run.smithi131.stderr:1+0 records out 2024-08-02T23:39:37.057 INFO:teuthology.orchestra.run.smithi131.stderr:512 bytes copied, 0.000224115 s, 2.3 MB/s 2024-08-02T23:39:37.059 DEBUG:teuthology.orchestra.run.smithi131:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-08-02T23:39:37.116 DEBUG:teuthology.orchestra.run.smithi131:> stat /dev/vg_nvme/lv_4 2024-08-02T23:39:37.173 INFO:teuthology.orchestra.run.smithi131.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-08-02T23:39:37.173 INFO:teuthology.orchestra.run.smithi131.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-08-02T23:39:37.173 INFO:teuthology.orchestra.run.smithi131.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-08-02T23:39:37.173 INFO:teuthology.orchestra.run.smithi131.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-08-02T23:39:37.173 INFO:teuthology.orchestra.run.smithi131.stdout:Context: system_u:object_r:device_t:s0 2024-08-02T23:39:37.173 INFO:teuthology.orchestra.run.smithi131.stdout:Access: 2024-08-02 23:39:30.769409084 +0000 2024-08-02T23:39:37.173 INFO:teuthology.orchestra.run.smithi131.stdout:Modify: 2024-08-02 23:35:38.796457124 +0000 2024-08-02T23:39:37.173 INFO:teuthology.orchestra.run.smithi131.stdout:Change: 2024-08-02 23:35:38.796457124 +0000 2024-08-02T23:39:37.173 INFO:teuthology.orchestra.run.smithi131.stdout: Birth: 2024-08-02 23:35:38.796457124 +0000 2024-08-02T23:39:37.174 DEBUG:teuthology.orchestra.run.smithi131:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-08-02T23:39:37.238 INFO:teuthology.orchestra.run.smithi131.stderr:1+0 records in 2024-08-02T23:39:37.238 INFO:teuthology.orchestra.run.smithi131.stderr:1+0 records out 2024-08-02T23:39:37.238 INFO:teuthology.orchestra.run.smithi131.stderr:512 bytes copied, 0.000188794 s, 2.7 MB/s 2024-08-02T23:39:37.239 DEBUG:teuthology.orchestra.run.smithi131:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-08-02T23:39:37.296 INFO:tasks.ceph:osd dev map: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.3': '/dev/vg_nvme/lv_2'} 2024-08-02T23:39:37.296 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi073.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2'}, Remote(name='ubuntu@smithi131.front.sepia.ceph.com'): {'osd.2': '/dev/vg_nvme/lv_1', 'osd.3': '/dev/vg_nvme/lv_2'}} 2024-08-02T23:39:37.296 INFO:tasks.ceph:Generating config... 2024-08-02T23:39:37.297 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2024-08-02T23:39:37.298 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-08-02T23:39:37.298 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-08-02T23:39:37.298 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-08-02T23:39:37.298 INFO:tasks.ceph:[mon] paxos service trim min = 10 2024-08-02T23:39:37.298 INFO:tasks.ceph:[mon] debug mon = 20 2024-08-02T23:39:37.298 INFO:tasks.ceph:[mon] debug ms = 1 2024-08-02T23:39:37.298 INFO:tasks.ceph:[mon] debug paxos = 20 2024-08-02T23:39:37.298 INFO:tasks.ceph:[mon] mon warn on pool no app = False 2024-08-02T23:39:37.298 INFO:tasks.ceph:[mon] osd pool default crimson = True 2024-08-02T23:39:37.298 INFO:tasks.ceph:[osd] debug monc = 20 2024-08-02T23:39:37.298 INFO:tasks.ceph:[osd] crimson alien thread cpu cores = 6-7 2024-08-02T23:39:37.298 INFO:tasks.ceph:[osd] crimson osd obc lru size = 10 2024-08-02T23:39:37.299 INFO:tasks.ceph:[osd] debug ms = 1 2024-08-02T23:39:37.299 INFO:tasks.ceph:[osd] debug osd = 20 2024-08-02T23:39:37.299 INFO:tasks.ceph:[osd] osd blocked scrub grace period = 3600 2024-08-02T23:39:37.299 INFO:tasks.ceph:[osd] osd class default list = * 2024-08-02T23:39:37.299 INFO:tasks.ceph:[osd] osd class load list = * 2024-08-02T23:39:37.299 INFO:tasks.ceph:[osd] osd objectstore = seastore 2024-08-02T23:39:37.299 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-08-02T23:39:37.299 INFO:tasks.ceph:[client] debug ms = 1 2024-08-02T23:39:37.299 INFO:tasks.ceph:[global] enable experimental unrecoverable data corrupting features = crimson 2024-08-02T23:39:37.299 INFO:tasks.ceph:[global] ms client mode = crc 2024-08-02T23:39:37.299 INFO:tasks.ceph:[global] ms cluster mode = crc 2024-08-02T23:39:37.299 INFO:tasks.ceph:[global] ms mon client mode = crc 2024-08-02T23:39:37.299 INFO:tasks.ceph:[global] ms mon cluster mode = crc 2024-08-02T23:39:37.299 INFO:tasks.ceph:[global] ms mon service mode = crc 2024-08-02T23:39:37.300 INFO:tasks.ceph:[global] ms service mode = crc 2024-08-02T23:39:37.300 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-08-02T23:39:37.300 INFO:tasks.ceph:[mgr] debug ms = 1 2024-08-02T23:39:37.300 INFO:tasks.ceph:[mgr] mgr stats period = 30 2024-08-02T23:39:37.300 INFO:tasks.ceph:[osd.0] crimson seastar cpu cores = 0-2 2024-08-02T23:39:37.300 INFO:tasks.ceph:[osd.1] crimson seastar cpu cores = 3-5 2024-08-02T23:39:37.300 INFO:tasks.ceph:[osd.2] crimson seastar cpu cores = 0-2 2024-08-02T23:39:37.300 INFO:tasks.ceph:[osd.3] crimson seastar cpu cores = 3-5 2024-08-02T23:39:37.300 INFO:tasks.ceph:Setting up mon.a... 2024-08-02T23:39:37.300 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-08-02T23:39:37.362 INFO:teuthology.orchestra.run.smithi073.stdout:creating /etc/ceph/ceph.keyring 2024-08-02T23:39:37.365 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-08-02T23:39:37.416 DEBUG:teuthology.orchestra.run.smithi073:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-08-02T23:39:37.441 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.73')] 2024-08-02T23:39:37.441 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.73', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug monc': 20, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug ms': 1, 'debug osd': 20, 'osd blocked scrub grace period': 3600, 'osd class default list': '*', 'osd class load list': '*', 'osd objectstore': 'seastore', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false', 'mgr stats period': 30}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10, 'mon warn on pool no app': False, 'osd pool default crimson': True}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug ms': 1}, 'mon.a': {}, 'osd.0': {'crimson seastar cpu cores': '0-2'}, 'osd.1': {'crimson seastar cpu cores': '3-5'}, 'osd.2': {'crimson seastar cpu cores': '0-2'}, 'osd.3': {'crimson seastar cpu cores': '3-5'}} 2024-08-02T23:39:37.442 DEBUG:teuthology.orchestra.run.smithi073:> set -ex 2024-08-02T23:39:37.442 DEBUG:teuthology.orchestra.run.smithi073:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-08-02T23:39:37.497 DEBUG:teuthology.orchestra.run.smithi073:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.73 --print /home/ubuntu/cephtest/ceph.monmap 2024-08-02T23:39:37.585 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:39:37.583+0000 7fcaca243e80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-02T23:39:37.585 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:39:37.583+0000 7fcaca243e80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-02T23:39:37.585 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:39:37.583+0000 7fcaca243e80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-02T23:39:37.586 INFO:teuthology.orchestra.run.smithi073.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-08-02T23:39:37.586 INFO:teuthology.orchestra.run.smithi073.stdout:monmaptool: generated fsid 7caeb70d-fd31-4bdf-94e1-d0f19ea8e3c6 2024-08-02T23:39:37.586 INFO:teuthology.orchestra.run.smithi073.stdout:setting min_mon_release = quincy 2024-08-02T23:39:37.586 INFO:teuthology.orchestra.run.smithi073.stdout:epoch 0 2024-08-02T23:39:37.586 INFO:teuthology.orchestra.run.smithi073.stdout:fsid 7caeb70d-fd31-4bdf-94e1-d0f19ea8e3c6 2024-08-02T23:39:37.586 INFO:teuthology.orchestra.run.smithi073.stdout:last_changed 2024-08-02T23:39:37.584701+0000 2024-08-02T23:39:37.586 INFO:teuthology.orchestra.run.smithi073.stdout:created 2024-08-02T23:39:37.584701+0000 2024-08-02T23:39:37.586 INFO:teuthology.orchestra.run.smithi073.stdout:min_mon_release 17 (quincy) 2024-08-02T23:39:37.586 INFO:teuthology.orchestra.run.smithi073.stdout:election_strategy: 1 2024-08-02T23:39:37.587 INFO:teuthology.orchestra.run.smithi073.stdout:0: [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] mon.a 2024-08-02T23:39:37.587 INFO:teuthology.orchestra.run.smithi073.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (1 monitors) 2024-08-02T23:39:37.588 DEBUG:teuthology.orchestra.run.smithi073:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-08-02T23:39:37.603 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 7caeb70d-fd31-4bdf-94e1-d0f19ea8e3c6... 2024-08-02T23:39:37.604 DEBUG:teuthology.orchestra.run.smithi073:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-08-02T23:39:37.645 DEBUG:teuthology.orchestra.run.smithi131:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-08-02T23:39:37.685 INFO:teuthology.orchestra.run.smithi073.stdout:[global] 2024-08-02T23:39:37.685 INFO:teuthology.orchestra.run.smithi073.stdout: chdir = "" 2024-08-02T23:39:37.685 INFO:teuthology.orchestra.run.smithi073.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-08-02T23:39:37.685 INFO:teuthology.orchestra.run.smithi073.stdout: auth supported = cephx 2024-08-02T23:39:37.686 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-08-02T23:39:37.686 INFO:teuthology.orchestra.run.smithi073.stdout: filestore xattr use omap = true 2024-08-02T23:39:37.686 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-08-02T23:39:37.686 INFO:teuthology.orchestra.run.smithi073.stdout: mon clock drift allowed = 1.000 2024-08-02T23:39:37.686 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-08-02T23:39:37.686 INFO:teuthology.orchestra.run.smithi073.stdout: osd crush chooseleaf type = 0 2024-08-02T23:39:37.686 INFO:teuthology.orchestra.run.smithi073.stdout: auth debug = true 2024-08-02T23:39:37.686 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-08-02T23:39:37.686 INFO:teuthology.orchestra.run.smithi073.stdout: ms die on old message = true 2024-08-02T23:39:37.686 INFO:teuthology.orchestra.run.smithi073.stdout: ms die on bug = true 2024-08-02T23:39:37.686 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-08-02T23:39:37.686 INFO:teuthology.orchestra.run.smithi073.stdout: mon max pg per osd = 10000 # >= luminous 2024-08-02T23:39:37.686 INFO:teuthology.orchestra.run.smithi073.stdout: mon pg warn max object skew = 0 2024-08-02T23:39:37.687 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-08-02T23:39:37.687 INFO:teuthology.orchestra.run.smithi073.stdout: # disable pg_autoscaler by default for new pools 2024-08-02T23:39:37.687 INFO:teuthology.orchestra.run.smithi073.stdout: osd_pool_default_pg_autoscale_mode = off 2024-08-02T23:39:37.687 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-08-02T23:39:37.687 INFO:teuthology.orchestra.run.smithi073.stdout: osd pool default size = 2 2024-08-02T23:39:37.687 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-08-02T23:39:37.687 INFO:teuthology.orchestra.run.smithi073.stdout: mon osd allow primary affinity = true 2024-08-02T23:39:37.687 INFO:teuthology.orchestra.run.smithi073.stdout: mon osd allow pg remap = true 2024-08-02T23:39:37.687 INFO:teuthology.orchestra.run.smithi073.stdout: mon warn on legacy crush tunables = false 2024-08-02T23:39:37.687 INFO:teuthology.orchestra.run.smithi073.stdout: mon warn on crush straw calc version zero = false 2024-08-02T23:39:37.687 INFO:teuthology.orchestra.run.smithi073.stdout: mon warn on no sortbitwise = false 2024-08-02T23:39:37.687 INFO:teuthology.orchestra.run.smithi073.stdout: mon warn on osd down out interval zero = false 2024-08-02T23:39:37.687 INFO:teuthology.orchestra.run.smithi073.stdout: mon warn on too few osds = false 2024-08-02T23:39:37.687 INFO:teuthology.orchestra.run.smithi073.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-08-02T23:39:37.688 INFO:teuthology.orchestra.run.smithi073.stdout: mon_warn_on_pool_no_redundancy = false 2024-08-02T23:39:37.688 INFO:teuthology.orchestra.run.smithi073.stdout: mon_allow_pool_size_one = true 2024-08-02T23:39:37.688 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-08-02T23:39:37.688 INFO:teuthology.orchestra.run.smithi073.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-08-02T23:39:37.688 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-08-02T23:39:37.688 INFO:teuthology.orchestra.run.smithi073.stdout: osd default data pool replay window = 5 2024-08-02T23:39:37.688 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-08-02T23:39:37.688 INFO:teuthology.orchestra.run.smithi073.stdout: mon allow pool delete = true 2024-08-02T23:39:37.688 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-08-02T23:39:37.688 INFO:teuthology.orchestra.run.smithi073.stdout: mon cluster log file level = debug 2024-08-02T23:39:37.688 INFO:teuthology.orchestra.run.smithi073.stdout: debug asserts on shutdown = true 2024-08-02T23:39:37.688 INFO:teuthology.orchestra.run.smithi073.stdout: mon health detail to clog = false 2024-08-02T23:39:37.688 INFO:teuthology.orchestra.run.smithi073.stdout: mon host = 172.21.15.73 2024-08-02T23:39:37.688 INFO:teuthology.orchestra.run.smithi073.stdout: enable experimental unrecoverable data corrupting features = crimson 2024-08-02T23:39:37.689 INFO:teuthology.orchestra.run.smithi073.stdout: ms client mode = crc 2024-08-02T23:39:37.689 INFO:teuthology.orchestra.run.smithi073.stdout: ms cluster mode = crc 2024-08-02T23:39:37.689 INFO:teuthology.orchestra.run.smithi073.stdout: ms mon client mode = crc 2024-08-02T23:39:37.689 INFO:teuthology.orchestra.run.smithi073.stdout: ms mon cluster mode = crc 2024-08-02T23:39:37.689 INFO:teuthology.orchestra.run.smithi073.stdout: ms mon service mode = crc 2024-08-02T23:39:37.689 INFO:teuthology.orchestra.run.smithi073.stdout: ms service mode = crc 2024-08-02T23:39:37.689 INFO:teuthology.orchestra.run.smithi073.stdout: fsid = 7caeb70d-fd31-4bdf-94e1-d0f19ea8e3c6 2024-08-02T23:39:37.689 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-08-02T23:39:37.689 INFO:teuthology.orchestra.run.smithi073.stdout:[osd] 2024-08-02T23:39:37.689 INFO:teuthology.orchestra.run.smithi073.stdout: osd journal size = 100 2024-08-02T23:39:37.689 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-08-02T23:39:37.689 INFO:teuthology.orchestra.run.smithi073.stdout: osd scrub load threshold = 5.0 2024-08-02T23:39:37.689 INFO:teuthology.orchestra.run.smithi073.stdout: osd scrub max interval = 600 2024-08-02T23:39:37.689 INFO:teuthology.orchestra.run.smithi073.stdout: osd mclock profile = high_recovery_ops 2024-08-02T23:39:37.689 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-08-02T23:39:37.690 INFO:teuthology.orchestra.run.smithi073.stdout: osd recover clone overlap = true 2024-08-02T23:39:37.690 INFO:teuthology.orchestra.run.smithi073.stdout: osd recovery max chunk = 1048576 2024-08-02T23:39:37.690 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-08-02T23:39:37.690 INFO:teuthology.orchestra.run.smithi073.stdout: osd debug shutdown = true 2024-08-02T23:39:37.690 INFO:teuthology.orchestra.run.smithi073.stdout: osd debug op order = true 2024-08-02T23:39:37.690 INFO:teuthology.orchestra.run.smithi073.stdout: osd debug verify stray on activate = true 2024-08-02T23:39:37.690 INFO:teuthology.orchestra.run.smithi073.stdout: osd debug trim objects = true 2024-08-02T23:39:37.690 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-08-02T23:39:37.690 INFO:teuthology.orchestra.run.smithi073.stdout: osd open classes on start = true 2024-08-02T23:39:37.690 INFO:teuthology.orchestra.run.smithi073.stdout: osd debug pg log writeout = true 2024-08-02T23:39:37.690 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-08-02T23:39:37.690 INFO:teuthology.orchestra.run.smithi073.stdout: osd deep scrub update digest min age = 30 2024-08-02T23:39:37.690 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-08-02T23:39:37.691 INFO:teuthology.orchestra.run.smithi073.stdout: osd map max advance = 10 2024-08-02T23:39:37.691 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-08-02T23:39:37.691 INFO:teuthology.orchestra.run.smithi073.stdout: journal zero on create = true 2024-08-02T23:39:37.691 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-08-02T23:39:37.691 INFO:teuthology.orchestra.run.smithi073.stdout: filestore ondisk finisher threads = 3 2024-08-02T23:39:37.691 INFO:teuthology.orchestra.run.smithi073.stdout: filestore apply finisher threads = 3 2024-08-02T23:39:37.691 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-08-02T23:39:37.691 INFO:teuthology.orchestra.run.smithi073.stdout: bdev debug aio = true 2024-08-02T23:39:37.691 INFO:teuthology.orchestra.run.smithi073.stdout: osd debug misdirected ops = true 2024-08-02T23:39:37.691 INFO:teuthology.orchestra.run.smithi073.stdout: debug monc = 20 2024-08-02T23:39:37.691 INFO:teuthology.orchestra.run.smithi073.stdout: crimson alien thread cpu cores = 6-7 2024-08-02T23:39:37.691 INFO:teuthology.orchestra.run.smithi073.stdout: crimson osd obc lru size = 10 2024-08-02T23:39:37.691 INFO:teuthology.orchestra.run.smithi073.stdout: debug ms = 1 2024-08-02T23:39:37.691 INFO:teuthology.orchestra.run.smithi073.stdout: debug osd = 20 2024-08-02T23:39:37.692 INFO:teuthology.orchestra.run.smithi073.stdout: osd blocked scrub grace period = 3600 2024-08-02T23:39:37.692 INFO:teuthology.orchestra.run.smithi073.stdout: osd class default list = * 2024-08-02T23:39:37.692 INFO:teuthology.orchestra.run.smithi073.stdout: osd class load list = * 2024-08-02T23:39:37.692 INFO:teuthology.orchestra.run.smithi073.stdout: osd objectstore = seastore 2024-08-02T23:39:37.692 INFO:teuthology.orchestra.run.smithi073.stdout: osd shutdown pgref assert = True 2024-08-02T23:39:37.692 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-08-02T23:39:37.692 INFO:teuthology.orchestra.run.smithi073.stdout:[mgr] 2024-08-02T23:39:37.692 INFO:teuthology.orchestra.run.smithi073.stdout: debug ms = 1 2024-08-02T23:39:37.692 INFO:teuthology.orchestra.run.smithi073.stdout: debug mgr = 20 2024-08-02T23:39:37.692 INFO:teuthology.orchestra.run.smithi073.stdout: debug mon = 20 2024-08-02T23:39:37.692 INFO:teuthology.orchestra.run.smithi073.stdout: debug auth = 20 2024-08-02T23:39:37.692 INFO:teuthology.orchestra.run.smithi073.stdout: mon reweight min pgs per osd = 4 2024-08-02T23:39:37.692 INFO:teuthology.orchestra.run.smithi073.stdout: mon reweight min bytes per osd = 10 2024-08-02T23:39:37.692 INFO:teuthology.orchestra.run.smithi073.stdout: mgr/telemetry/nag = false 2024-08-02T23:39:37.692 INFO:teuthology.orchestra.run.smithi073.stdout: mgr stats period = 30 2024-08-02T23:39:37.692 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-08-02T23:39:37.693 INFO:teuthology.orchestra.run.smithi073.stdout:[mon] 2024-08-02T23:39:37.693 INFO:teuthology.orchestra.run.smithi073.stdout: debug ms = 1 2024-08-02T23:39:37.693 INFO:teuthology.orchestra.run.smithi073.stdout: debug mon = 20 2024-08-02T23:39:37.693 INFO:teuthology.orchestra.run.smithi073.stdout: debug paxos = 20 2024-08-02T23:39:37.693 INFO:teuthology.orchestra.run.smithi073.stdout: debug auth = 20 2024-08-02T23:39:37.693 INFO:teuthology.orchestra.run.smithi073.stdout: mon data avail warn = 5 2024-08-02T23:39:37.693 INFO:teuthology.orchestra.run.smithi073.stdout: mon mgr mkfs grace = 240 2024-08-02T23:39:37.693 INFO:teuthology.orchestra.run.smithi073.stdout: mon reweight min pgs per osd = 4 2024-08-02T23:39:37.693 INFO:teuthology.orchestra.run.smithi073.stdout: mon osd reporter subtree level = osd 2024-08-02T23:39:37.693 INFO:teuthology.orchestra.run.smithi073.stdout: mon osd prime pg temp = true 2024-08-02T23:39:37.693 INFO:teuthology.orchestra.run.smithi073.stdout: mon reweight min bytes per osd = 10 2024-08-02T23:39:37.693 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-08-02T23:39:37.693 INFO:teuthology.orchestra.run.smithi073.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-08-02T23:39:37.693 INFO:teuthology.orchestra.run.smithi073.stdout: auth mon ticket ttl = 660 # 11m 2024-08-02T23:39:37.693 INFO:teuthology.orchestra.run.smithi073.stdout: auth service ticket ttl = 240 # 4m 2024-08-02T23:39:37.693 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-08-02T23:39:37.694 INFO:teuthology.orchestra.run.smithi073.stdout: # don't complain about insecure global_id in the test suite 2024-08-02T23:39:37.694 INFO:teuthology.orchestra.run.smithi073.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-08-02T23:39:37.694 INFO:teuthology.orchestra.run.smithi073.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-08-02T23:39:37.694 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-08-02T23:39:37.694 INFO:teuthology.orchestra.run.smithi073.stdout: # 1m isn't quite enough 2024-08-02T23:39:37.694 INFO:teuthology.orchestra.run.smithi073.stdout: mon_down_mkfs_grace = 2m 2024-08-02T23:39:37.694 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-08-02T23:39:37.694 INFO:teuthology.orchestra.run.smithi073.stdout: mon_warn_on_filestore_osds = false 2024-08-02T23:39:37.694 INFO:teuthology.orchestra.run.smithi073.stdout: mon min osdmap epochs = 50 2024-08-02T23:39:37.694 INFO:teuthology.orchestra.run.smithi073.stdout: mon osdmap full prune interval = 2 2024-08-02T23:39:37.694 INFO:teuthology.orchestra.run.smithi073.stdout: mon osdmap full prune min = 15 2024-08-02T23:39:37.694 INFO:teuthology.orchestra.run.smithi073.stdout: mon osdmap full prune txsize = 2 2024-08-02T23:39:37.694 INFO:teuthology.orchestra.run.smithi073.stdout: paxos service trim min = 10 2024-08-02T23:39:37.694 INFO:teuthology.orchestra.run.smithi073.stdout: mon warn on pool no app = False 2024-08-02T23:39:37.695 INFO:teuthology.orchestra.run.smithi073.stdout: osd pool default crimson = True 2024-08-02T23:39:37.695 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-08-02T23:39:37.695 INFO:teuthology.orchestra.run.smithi073.stdout:[client] 2024-08-02T23:39:37.695 INFO:teuthology.orchestra.run.smithi073.stdout: rgw cache enabled = true 2024-08-02T23:39:37.695 INFO:teuthology.orchestra.run.smithi073.stdout: rgw enable ops log = true 2024-08-02T23:39:37.695 INFO:teuthology.orchestra.run.smithi073.stdout: rgw enable usage log = true 2024-08-02T23:39:37.695 INFO:teuthology.orchestra.run.smithi073.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-08-02T23:39:37.695 INFO:teuthology.orchestra.run.smithi073.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-08-02T23:39:37.695 INFO:teuthology.orchestra.run.smithi073.stdout: debug ms = 1 2024-08-02T23:39:37.695 INFO:teuthology.orchestra.run.smithi073.stdout:[mon.a] 2024-08-02T23:39:37.695 INFO:teuthology.orchestra.run.smithi073.stdout:[osd.0] 2024-08-02T23:39:37.695 INFO:teuthology.orchestra.run.smithi073.stdout: crimson seastar cpu cores = 0-2 2024-08-02T23:39:37.695 INFO:teuthology.orchestra.run.smithi073.stdout:[osd.1] 2024-08-02T23:39:37.695 INFO:teuthology.orchestra.run.smithi073.stdout: crimson seastar cpu cores = 3-5 2024-08-02T23:39:37.696 INFO:teuthology.orchestra.run.smithi073.stdout:[osd.2] 2024-08-02T23:39:37.696 INFO:teuthology.orchestra.run.smithi073.stdout: crimson seastar cpu cores = 0-2 2024-08-02T23:39:37.696 INFO:teuthology.orchestra.run.smithi073.stdout:[osd.3] 2024-08-02T23:39:37.696 INFO:teuthology.orchestra.run.smithi073.stdout: crimson seastar cpu cores = 3-5 2024-08-02T23:39:37.696 INFO:teuthology.orchestra.run.smithi131.stdout:[global] 2024-08-02T23:39:37.696 INFO:teuthology.orchestra.run.smithi131.stdout: chdir = "" 2024-08-02T23:39:37.696 INFO:teuthology.orchestra.run.smithi131.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-08-02T23:39:37.696 INFO:teuthology.orchestra.run.smithi131.stdout: auth supported = cephx 2024-08-02T23:39:37.697 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-08-02T23:39:37.697 INFO:teuthology.orchestra.run.smithi131.stdout: filestore xattr use omap = true 2024-08-02T23:39:37.697 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-08-02T23:39:37.697 INFO:teuthology.orchestra.run.smithi131.stdout: mon clock drift allowed = 1.000 2024-08-02T23:39:37.697 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-08-02T23:39:37.697 INFO:teuthology.orchestra.run.smithi131.stdout: osd crush chooseleaf type = 0 2024-08-02T23:39:37.697 INFO:teuthology.orchestra.run.smithi131.stdout: auth debug = true 2024-08-02T23:39:37.697 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-08-02T23:39:37.697 INFO:teuthology.orchestra.run.smithi131.stdout: ms die on old message = true 2024-08-02T23:39:37.697 INFO:teuthology.orchestra.run.smithi131.stdout: ms die on bug = true 2024-08-02T23:39:37.697 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-08-02T23:39:37.697 INFO:teuthology.orchestra.run.smithi131.stdout: mon max pg per osd = 10000 # >= luminous 2024-08-02T23:39:37.697 INFO:teuthology.orchestra.run.smithi131.stdout: mon pg warn max object skew = 0 2024-08-02T23:39:37.697 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-08-02T23:39:37.698 INFO:teuthology.orchestra.run.smithi131.stdout: # disable pg_autoscaler by default for new pools 2024-08-02T23:39:37.698 INFO:teuthology.orchestra.run.smithi131.stdout: osd_pool_default_pg_autoscale_mode = off 2024-08-02T23:39:37.698 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-08-02T23:39:37.698 INFO:teuthology.orchestra.run.smithi131.stdout: osd pool default size = 2 2024-08-02T23:39:37.698 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-08-02T23:39:37.698 INFO:teuthology.orchestra.run.smithi131.stdout: mon osd allow primary affinity = true 2024-08-02T23:39:37.698 INFO:teuthology.orchestra.run.smithi131.stdout: mon osd allow pg remap = true 2024-08-02T23:39:37.698 INFO:teuthology.orchestra.run.smithi131.stdout: mon warn on legacy crush tunables = false 2024-08-02T23:39:37.698 INFO:teuthology.orchestra.run.smithi131.stdout: mon warn on crush straw calc version zero = false 2024-08-02T23:39:37.698 INFO:teuthology.orchestra.run.smithi131.stdout: mon warn on no sortbitwise = false 2024-08-02T23:39:37.698 INFO:teuthology.orchestra.run.smithi131.stdout: mon warn on osd down out interval zero = false 2024-08-02T23:39:37.698 INFO:teuthology.orchestra.run.smithi131.stdout: mon warn on too few osds = false 2024-08-02T23:39:37.698 INFO:teuthology.orchestra.run.smithi131.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-08-02T23:39:37.699 INFO:teuthology.orchestra.run.smithi131.stdout: mon_warn_on_pool_no_redundancy = false 2024-08-02T23:39:37.699 INFO:teuthology.orchestra.run.smithi131.stdout: mon_allow_pool_size_one = true 2024-08-02T23:39:37.699 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-08-02T23:39:37.699 INFO:teuthology.orchestra.run.smithi131.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-08-02T23:39:37.699 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-08-02T23:39:37.699 INFO:teuthology.orchestra.run.smithi131.stdout: osd default data pool replay window = 5 2024-08-02T23:39:37.699 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-08-02T23:39:37.699 INFO:teuthology.orchestra.run.smithi131.stdout: mon allow pool delete = true 2024-08-02T23:39:37.699 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-08-02T23:39:37.699 INFO:teuthology.orchestra.run.smithi131.stdout: mon cluster log file level = debug 2024-08-02T23:39:37.699 INFO:teuthology.orchestra.run.smithi131.stdout: debug asserts on shutdown = true 2024-08-02T23:39:37.699 INFO:teuthology.orchestra.run.smithi131.stdout: mon health detail to clog = false 2024-08-02T23:39:37.699 INFO:teuthology.orchestra.run.smithi131.stdout: mon host = 172.21.15.73 2024-08-02T23:39:37.700 INFO:teuthology.orchestra.run.smithi131.stdout: enable experimental unrecoverable data corrupting features = crimson 2024-08-02T23:39:37.700 INFO:teuthology.orchestra.run.smithi131.stdout: ms client mode = crc 2024-08-02T23:39:37.700 INFO:teuthology.orchestra.run.smithi131.stdout: ms cluster mode = crc 2024-08-02T23:39:37.700 INFO:teuthology.orchestra.run.smithi131.stdout: ms mon client mode = crc 2024-08-02T23:39:37.700 INFO:teuthology.orchestra.run.smithi131.stdout: ms mon cluster mode = crc 2024-08-02T23:39:37.700 INFO:teuthology.orchestra.run.smithi131.stdout: ms mon service mode = crc 2024-08-02T23:39:37.700 INFO:teuthology.orchestra.run.smithi131.stdout: ms service mode = crc 2024-08-02T23:39:37.700 INFO:teuthology.orchestra.run.smithi131.stdout: fsid = 7caeb70d-fd31-4bdf-94e1-d0f19ea8e3c6 2024-08-02T23:39:37.700 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-08-02T23:39:37.700 INFO:teuthology.orchestra.run.smithi131.stdout:[osd] 2024-08-02T23:39:37.700 INFO:teuthology.orchestra.run.smithi131.stdout: osd journal size = 100 2024-08-02T23:39:37.700 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-08-02T23:39:37.700 INFO:teuthology.orchestra.run.smithi131.stdout: osd scrub load threshold = 5.0 2024-08-02T23:39:37.700 INFO:teuthology.orchestra.run.smithi131.stdout: osd scrub max interval = 600 2024-08-02T23:39:37.701 INFO:teuthology.orchestra.run.smithi131.stdout: osd mclock profile = high_recovery_ops 2024-08-02T23:39:37.701 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-08-02T23:39:37.701 INFO:teuthology.orchestra.run.smithi131.stdout: osd recover clone overlap = true 2024-08-02T23:39:37.701 INFO:teuthology.orchestra.run.smithi131.stdout: osd recovery max chunk = 1048576 2024-08-02T23:39:37.701 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-08-02T23:39:37.701 INFO:teuthology.orchestra.run.smithi131.stdout: osd debug shutdown = true 2024-08-02T23:39:37.701 INFO:teuthology.orchestra.run.smithi131.stdout: osd debug op order = true 2024-08-02T23:39:37.701 INFO:teuthology.orchestra.run.smithi131.stdout: osd debug verify stray on activate = true 2024-08-02T23:39:37.701 INFO:teuthology.orchestra.run.smithi131.stdout: osd debug trim objects = true 2024-08-02T23:39:37.701 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-08-02T23:39:37.701 INFO:teuthology.orchestra.run.smithi131.stdout: osd open classes on start = true 2024-08-02T23:39:37.701 INFO:teuthology.orchestra.run.smithi131.stdout: osd debug pg log writeout = true 2024-08-02T23:39:37.701 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-08-02T23:39:37.701 INFO:teuthology.orchestra.run.smithi131.stdout: osd deep scrub update digest min age = 30 2024-08-02T23:39:37.701 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-08-02T23:39:37.702 INFO:teuthology.orchestra.run.smithi131.stdout: osd map max advance = 10 2024-08-02T23:39:37.702 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-08-02T23:39:37.702 INFO:teuthology.orchestra.run.smithi131.stdout: journal zero on create = true 2024-08-02T23:39:37.702 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-08-02T23:39:37.702 INFO:teuthology.orchestra.run.smithi131.stdout: filestore ondisk finisher threads = 3 2024-08-02T23:39:37.702 INFO:teuthology.orchestra.run.smithi131.stdout: filestore apply finisher threads = 3 2024-08-02T23:39:37.702 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-08-02T23:39:37.702 INFO:teuthology.orchestra.run.smithi131.stdout: bdev debug aio = true 2024-08-02T23:39:37.702 INFO:teuthology.orchestra.run.smithi131.stdout: osd debug misdirected ops = true 2024-08-02T23:39:37.702 INFO:teuthology.orchestra.run.smithi131.stdout: debug monc = 20 2024-08-02T23:39:37.702 INFO:teuthology.orchestra.run.smithi131.stdout: crimson alien thread cpu cores = 6-7 2024-08-02T23:39:37.702 INFO:teuthology.orchestra.run.smithi131.stdout: crimson osd obc lru size = 10 2024-08-02T23:39:37.702 INFO:teuthology.orchestra.run.smithi131.stdout: debug ms = 1 2024-08-02T23:39:37.702 INFO:teuthology.orchestra.run.smithi131.stdout: debug osd = 20 2024-08-02T23:39:37.703 INFO:teuthology.orchestra.run.smithi131.stdout: osd blocked scrub grace period = 3600 2024-08-02T23:39:37.703 INFO:teuthology.orchestra.run.smithi131.stdout: osd class default list = * 2024-08-02T23:39:37.703 INFO:teuthology.orchestra.run.smithi131.stdout: osd class load list = * 2024-08-02T23:39:37.703 INFO:teuthology.orchestra.run.smithi131.stdout: osd objectstore = seastore 2024-08-02T23:39:37.703 INFO:teuthology.orchestra.run.smithi131.stdout: osd shutdown pgref assert = True 2024-08-02T23:39:37.703 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-08-02T23:39:37.703 INFO:teuthology.orchestra.run.smithi131.stdout:[mgr] 2024-08-02T23:39:37.703 INFO:teuthology.orchestra.run.smithi131.stdout: debug ms = 1 2024-08-02T23:39:37.703 INFO:teuthology.orchestra.run.smithi131.stdout: debug mgr = 20 2024-08-02T23:39:37.703 INFO:teuthology.orchestra.run.smithi131.stdout: debug mon = 20 2024-08-02T23:39:37.703 INFO:teuthology.orchestra.run.smithi131.stdout: debug auth = 20 2024-08-02T23:39:37.703 INFO:teuthology.orchestra.run.smithi131.stdout: mon reweight min pgs per osd = 4 2024-08-02T23:39:37.703 INFO:teuthology.orchestra.run.smithi131.stdout: mon reweight min bytes per osd = 10 2024-08-02T23:39:37.703 INFO:teuthology.orchestra.run.smithi131.stdout: mgr/telemetry/nag = false 2024-08-02T23:39:37.703 INFO:teuthology.orchestra.run.smithi131.stdout: mgr stats period = 30 2024-08-02T23:39:37.704 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-08-02T23:39:37.704 INFO:teuthology.orchestra.run.smithi131.stdout:[mon] 2024-08-02T23:39:37.704 INFO:teuthology.orchestra.run.smithi131.stdout: debug ms = 1 2024-08-02T23:39:37.704 INFO:teuthology.orchestra.run.smithi131.stdout: debug mon = 20 2024-08-02T23:39:37.704 INFO:teuthology.orchestra.run.smithi131.stdout: debug paxos = 20 2024-08-02T23:39:37.704 INFO:teuthology.orchestra.run.smithi131.stdout: debug auth = 20 2024-08-02T23:39:37.704 INFO:teuthology.orchestra.run.smithi131.stdout: mon data avail warn = 5 2024-08-02T23:39:37.704 INFO:teuthology.orchestra.run.smithi131.stdout: mon mgr mkfs grace = 240 2024-08-02T23:39:37.704 INFO:teuthology.orchestra.run.smithi131.stdout: mon reweight min pgs per osd = 4 2024-08-02T23:39:37.704 INFO:teuthology.orchestra.run.smithi131.stdout: mon osd reporter subtree level = osd 2024-08-02T23:39:37.704 INFO:teuthology.orchestra.run.smithi131.stdout: mon osd prime pg temp = true 2024-08-02T23:39:37.704 INFO:teuthology.orchestra.run.smithi131.stdout: mon reweight min bytes per osd = 10 2024-08-02T23:39:37.704 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-08-02T23:39:37.704 INFO:teuthology.orchestra.run.smithi131.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-08-02T23:39:37.704 INFO:teuthology.orchestra.run.smithi131.stdout: auth mon ticket ttl = 660 # 11m 2024-08-02T23:39:37.704 INFO:teuthology.orchestra.run.smithi131.stdout: auth service ticket ttl = 240 # 4m 2024-08-02T23:39:37.705 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-08-02T23:39:37.705 INFO:teuthology.orchestra.run.smithi131.stdout: # don't complain about insecure global_id in the test suite 2024-08-02T23:39:37.705 INFO:teuthology.orchestra.run.smithi131.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-08-02T23:39:37.705 INFO:teuthology.orchestra.run.smithi131.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-08-02T23:39:37.705 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-08-02T23:39:37.705 INFO:teuthology.orchestra.run.smithi131.stdout: # 1m isn't quite enough 2024-08-02T23:39:37.705 INFO:teuthology.orchestra.run.smithi131.stdout: mon_down_mkfs_grace = 2m 2024-08-02T23:39:37.705 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-08-02T23:39:37.705 INFO:teuthology.orchestra.run.smithi131.stdout: mon_warn_on_filestore_osds = false 2024-08-02T23:39:37.705 INFO:teuthology.orchestra.run.smithi131.stdout: mon min osdmap epochs = 50 2024-08-02T23:39:37.705 INFO:teuthology.orchestra.run.smithi131.stdout: mon osdmap full prune interval = 2 2024-08-02T23:39:37.705 INFO:teuthology.orchestra.run.smithi131.stdout: mon osdmap full prune min = 15 2024-08-02T23:39:37.705 INFO:teuthology.orchestra.run.smithi131.stdout: mon osdmap full prune txsize = 2 2024-08-02T23:39:37.706 INFO:teuthology.orchestra.run.smithi131.stdout: paxos service trim min = 10 2024-08-02T23:39:37.706 INFO:teuthology.orchestra.run.smithi131.stdout: mon warn on pool no app = False 2024-08-02T23:39:37.706 INFO:teuthology.orchestra.run.smithi131.stdout: osd pool default crimson = True 2024-08-02T23:39:37.706 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-08-02T23:39:37.706 INFO:teuthology.orchestra.run.smithi131.stdout:[client] 2024-08-02T23:39:37.706 INFO:teuthology.orchestra.run.smithi131.stdout: rgw cache enabled = true 2024-08-02T23:39:37.706 INFO:teuthology.orchestra.run.smithi131.stdout: rgw enable ops log = true 2024-08-02T23:39:37.706 INFO:teuthology.orchestra.run.smithi131.stdout: rgw enable usage log = true 2024-08-02T23:39:37.706 INFO:teuthology.orchestra.run.smithi131.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-08-02T23:39:37.706 INFO:teuthology.orchestra.run.smithi131.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-08-02T23:39:37.706 INFO:teuthology.orchestra.run.smithi131.stdout: debug ms = 1 2024-08-02T23:39:37.706 INFO:teuthology.orchestra.run.smithi131.stdout:[mon.a] 2024-08-02T23:39:37.706 INFO:teuthology.orchestra.run.smithi131.stdout:[osd.0] 2024-08-02T23:39:37.706 INFO:teuthology.orchestra.run.smithi131.stdout: crimson seastar cpu cores = 0-2 2024-08-02T23:39:37.706 INFO:teuthology.orchestra.run.smithi131.stdout:[osd.1] 2024-08-02T23:39:37.707 INFO:teuthology.orchestra.run.smithi131.stdout: crimson seastar cpu cores = 3-5 2024-08-02T23:39:37.707 INFO:teuthology.orchestra.run.smithi131.stdout:[osd.2] 2024-08-02T23:39:37.707 INFO:teuthology.orchestra.run.smithi131.stdout: crimson seastar cpu cores = 0-2 2024-08-02T23:39:37.707 INFO:teuthology.orchestra.run.smithi131.stdout:[osd.3] 2024-08-02T23:39:37.707 INFO:teuthology.orchestra.run.smithi131.stdout: crimson seastar cpu cores = 3-5 2024-08-02T23:39:37.708 INFO:tasks.ceph:Creating admin key on mon.a... 2024-08-02T23:39:37.708 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-08-02T23:39:37.809 INFO:tasks.ceph:Copying monmap to all nodes... 2024-08-02T23:39:37.809 DEBUG:teuthology.orchestra.run.smithi073:> set -ex 2024-08-02T23:39:37.809 DEBUG:teuthology.orchestra.run.smithi073:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-08-02T23:39:37.824 DEBUG:teuthology.orchestra.run.smithi073:> set -ex 2024-08-02T23:39:37.824 DEBUG:teuthology.orchestra.run.smithi073:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-08-02T23:39:37.880 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi073.front.sepia.ceph.com 2024-08-02T23:39:37.880 DEBUG:teuthology.orchestra.run.smithi073:> set -ex 2024-08-02T23:39:37.880 DEBUG:teuthology.orchestra.run.smithi073:> sudo dd of=/etc/ceph/ceph.keyring 2024-08-02T23:39:37.880 DEBUG:teuthology.orchestra.run.smithi073:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-08-02T23:39:37.953 DEBUG:teuthology.orchestra.run.smithi073:> set -ex 2024-08-02T23:39:37.954 DEBUG:teuthology.orchestra.run.smithi073:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-08-02T23:39:38.009 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi131.front.sepia.ceph.com 2024-08-02T23:39:38.009 DEBUG:teuthology.orchestra.run.smithi131:> set -ex 2024-08-02T23:39:38.009 DEBUG:teuthology.orchestra.run.smithi131:> sudo dd of=/etc/ceph/ceph.keyring 2024-08-02T23:39:38.009 DEBUG:teuthology.orchestra.run.smithi131:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-08-02T23:39:38.049 DEBUG:teuthology.orchestra.run.smithi131:> set -ex 2024-08-02T23:39:38.049 DEBUG:teuthology.orchestra.run.smithi131:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-08-02T23:39:38.107 INFO:tasks.ceph:Setting up mon nodes... 2024-08-02T23:39:38.107 INFO:tasks.ceph:Setting up mgr nodes... 2024-08-02T23:39:38.108 DEBUG:teuthology.orchestra.run.smithi131:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-08-02T23:39:38.214 INFO:teuthology.orchestra.run.smithi131.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-08-02T23:39:38.217 INFO:tasks.ceph:Setting up mds nodes... 2024-08-02T23:39:38.217 INFO:tasks.ceph_client:Setting up client nodes... 2024-08-02T23:39:38.217 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-08-02T23:39:38.264 INFO:teuthology.orchestra.run.smithi073.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-08-02T23:39:38.277 DEBUG:teuthology.orchestra.run.smithi131:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-08-02T23:39:38.325 INFO:teuthology.orchestra.run.smithi131.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-08-02T23:39:38.339 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-08-02T23:39:38.339 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi073.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2'}, Remote(name='ubuntu@smithi131.front.sepia.ceph.com'): {'osd.2': '/dev/vg_nvme/lv_1', 'osd.3': '/dev/vg_nvme/lv_2'}} 2024-08-02T23:39:38.339 DEBUG:teuthology.orchestra.run.smithi073:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-08-02T23:39:38.363 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2'} 2024-08-02T23:39:38.363 INFO:tasks.ceph:role: osd.0 2024-08-02T23:39:38.363 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi073.front.sepia.ceph.com 2024-08-02T23:39:38.363 DEBUG:teuthology.orchestra.run.smithi073:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-08-02T23:39:38.435 INFO:teuthology.orchestra.run.smithi073.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-08-02T23:39:38.435 INFO:teuthology.orchestra.run.smithi073.stdout: = sectsz=512 attr=2, projid32bit=1 2024-08-02T23:39:38.435 INFO:teuthology.orchestra.run.smithi073.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-08-02T23:39:38.435 INFO:teuthology.orchestra.run.smithi073.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-08-02T23:39:38.435 INFO:teuthology.orchestra.run.smithi073.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-08-02T23:39:38.435 INFO:teuthology.orchestra.run.smithi073.stdout: = sunit=0 swidth=0 blks 2024-08-02T23:39:38.435 INFO:teuthology.orchestra.run.smithi073.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-08-02T23:39:38.435 INFO:teuthology.orchestra.run.smithi073.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-08-02T23:39:38.435 INFO:teuthology.orchestra.run.smithi073.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-08-02T23:39:38.436 INFO:teuthology.orchestra.run.smithi073.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-08-02T23:39:38.957 INFO:teuthology.orchestra.run.smithi073.stdout:Discarding blocks...Done. 2024-08-02T23:39:38.959 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi073.front.sepia.ceph.com -o noatime 2024-08-02T23:39:38.959 DEBUG:teuthology.orchestra.run.smithi073:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-08-02T23:39:39.117 DEBUG:teuthology.orchestra.run.smithi073:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-08-02T23:39:39.149 DEBUG:teuthology.orchestra.run.smithi073:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-08-02T23:39:39.214 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2'} 2024-08-02T23:39:39.214 INFO:tasks.ceph:role: osd.1 2024-08-02T23:39:39.214 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi073.front.sepia.ceph.com 2024-08-02T23:39:39.214 DEBUG:teuthology.orchestra.run.smithi073:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-08-02T23:39:39.287 INFO:teuthology.orchestra.run.smithi073.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-08-02T23:39:39.287 INFO:teuthology.orchestra.run.smithi073.stdout: = sectsz=512 attr=2, projid32bit=1 2024-08-02T23:39:39.287 INFO:teuthology.orchestra.run.smithi073.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-08-02T23:39:39.287 INFO:teuthology.orchestra.run.smithi073.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-08-02T23:39:39.287 INFO:teuthology.orchestra.run.smithi073.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-08-02T23:39:39.287 INFO:teuthology.orchestra.run.smithi073.stdout: = sunit=0 swidth=0 blks 2024-08-02T23:39:39.287 INFO:teuthology.orchestra.run.smithi073.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-08-02T23:39:39.288 INFO:teuthology.orchestra.run.smithi073.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-08-02T23:39:39.288 INFO:teuthology.orchestra.run.smithi073.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-08-02T23:39:39.288 INFO:teuthology.orchestra.run.smithi073.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-08-02T23:39:39.743 INFO:teuthology.orchestra.run.smithi073.stdout:Discarding blocks...Done. 2024-08-02T23:39:39.745 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi073.front.sepia.ceph.com -o noatime 2024-08-02T23:39:39.745 DEBUG:teuthology.orchestra.run.smithi073:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-08-02T23:39:39.837 DEBUG:teuthology.orchestra.run.smithi073:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-08-02T23:39:39.865 DEBUG:teuthology.orchestra.run.smithi073:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-08-02T23:39:40.410 INFO:teuthology.orchestra.run.smithi073.stderr:WARNING: debug mode. Not for benchmarking or production 2024-08-02T23:39:40.411 INFO:teuthology.orchestra.run.smithi073.stderr:INFO 2024-08-02 23:39:40,410 seastar - Reactor backend: linux-aio 2024-08-02T23:39:40.433 INFO:teuthology.orchestra.run.smithi073.stderr:==29281==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-08-02T23:39:40.477 INFO:teuthology.orchestra.run.smithi073.stderr:INFO 2024-08-02 23:39:40,476 [shard 0:main] osd - bypassing the config fetch due to --no-mon-config 2024-08-02T23:39:40.477 INFO:teuthology.orchestra.run.smithi073.stderr:INFO 2024-08-02 23:39:40,476 [shard 0:main] osd - get_early_config: set --thread-affinity 1 --cpuset 0-2 2024-08-02T23:39:40.478 INFO:teuthology.orchestra.run.smithi073.stderr:DEBUG 2024-08-02 23:39:40,477 [shard 0:main] seastar - reactor::drain 2024-08-02T23:39:40.965 INFO:teuthology.orchestra.run.smithi073.stderr:----------------------------------------------------- 2024-08-02T23:39:40.965 INFO:teuthology.orchestra.run.smithi073.stderr:Suppressions used: 2024-08-02T23:39:40.965 INFO:teuthology.orchestra.run.smithi073.stderr: count bytes template 2024-08-02T23:39:40.965 INFO:teuthology.orchestra.run.smithi073.stderr: 1 8 MallocExtension::Register 2024-08-02T23:39:40.965 INFO:teuthology.orchestra.run.smithi073.stderr:----------------------------------------------------- 2024-08-02T23:39:40.965 INFO:teuthology.orchestra.run.smithi073.stderr: 2024-08-02T23:39:41.019 INFO:teuthology.orchestra.run.smithi073.stderr:WARNING: debug mode. Not for benchmarking or production 2024-08-02T23:39:41.021 INFO:teuthology.orchestra.run.smithi073.stderr:INFO 2024-08-02 23:39:41,020 seastar - Reactor backend: linux-aio 2024-08-02T23:39:41.048 INFO:teuthology.orchestra.run.smithi073.stderr:==29278==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-08-02T23:39:41.200 INFO:teuthology.orchestra.run.smithi073.stderr:Reactor stalled for 32 ms on shard 1. Backtrace: 0x45d5d 0x2ff736d6 0x2fd685a8 0x2fe9a5f0 0x2fe9a974 0x2fe9aabe 0x2fe9af14 0x3e6ef 0xd5553 0x2c64a 0x2857c 0xb6323 0x2f935b89 0x2f9364ee 0x2fde5b25 0x2fdee212 0x2fbcc0fb 0x2f9af75a 0x2bc34da7 0x2ba75d4f 0x2b8326f9 0x2af60d29 0x2a9cdb02 0x2b2c2edc 0x2b2c3635 0x2b2fac9d 0x2fe5e835 0x2ff88b6a 0x3012af2e 0x301305c9 0x30130b87 0x30c260cf 0x89c01 0x10ec3f 2024-08-02T23:39:41.200 INFO:teuthology.orchestra.run.smithi073.stderr:kernel callstack: 2024-08-02T23:39:41.200 INFO:teuthology.orchestra.run.smithi073.stderr:Reactor stalled for 32 ms on shard 2. Backtrace: 0x45d5d 0x2ff736d6 0x2fd685a8 0x2fe9a5f0 0x2fe9a974 0x2fe9aabe 0x2fe9af14 0x3e6ef 0x2fdec807 0x2fbcc0fb 0x2f9af75a 0x2bc34da7 0x2ba75d4f 0x2b8326f9 0x2af60d29 0x2a9cdb02 0x2b2c2edc 0x2b2c3635 0x2b2fac9d 0x2fe5e835 0x2ff88b6a 0x3012af2e 0x301305c9 0x30130b87 0x30c260cf 0x89c01 0x10ec3f 2024-08-02T23:39:41.200 INFO:teuthology.orchestra.run.smithi073.stderr:kernel callstack: 2024-08-02T23:39:41.499 INFO:teuthology.orchestra.run.smithi073.stderr:Reactor stalled for 32 ms on shard 1. Backtrace: 0x45d5d 0x2ff736d6 0x2fd685a8 0x2fe9a5f0 0x2fe9a974 0x2fe9aabe 0x2fe9af14 0x3e6ef 0x2fcc4304 0x2fdb2abe 0x2fdb339f 0x2fde6888 0x2fdee212 0x2fbcc0fb 0x2f9af75a 0x2bc30664 0x2ba75d4f 0x2b8326f9 0x2af60d29 0x2a9c65a0 0x2b2c1320 0x2b2c1a79 0x2b2fac9d 0x2fe5e835 0x2ff88b6a 0x3012af2e 0x301305c9 0x30130b87 0x30c260cf 0x89c01 0x10ec3f 2024-08-02T23:39:41.499 INFO:teuthology.orchestra.run.smithi073.stderr:kernel callstack: 2024-08-02T23:39:41.499 INFO:teuthology.orchestra.run.smithi073.stderr:Reactor stalled for 32 ms on shard 2. Backtrace: 0x45d5d 0x2ff736d6 0x2fd685a8 0x2fe9a5f0 0x2fe9a974 0x2fe9aabe 0x2fe9af14 0x3e6ef 0xd596a 0xd5ac0 0xd554b 0x287b7 0xb738c 0x24c1a011 0x24c1a07b 0x2bc34f0e 0x2ba75d4f 0x2b8326f9 0x2af60d29 0x2a9c65a0 0x2b2c1320 0x2b2c1a79 0x2b2fac9d 0x2fe5e835 0x2ff88b6a 0x3012af2e 0x301305c9 0x30130b87 0x30c260cf 0x89c01 0x10ec3f 2024-08-02T23:39:41.500 INFO:teuthology.orchestra.run.smithi073.stderr:kernel callstack: 2024-08-02T23:39:42.219 INFO:teuthology.orchestra.run.smithi073.stderr:----------------------------------------------------- 2024-08-02T23:39:42.219 INFO:teuthology.orchestra.run.smithi073.stderr:Suppressions used: 2024-08-02T23:39:42.219 INFO:teuthology.orchestra.run.smithi073.stderr: count bytes template 2024-08-02T23:39:42.219 INFO:teuthology.orchestra.run.smithi073.stderr: 1 8 MallocExtension::Register 2024-08-02T23:39:42.219 INFO:teuthology.orchestra.run.smithi073.stderr:----------------------------------------------------- 2024-08-02T23:39:42.219 INFO:teuthology.orchestra.run.smithi073.stderr: 2024-08-02T23:39:42.220 INFO:teuthology.orchestra.run.smithi073.stdout:created object store /var/lib/ceph/osd/ceph-0 for osd.0 fsid 7caeb70d-fd31-4bdf-94e1-d0f19ea8e3c6 2024-08-02T23:39:42.263 DEBUG:teuthology.orchestra.run.smithi073:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-08-02T23:39:42.288 DEBUG:teuthology.orchestra.run.smithi073:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-08-02T23:39:42.830 INFO:teuthology.orchestra.run.smithi073.stderr:WARNING: debug mode. Not for benchmarking or production 2024-08-02T23:39:42.830 INFO:teuthology.orchestra.run.smithi073.stderr:INFO 2024-08-02 23:39:42,830 seastar - Reactor backend: linux-aio 2024-08-02T23:39:42.852 INFO:teuthology.orchestra.run.smithi073.stderr:==29328==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-08-02T23:39:42.896 INFO:teuthology.orchestra.run.smithi073.stderr:INFO 2024-08-02 23:39:42,895 [shard 0:main] osd - bypassing the config fetch due to --no-mon-config 2024-08-02T23:39:42.896 INFO:teuthology.orchestra.run.smithi073.stderr:INFO 2024-08-02 23:39:42,895 [shard 0:main] osd - get_early_config: set --thread-affinity 1 --cpuset 3-5 2024-08-02T23:39:42.896 INFO:teuthology.orchestra.run.smithi073.stderr:DEBUG 2024-08-02 23:39:42,896 [shard 0:main] seastar - reactor::drain 2024-08-02T23:39:43.371 INFO:teuthology.orchestra.run.smithi073.stderr:----------------------------------------------------- 2024-08-02T23:39:43.371 INFO:teuthology.orchestra.run.smithi073.stderr:Suppressions used: 2024-08-02T23:39:43.372 INFO:teuthology.orchestra.run.smithi073.stderr: count bytes template 2024-08-02T23:39:43.372 INFO:teuthology.orchestra.run.smithi073.stderr: 1 8 MallocExtension::Register 2024-08-02T23:39:43.372 INFO:teuthology.orchestra.run.smithi073.stderr:----------------------------------------------------- 2024-08-02T23:39:43.372 INFO:teuthology.orchestra.run.smithi073.stderr: 2024-08-02T23:39:43.428 INFO:teuthology.orchestra.run.smithi073.stderr:WARNING: debug mode. Not for benchmarking or production 2024-08-02T23:39:43.430 INFO:teuthology.orchestra.run.smithi073.stderr:INFO 2024-08-02 23:39:43,429 seastar - Reactor backend: linux-aio 2024-08-02T23:39:43.466 INFO:teuthology.orchestra.run.smithi073.stderr:==29326==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-08-02T23:39:43.612 INFO:teuthology.orchestra.run.smithi073.stderr:Reactor stalled for 32 ms on shard 2. Backtrace: 0x45d5d 0x2ff736d6 0x2fd685a8 0x2fe9a5f0 0x2fe9a974 0x2fe9aabe 0x2fe9af14 0x3e6ef 0x2fbc5420 0x2fc3501c 0x2fde6aca 0x2fdee212 0x2fbcc0fb 0x2f9af75a 0x2bc34da7 0x2ba75d4f 0x2b8326f9 0x2af60d29 0x2a9cdb02 0x2b2c2edc 0x2b2c3635 0x2b2fac9d 0x2fe5e835 0x2ff88b6a 0x3012af2e 0x301305c9 0x30130b87 0x30c260cf 0x89c01 0x10ec3f 2024-08-02T23:39:43.612 INFO:teuthology.orchestra.run.smithi073.stderr:kernel callstack: 2024-08-02T23:39:43.612 INFO:teuthology.orchestra.run.smithi073.stderr:Reactor stalled for 32 ms on shard 1. Backtrace: 0x45d5d 0x2ff736d6 0x2fd685a8 0x2fe9a5f0 0x2fe9a974 0x2fe9aabe 0x2fe9af14 0x3e6ef 0xccf04 0xd5bed 0xd554b 0x2c64a 0x282da 0xb49e1 0x2051527f 0x2fdb29f5 0x2fdb339f 0x2fde6888 0x2fdee212 0x2fbcc0fb 0x2f9af75a 0x2bc34da7 0x2ba75d4f 0x2b8326f9 0x2af60d29 0x2a9cdb02 0x2b2c2edc 0x2b2c3635 0x2b2fac9d 0x2fe5e835 0x2ff88b6a 0x3012af2e 0x301305c9 0x30130b87 0x30c260cf 0x89c01 0x10ec3f 2024-08-02T23:39:43.612 INFO:teuthology.orchestra.run.smithi073.stderr:kernel callstack: 2024-08-02T23:39:43.929 INFO:teuthology.orchestra.run.smithi073.stderr:Reactor stalled for 32 ms on shard 2. Backtrace: 0x45d5d 0x2ff736d6 0x2fd685a8 0x2fe9a5f0 0x2fe9a974 0x2fe9aabe 0x2fe9af14 0x3e6ef 0xd5932 0xd5ac0 0xd554b 0x287b7 0xb738c 0x24c1a011 0x24c1a07b 0x2a90df11 0x2bc38158 0x2ba75d4f 0x2b8326f9 0x2af60d29 0x2a9c65a0 0x2b2c1320 0x2b2c1a79 0x2b2fac9d 0x2fe5e835 0x2ff88b6a 0x3012af2e 0x301305c9 0x30130b87 0x30c260cf 0x89c01 0x10ec3f 2024-08-02T23:39:43.930 INFO:teuthology.orchestra.run.smithi073.stderr:Reactor stalled for 32 ms on shard 1. Backtrace: 0x45d5d 0x2ff736d6 0x2fd685a8 0x2fe9a5f0 0x2fe9a974 0x2fe9aabe 0x2fe9af14 0x3e6ef 0x162c3f 0x2c6e3 0x282da 0xb49e1 0x2051527f 0x2fdec3d8 0x2fbcc0fb 0x2f9af75a 0x2bc3803d 0x2ba75d4f 0x2b8326f9 0x2af60d29 0x2a9c65a0 0x2b2c1320 0x2b2c1a79 0x2b2fac9d 0x2fe5e835 0x2ff88b6a 0x3012af2e 0x301305c9 0x30130b87 0x30c260cf 0x89c01 0x10ec3f 2024-08-02T23:39:43.930 INFO:teuthology.orchestra.run.smithi073.stderr:kernel callstack: 2024-08-02T23:39:43.930 INFO:teuthology.orchestra.run.smithi073.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff9ca8bba6 0xffffffff9d6c8d62 0xffffffff9d800ba2 2024-08-02T23:39:44.019 INFO:teuthology.orchestra.run.smithi073.stderr:Reactor stalled for 65 ms on shard 2. Backtrace: 0x45d5d 0x2ff736d6 0x2fd685a8 0x2fe9a5f0 0x2fe9a974 0x2fe9aabe 0x2fe9af14 0x3e6ef 0x2bf094c8 0x2c04910d 0x2bef1df3 0x2b71f071 0x2b721886 0x2b722241 0x2b86d257 0x2b89139f 0x2b1307e8 0x2fe5e835 0x2ff88b6a 0x3012af2e 0x301305c9 0x30130b87 0x30c260cf 0x89c01 0x10ec3f 2024-08-02T23:39:44.020 INFO:teuthology.orchestra.run.smithi073.stderr:kernel callstack: 2024-08-02T23:39:44.020 INFO:teuthology.orchestra.run.smithi073.stderr:Reactor stalled for 65 ms on shard 1. Backtrace: 0x45d5d 0x2ff736d6 0x2fd685a8 0x2fe9a5f0 0x2fe9a974 0x2fe9aabe 0x2fe9af14 0x3e6ef 0x2bf09417 0x2c04910d 0x2bef1df3 0x2b71f071 0x2b721886 0x2b722241 0x2b86d257 0x2b89139f 0x2b1307e8 0x2fe5e835 0x2ff88b6a 0x3012af2e 0x301305c9 0x30130b87 0x30c260cf 0x89c01 0x10ec3f 2024-08-02T23:39:44.020 INFO:teuthology.orchestra.run.smithi073.stderr:kernel callstack: 2024-08-02T23:39:44.677 INFO:teuthology.orchestra.run.smithi073.stderr:----------------------------------------------------- 2024-08-02T23:39:44.677 INFO:teuthology.orchestra.run.smithi073.stderr:Suppressions used: 2024-08-02T23:39:44.677 INFO:teuthology.orchestra.run.smithi073.stderr: count bytes template 2024-08-02T23:39:44.677 INFO:teuthology.orchestra.run.smithi073.stderr: 1 8 MallocExtension::Register 2024-08-02T23:39:44.677 INFO:teuthology.orchestra.run.smithi073.stderr:----------------------------------------------------- 2024-08-02T23:39:44.677 INFO:teuthology.orchestra.run.smithi073.stderr: 2024-08-02T23:39:44.678 INFO:teuthology.orchestra.run.smithi073.stdout:created object store /var/lib/ceph/osd/ceph-1 for osd.1 fsid 7caeb70d-fd31-4bdf-94e1-d0f19ea8e3c6 2024-08-02T23:39:44.722 DEBUG:teuthology.orchestra.run.smithi073:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-08-02T23:39:44.747 DEBUG:teuthology.orchestra.run.smithi131:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-08-02T23:39:44.777 INFO:tasks.ceph:roles_to_devs: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.3': '/dev/vg_nvme/lv_2'} 2024-08-02T23:39:44.777 INFO:tasks.ceph:role: osd.2 2024-08-02T23:39:44.778 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi131.front.sepia.ceph.com 2024-08-02T23:39:44.778 DEBUG:teuthology.orchestra.run.smithi131:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-08-02T23:39:44.853 INFO:teuthology.orchestra.run.smithi131.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-08-02T23:39:44.853 INFO:teuthology.orchestra.run.smithi131.stdout: = sectsz=512 attr=2, projid32bit=1 2024-08-02T23:39:44.853 INFO:teuthology.orchestra.run.smithi131.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-08-02T23:39:44.854 INFO:teuthology.orchestra.run.smithi131.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-08-02T23:39:44.854 INFO:teuthology.orchestra.run.smithi131.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-08-02T23:39:44.854 INFO:teuthology.orchestra.run.smithi131.stdout: = sunit=0 swidth=0 blks 2024-08-02T23:39:44.854 INFO:teuthology.orchestra.run.smithi131.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-08-02T23:39:44.854 INFO:teuthology.orchestra.run.smithi131.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-08-02T23:39:44.854 INFO:teuthology.orchestra.run.smithi131.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-08-02T23:39:44.854 INFO:teuthology.orchestra.run.smithi131.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-08-02T23:39:45.370 INFO:teuthology.orchestra.run.smithi131.stdout:Discarding blocks...Done. 2024-08-02T23:39:45.373 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi131.front.sepia.ceph.com -o noatime 2024-08-02T23:39:45.373 DEBUG:teuthology.orchestra.run.smithi131:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-2 2024-08-02T23:39:45.518 DEBUG:teuthology.orchestra.run.smithi131:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-08-02T23:39:45.553 DEBUG:teuthology.orchestra.run.smithi131:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-08-02T23:39:45.620 INFO:tasks.ceph:roles_to_devs: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.3': '/dev/vg_nvme/lv_2'} 2024-08-02T23:39:45.620 INFO:tasks.ceph:role: osd.3 2024-08-02T23:39:45.620 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi131.front.sepia.ceph.com 2024-08-02T23:39:45.620 DEBUG:teuthology.orchestra.run.smithi131:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-08-02T23:39:45.695 INFO:teuthology.orchestra.run.smithi131.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-08-02T23:39:45.695 INFO:teuthology.orchestra.run.smithi131.stdout: = sectsz=512 attr=2, projid32bit=1 2024-08-02T23:39:45.695 INFO:teuthology.orchestra.run.smithi131.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-08-02T23:39:45.695 INFO:teuthology.orchestra.run.smithi131.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-08-02T23:39:45.695 INFO:teuthology.orchestra.run.smithi131.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-08-02T23:39:45.695 INFO:teuthology.orchestra.run.smithi131.stdout: = sunit=0 swidth=0 blks 2024-08-02T23:39:45.695 INFO:teuthology.orchestra.run.smithi131.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-08-02T23:39:45.695 INFO:teuthology.orchestra.run.smithi131.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-08-02T23:39:45.695 INFO:teuthology.orchestra.run.smithi131.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-08-02T23:39:45.695 INFO:teuthology.orchestra.run.smithi131.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-08-02T23:39:46.138 INFO:teuthology.orchestra.run.smithi131.stdout:Discarding blocks...Done. 2024-08-02T23:39:46.140 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi131.front.sepia.ceph.com -o noatime 2024-08-02T23:39:46.140 DEBUG:teuthology.orchestra.run.smithi131:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-3 2024-08-02T23:39:46.232 DEBUG:teuthology.orchestra.run.smithi131:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-08-02T23:39:46.262 DEBUG:teuthology.orchestra.run.smithi131:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-08-02T23:39:46.847 INFO:teuthology.orchestra.run.smithi131.stderr:WARNING: debug mode. Not for benchmarking or production 2024-08-02T23:39:46.848 INFO:teuthology.orchestra.run.smithi131.stderr:INFO 2024-08-02 23:39:46,847 seastar - Reactor backend: linux-aio 2024-08-02T23:39:46.871 INFO:teuthology.orchestra.run.smithi131.stderr:==29152==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-08-02T23:39:46.917 INFO:teuthology.orchestra.run.smithi131.stderr:INFO 2024-08-02 23:39:46,916 [shard 0:main] osd - bypassing the config fetch due to --no-mon-config 2024-08-02T23:39:46.917 INFO:teuthology.orchestra.run.smithi131.stderr:INFO 2024-08-02 23:39:46,916 [shard 0:main] osd - get_early_config: set --thread-affinity 1 --cpuset 0-2 2024-08-02T23:39:46.918 INFO:teuthology.orchestra.run.smithi131.stderr:DEBUG 2024-08-02 23:39:46,917 [shard 0:main] seastar - reactor::drain 2024-08-02T23:39:47.403 INFO:teuthology.orchestra.run.smithi131.stderr:----------------------------------------------------- 2024-08-02T23:39:47.403 INFO:teuthology.orchestra.run.smithi131.stderr:Suppressions used: 2024-08-02T23:39:47.403 INFO:teuthology.orchestra.run.smithi131.stderr: count bytes template 2024-08-02T23:39:47.404 INFO:teuthology.orchestra.run.smithi131.stderr: 1 8 MallocExtension::Register 2024-08-02T23:39:47.404 INFO:teuthology.orchestra.run.smithi131.stderr:----------------------------------------------------- 2024-08-02T23:39:47.404 INFO:teuthology.orchestra.run.smithi131.stderr: 2024-08-02T23:39:47.462 INFO:teuthology.orchestra.run.smithi131.stderr:WARNING: debug mode. Not for benchmarking or production 2024-08-02T23:39:47.463 INFO:teuthology.orchestra.run.smithi131.stderr:INFO 2024-08-02 23:39:47,463 seastar - Reactor backend: linux-aio 2024-08-02T23:39:47.491 INFO:teuthology.orchestra.run.smithi131.stderr:==29148==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-08-02T23:39:47.643 INFO:teuthology.orchestra.run.smithi131.stderr:Reactor stalled for 32 ms on shard 1. Backtrace: 0x45d5d 0x2ff736d6 0x2fd685a8 0x2fe9a5f0 0x2fe9a974 0x2fe9aabe 0x2fe9af14 0x3e6ef 0x2c29c 0x2c5a7 0x2857c 0xb6323 0x2f937dc7 0x2fdec849 0x2fbcc0fb 0x2f9af75a 0x2bc34da7 0x2ba75d4f 0x2b8326f9 0x2af60d29 0x2a9cdb02 0x2b2c2edc 0x2b2c3635 0x2b2fac9d 0x2fe5e835 0x2ff88b6a 0x3012af2e 0x301305c9 0x30130b87 0x30c260cf 0x89c01 0x10ec3f 2024-08-02T23:39:47.643 INFO:teuthology.orchestra.run.smithi131.stderr:Reactor stalled for 32 ms on shard 2. Backtrace: 0x45d5d 0x2ff736d6 0x2fd685a8 0x2fe9a5f0 0x2fe9a974 0x2fe9aabe 0x2fe9af14 0x3e6ef 0x2fbe78f1 0x2fbe9a65 0x2fde71fd 0x2fdee212 0x2fbcc0fb 0x2f9af75a 0x2bc34da7 0x2ba75d4f 0x2b8326f9 0x2af60d29 0x2a9cdb02 0x2b2c2edc 0x2b2c3635 0x2b2fac9d 0x2fe5e835 0x2ff88b6a 0x3012af2e 0x301305c9 0x30130b87 0x30c260cf 0x89c01 0x10ec3f 2024-08-02T23:39:47.643 INFO:teuthology.orchestra.run.smithi131.stderr:kernel callstack: 2024-08-02T23:39:47.643 INFO:teuthology.orchestra.run.smithi131.stderr:kernel callstack: 2024-08-02T23:39:47.942 INFO:teuthology.orchestra.run.smithi131.stderr:Reactor stalled for 32 ms on shard 1. Backtrace: 0x45d5d 0x2ff736d6 0x2fd685a8 0x2fe9a5f0 0x2fe9a974 0x2fe9aabe 0x2fe9af14 0x3e6ef 0x2fbe7548 0x2fbe9a65 0x2fde71fd 0x2fdee212 0x2fbcc0fb 0x2f9af75a 0x2bc3803d 0x2ba75d4f 0x2b8326f9 0x2af60d29 0x2a9c65a0 0x2b2c1320 0x2b2c1a79 0x2b2fac9d 0x2fe5e835 0x2ff88b6a 0x3012af2e 0x301305c9 0x30130b87 0x30c260cf 0x89c01 0x10ec3f 2024-08-02T23:39:47.942 INFO:teuthology.orchestra.run.smithi131.stderr:kernel callstack: 2024-08-02T23:39:47.942 INFO:teuthology.orchestra.run.smithi131.stderr:Reactor stalled for 32 ms on shard 2. Backtrace: 0x45d5d 0x2ff736d6 0x2fd685a8 0x2fe9a5f0 0x2fe9a974 0x2fe9aabe 0x2fe9af14 0x3e6ef 0x39753 0x94437 0x94d46 0x2fce4b11 0x2b5b16c9 0x2bc37f79 0x2ba75d4f 0x2b8326f9 0x2af60d29 0x2a9c65a0 0x2b2c1320 0x2b2c1a79 0x2b2fac9d 0x2fe5e835 0x2ff88b6a 0x3012af2e 0x301305c9 0x30130b87 0x30c260cf 0x89c01 0x10ec3f 2024-08-02T23:39:47.942 INFO:teuthology.orchestra.run.smithi131.stderr:kernel callstack: 2024-08-02T23:39:48.681 INFO:teuthology.orchestra.run.smithi131.stderr:----------------------------------------------------- 2024-08-02T23:39:48.682 INFO:teuthology.orchestra.run.smithi131.stderr:Suppressions used: 2024-08-02T23:39:48.682 INFO:teuthology.orchestra.run.smithi131.stderr: count bytes template 2024-08-02T23:39:48.682 INFO:teuthology.orchestra.run.smithi131.stderr: 1 8 MallocExtension::Register 2024-08-02T23:39:48.682 INFO:teuthology.orchestra.run.smithi131.stderr:----------------------------------------------------- 2024-08-02T23:39:48.682 INFO:teuthology.orchestra.run.smithi131.stderr: 2024-08-02T23:39:48.683 INFO:teuthology.orchestra.run.smithi131.stdout:created object store /var/lib/ceph/osd/ceph-2 for osd.2 fsid 7caeb70d-fd31-4bdf-94e1-d0f19ea8e3c6 2024-08-02T23:39:48.726 DEBUG:teuthology.orchestra.run.smithi131:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-08-02T23:39:48.752 DEBUG:teuthology.orchestra.run.smithi131:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-08-02T23:39:49.295 INFO:teuthology.orchestra.run.smithi131.stderr:WARNING: debug mode. Not for benchmarking or production 2024-08-02T23:39:49.296 INFO:teuthology.orchestra.run.smithi131.stderr:INFO 2024-08-02 23:39:49,295 seastar - Reactor backend: linux-aio 2024-08-02T23:39:49.322 INFO:teuthology.orchestra.run.smithi131.stderr:==29201==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-08-02T23:39:49.367 INFO:teuthology.orchestra.run.smithi131.stderr:INFO 2024-08-02 23:39:49,366 [shard 0:main] osd - bypassing the config fetch due to --no-mon-config 2024-08-02T23:39:49.367 INFO:teuthology.orchestra.run.smithi131.stderr:INFO 2024-08-02 23:39:49,366 [shard 0:main] osd - get_early_config: set --thread-affinity 1 --cpuset 3-5 2024-08-02T23:39:49.367 INFO:teuthology.orchestra.run.smithi131.stderr:DEBUG 2024-08-02 23:39:49,367 [shard 0:main] seastar - reactor::drain 2024-08-02T23:39:49.849 INFO:teuthology.orchestra.run.smithi131.stderr:----------------------------------------------------- 2024-08-02T23:39:49.849 INFO:teuthology.orchestra.run.smithi131.stderr:Suppressions used: 2024-08-02T23:39:49.849 INFO:teuthology.orchestra.run.smithi131.stderr: count bytes template 2024-08-02T23:39:49.849 INFO:teuthology.orchestra.run.smithi131.stderr: 1 8 MallocExtension::Register 2024-08-02T23:39:49.849 INFO:teuthology.orchestra.run.smithi131.stderr:----------------------------------------------------- 2024-08-02T23:39:49.849 INFO:teuthology.orchestra.run.smithi131.stderr: 2024-08-02T23:39:49.898 INFO:teuthology.orchestra.run.smithi131.stderr:WARNING: debug mode. Not for benchmarking or production 2024-08-02T23:39:49.900 INFO:teuthology.orchestra.run.smithi131.stderr:INFO 2024-08-02 23:39:49,899 seastar - Reactor backend: linux-aio 2024-08-02T23:39:49.936 INFO:teuthology.orchestra.run.smithi131.stderr:==29199==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-08-02T23:39:50.092 INFO:teuthology.orchestra.run.smithi131.stderr:Reactor stalled for 32 ms on shard 2. Backtrace: 0x45d5d 0x2ff736d6 0x2fd685a8 0x2fe9a5f0 0x2fe9a974 0x2fe9aabe 0x2fe9af14 0x3e6ef 0x2fbc555b 0x2fbc5d1f 0x2fde6a02 0x2fdee212 0x2fbcc0fb 0x2f9af75a 0x2bc34da7 0x2ba75d4f 0x2b8326f9 0x2af60d29 0x2a9cdb02 0x2b2c2edc 0x2b2c3635 0x2b2fac9d 0x2fe5e835 0x2ff88b6a 0x3012af2e 0x301305c9 0x30130b87 0x30c260cf 0x89c01 0x10ec3f 2024-08-02T23:39:50.092 INFO:teuthology.orchestra.run.smithi131.stderr:Reactor stalled for 32 ms on shard 1. Backtrace: 0x45d5d 0x2ff736d6 0x2fd685a8 0x2fe9a5f0 0x2fe9a974 0x2fe9aabe 0x2fe9af14 0x3e6ef 0x2fbe78d9 0x2fbe9a65 0x2fbe9d0b 0x2fde6e7d 0x2fdee212 0x2fbcc0fb 0x2f9af75a 0x2bc34da7 0x2ba75d4f 0x2b8326f9 0x2af60d29 0x2a9cdb02 0x2b2c2edc 0x2b2c3635 0x2b2fac9d 0x2fe5e835 0x2ff88b6a 0x3012af2e 0x301305c9 0x30130b87 0x30c260cf 0x89c01 0x10ec3f 2024-08-02T23:39:50.092 INFO:teuthology.orchestra.run.smithi131.stderr:kernel callstack: 2024-08-02T23:39:50.092 INFO:teuthology.orchestra.run.smithi131.stderr:kernel callstack: 2024-08-02T23:39:50.408 INFO:teuthology.orchestra.run.smithi131.stderr:Reactor stalled for 32 ms on shard 2. Backtrace: 0x45d5d 0x2ff736d6 0x2fd685a8 0x2fe9a5f0 0x2fe9a974 0x2fe9aabe 0x2fe9af14 0x3e6ef 0xd5a80 0xd554b 0x287b7 0xb4674 0x24c19f11 0x2fdef56d 0x2fbcc0fb 0x2f9af75a 0x2bc3803d 0x2ba75d4f 0x2b8326f9 0x2af60d29 0x2a9c65a0 0x2b2c1320 0x2b2c1a79 0x2b2fac9d 0x2fe5e835 0x2ff88b6a 0x3012af2e 0x301305c9 0x30130b87 0x30c260cf 0x89c01 0x10ec3f 2024-08-02T23:39:50.409 INFO:teuthology.orchestra.run.smithi131.stderr:kernel callstack: 2024-08-02T23:39:50.409 INFO:teuthology.orchestra.run.smithi131.stderr:Reactor stalled for 32 ms on shard 1. Backtrace: 0x45d5d 0x2ff736d6 0x2fd685a8 0x2fe9a5f0 0x2fe9a974 0x2fe9aabe 0x2fe9af14 0x3e6ef 0xd5934 0xd5ac0 0xd554b 0x287b7 0xb738c 0x24c1a011 0x2a90df11 0x2bc38158 0x2ba75d4f 0x2b8326f9 0x2af60d29 0x2a9c65a0 0x2b2c1320 0x2b2c1a79 0x2b2fac9d 0x2fe5e835 0x2ff88b6a 0x3012af2e 0x301305c9 0x30130b87 0x30c260cf 0x89c01 0x10ec3f 2024-08-02T23:39:50.409 INFO:teuthology.orchestra.run.smithi131.stderr:kernel callstack: 2024-08-02T23:39:50.499 INFO:teuthology.orchestra.run.smithi131.stderr:Reactor stalled for 65 ms on shard 1. Backtrace: 0x45d5d 0x2ff736d6 0x2fd685a8 0x2fe9a5f0 0x2fe9a974 0x2fe9aabe 0x2fe9af14 0x3e6ef 0x2bf093af 0x2c04910d 0x2bef1df3 0x2b71f071 0x2b721886 0x2b722241 0x2b86d257 0x2b89139f 0x2b1307e8 0x2fe5e835 0x2ff88b6a 0x3012af2e 0x301305c9 0x30130b87 0x30c260cf 0x89c01 0x10ec3f 2024-08-02T23:39:50.499 INFO:teuthology.orchestra.run.smithi131.stderr:kernel callstack: 2024-08-02T23:39:50.499 INFO:teuthology.orchestra.run.smithi131.stderr:Reactor stalled for 65 ms on shard 2. Backtrace: 0x45d5d 0x2ff736d6 0x2fd685a8 0x2fe9a5f0 0x2fe9a974 0x2fe9aabe 0x2fe9af14 0x3e6ef 0x2bf09476 0x2c04910d 0x2bef1df3 0x2b71f071 0x2b721886 0x2b722241 0x2b86d257 0x2b89139f 0x2b1307e8 0x2fe5e835 0x2ff88b6a 0x3012af2e 0x301305c9 0x30130b87 0x30c260cf 0x89c01 0x10ec3f 2024-08-02T23:39:50.499 INFO:teuthology.orchestra.run.smithi131.stderr:kernel callstack: 2024-08-02T23:39:51.162 INFO:teuthology.orchestra.run.smithi131.stderr:----------------------------------------------------- 2024-08-02T23:39:51.162 INFO:teuthology.orchestra.run.smithi131.stderr:Suppressions used: 2024-08-02T23:39:51.162 INFO:teuthology.orchestra.run.smithi131.stderr: count bytes template 2024-08-02T23:39:51.163 INFO:teuthology.orchestra.run.smithi131.stderr: 1 8 MallocExtension::Register 2024-08-02T23:39:51.163 INFO:teuthology.orchestra.run.smithi131.stderr:----------------------------------------------------- 2024-08-02T23:39:51.163 INFO:teuthology.orchestra.run.smithi131.stderr: 2024-08-02T23:39:51.163 INFO:teuthology.orchestra.run.smithi131.stdout:created object store /var/lib/ceph/osd/ceph-3 for osd.3 fsid 7caeb70d-fd31-4bdf-94e1-d0f19ea8e3c6 2024-08-02T23:39:51.207 DEBUG:teuthology.orchestra.run.smithi131:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-08-02T23:39:51.231 INFO:tasks.ceph:Reading keys from all nodes... 2024-08-02T23:39:51.231 DEBUG:teuthology.orchestra.run.smithi073:> set -ex 2024-08-02T23:39:51.232 DEBUG:teuthology.orchestra.run.smithi073:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-08-02T23:39:51.263 DEBUG:teuthology.orchestra.run.smithi073:> set -ex 2024-08-02T23:39:51.263 DEBUG:teuthology.orchestra.run.smithi073:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-08-02T23:39:51.333 DEBUG:teuthology.orchestra.run.smithi131:> set -ex 2024-08-02T23:39:51.334 DEBUG:teuthology.orchestra.run.smithi131:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-08-02T23:39:51.358 DEBUG:teuthology.orchestra.run.smithi131:> set -ex 2024-08-02T23:39:51.358 DEBUG:teuthology.orchestra.run.smithi131:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-08-02T23:39:51.423 DEBUG:teuthology.orchestra.run.smithi131:> set -ex 2024-08-02T23:39:51.423 DEBUG:teuthology.orchestra.run.smithi131:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-08-02T23:39:51.488 DEBUG:teuthology.orchestra.run.smithi073:> set -ex 2024-08-02T23:39:51.488 DEBUG:teuthology.orchestra.run.smithi073:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-08-02T23:39:51.506 DEBUG:teuthology.orchestra.run.smithi131:> set -ex 2024-08-02T23:39:51.506 DEBUG:teuthology.orchestra.run.smithi131:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-08-02T23:39:51.543 INFO:tasks.ceph:Adding keys to all mons... 2024-08-02T23:39:51.543 DEBUG:teuthology.orchestra.run.smithi073:> sudo tee -a /etc/ceph/ceph.keyring 2024-08-02T23:39:51.575 INFO:teuthology.orchestra.run.smithi073.stdout:[osd.0] 2024-08-02T23:39:51.575 INFO:teuthology.orchestra.run.smithi073.stdout: key = AQA9bq1mJm5FBhAAp+TGlXxVoRpb159w2r/L1w== 2024-08-02T23:39:51.575 INFO:teuthology.orchestra.run.smithi073.stdout:[osd.1] 2024-08-02T23:39:51.575 INFO:teuthology.orchestra.run.smithi073.stdout: key = AQA/bq1m9t4eHxAA28HowZSv9Hv7q7qR7AdwDg== 2024-08-02T23:39:51.575 INFO:teuthology.orchestra.run.smithi073.stdout:[mgr.x] 2024-08-02T23:39:51.575 INFO:teuthology.orchestra.run.smithi073.stdout: key = AQA6bq1mla25DBAAECcb1A6dBRG+pm5lg4xwCw== 2024-08-02T23:39:51.575 INFO:teuthology.orchestra.run.smithi073.stdout:[osd.2] 2024-08-02T23:39:51.576 INFO:teuthology.orchestra.run.smithi073.stdout: key = AQBDbq1mhdutIBAAeQ3kcWav8lCUcH3txpkNWQ== 2024-08-02T23:39:51.576 INFO:teuthology.orchestra.run.smithi073.stdout:[osd.3] 2024-08-02T23:39:51.576 INFO:teuthology.orchestra.run.smithi073.stdout: key = AQBFbq1mReMxOxAAbl1GxYLbTz3XcujTs6DxeQ== 2024-08-02T23:39:51.576 INFO:teuthology.orchestra.run.smithi073.stdout:[client.0] 2024-08-02T23:39:51.576 INFO:teuthology.orchestra.run.smithi073.stdout: key = AQA6bq1mFra2DxAATV/i1BQQ5GU4DZ1rZ2K05w== 2024-08-02T23:39:51.576 INFO:teuthology.orchestra.run.smithi073.stdout:[client.1] 2024-08-02T23:39:51.576 INFO:teuthology.orchestra.run.smithi073.stdout: key = AQA6bq1m8UZiExAAv6rmWySNSilQU5YcS9gQ2A== 2024-08-02T23:39:51.577 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-08-02T23:39:51.682 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-08-02T23:39:51.740 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-08-02T23:39:51.797 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-08-02T23:39:51.853 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-08-02T23:39:51.911 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-08-02T23:39:51.970 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-08-02T23:39:52.029 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-08-02T23:39:52.029 DEBUG:teuthology.orchestra.run.smithi073:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-08-02T23:39:52.054 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-08-02T23:39:52.153 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:39:52.151+0000 7effd1a099c0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-02T23:39:52.153 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:39:52.151+0000 7effd1a099c0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-02T23:39:52.165 DEBUG:teuthology.orchestra.run.smithi073:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-08-02T23:39:52.190 DEBUG:teuthology.orchestra.run.smithi073:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-08-02T23:39:52.245 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-08-02T23:39:52.246 INFO:tasks.ceph.mon.a:Restarting daemon 2024-08-02T23:39:52.246 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-08-02T23:39:52.288 INFO:tasks.ceph.mon.a:Started 2024-08-02T23:39:52.288 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-08-02T23:39:52.288 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-08-02T23:39:52.288 DEBUG:teuthology.orchestra.run.smithi131:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-08-02T23:39:52.291 INFO:tasks.ceph.mgr.x:Started 2024-08-02T23:39:52.291 INFO:tasks.ceph:Setting crush tunables to default 2024-08-02T23:39:52.291 DEBUG:teuthology.orchestra.run.smithi073:> sudo ceph --cluster ceph osd crush tunables default 2024-08-02T23:39:52.366 INFO:tasks.ceph.mon.a.smithi073.stderr:2024-08-02T23:39:52.364+0000 7fd3754929c0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-02T23:39:52.366 INFO:tasks.ceph.mon.a.smithi073.stderr:2024-08-02T23:39:52.364+0000 7fd3754929c0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-02T23:39:52.366 INFO:tasks.ceph.mon.a.smithi073.stderr:2024-08-02T23:39:52.364+0000 7fd3754929c0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-02T23:39:52.380 INFO:tasks.ceph.mgr.x.smithi131.stderr:2024-08-02T23:39:52.377+0000 7ff194b6df00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-02T23:39:52.397 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:39:52.390+0000 7fd303959640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-02T23:39:52.397 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:39:52.396+0000 7fd303959640 1 Processor -- start 2024-08-02T23:39:52.398 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:39:52.396+0000 7fd303959640 1 -- start start 2024-08-02T23:39:52.399 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:39:52.397+0000 7fd303959640 1 --2- >> v2:172.21.15.73:3300/0 conn(0x7fd2fc0a2e10 0x7fd2fc1398c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-02T23:39:52.399 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:39:52.397+0000 7fd303959640 1 -- --> v1:172.21.15.73:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd2fc09c040 con 0x7fd2fc0a22f0 2024-08-02T23:39:52.399 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:39:52.397+0000 7fd303959640 1 -- --> v2:172.21.15.73:3300/0 -- mon_getmap magic: 0 -- 0x7fd2fc0a0ca0 con 0x7fd2fc0a2e10 2024-08-02T23:39:52.400 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:39:52.397+0000 7fd3016f5640 1 --1- >> v1:172.21.15.73:6789/0 conn(0x7fd2fc0a22f0 0x7fd2fc0a2700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.73:6789/0 says I am v1:172.21.15.73:44774/0 (socket says 172.21.15.73:44774) 2024-08-02T23:39:52.400 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:39:52.397+0000 7fd3016f5640 1 -- 172.21.15.73:0/311096033 learned_addr learned my addr 172.21.15.73:0/311096033 (peer_addr_for_me v1:172.21.15.73:0/0) 2024-08-02T23:39:52.400 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:39:52.397+0000 7fd300ef4640 1 --2- 172.21.15.73:0/311096033 >> v2:172.21.15.73:3300/0 conn(0x7fd2fc0a2e10 0x7fd2fc1398c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-02T23:39:52.400 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:39:52.397+0000 7fd2f3fff640 1 -- 172.21.15.73:0/311096033 <== mon.0 v1:172.21.15.73:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1718258129 0 0) 0x7fd2fc09c040 con 0x7fd2fc0a22f0 2024-08-02T23:39:52.400 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:39:52.397+0000 7fd2f3fff640 1 -- 172.21.15.73:0/311096033 --> v1:172.21.15.73:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd2e80034f0 con 0x7fd2fc0a22f0 2024-08-02T23:39:52.400 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:39:52.397+0000 7fd300ef4640 1 -- 172.21.15.73:0/311096033 >> v1:172.21.15.73:6789/0 conn(0x7fd2fc0a22f0 legacy=0x7fd2fc0a2700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-02T23:39:52.400 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:39:52.397+0000 7fd300ef4640 1 -- 172.21.15.73:0/311096033 --> v2:172.21.15.73:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2fc09c390 con 0x7fd2fc0a2e10 2024-08-02T23:39:52.401 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:39:52.398+0000 7fd300ef4640 1 --2- 172.21.15.73:0/311096033 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd2fc0a2e10 0x7fd2fc1398c0 crc :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59f5dc4ed0cf7eb6 server_cookie=0 in_seq=0 out_seq=0 2024-08-02T23:39:52.401 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:39:52.399+0000 7fd2f3fff640 1 -- 172.21.15.73:0/311096033 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fd2ec009ce0 con 0x7fd2fc0a2e10 2024-08-02T23:39:52.401 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:39:52.399+0000 7fd2f3fff640 1 -- 172.21.15.73:0/311096033 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7fd2ec009e40 con 0x7fd2fc0a2e10 2024-08-02T23:39:52.402 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:39:52.399+0000 7fd2f3fff640 1 -- 172.21.15.73:0/311096033 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fd2ec00e360 con 0x7fd2fc0a2e10 2024-08-02T23:39:52.402 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:39:52.399+0000 7fd303959640 1 -- 172.21.15.73:0/311096033 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd2fc0a2e10 msgr2=0x7fd2fc1398c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-02T23:39:52.402 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:39:52.399+0000 7fd303959640 1 --2- 172.21.15.73:0/311096033 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd2fc0a2e10 0x7fd2fc1398c0 crc :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:39:52.402 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:39:52.399+0000 7fd303959640 1 -- 172.21.15.73:0/311096033 shutdown_connections 2024-08-02T23:39:52.402 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:39:52.399+0000 7fd303959640 1 --2- 172.21.15.73:0/311096033 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd2fc0a2e10 0x7fd2fc1398c0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:39:52.402 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:39:52.399+0000 7fd303959640 1 -- 172.21.15.73:0/311096033 >> 172.21.15.73:0/311096033 conn(0x7fd2fc06cf80 msgr2=0x7fd2fc06d390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-02T23:39:52.402 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:39:52.400+0000 7fd303959640 1 -- 172.21.15.73:0/311096033 shutdown_connections 2024-08-02T23:39:52.402 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:39:52.400+0000 7fd303959640 1 -- 172.21.15.73:0/311096033 wait complete. 2024-08-02T23:39:52.402 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:39:52.400+0000 7fd303959640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-02T23:39:52.403 INFO:tasks.ceph.mgr.x.smithi131.stderr:2024-08-02T23:39:52.399+0000 7ff194b6df00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-02T23:39:52.403 INFO:tasks.ceph.mgr.x.smithi131.stderr:2024-08-02T23:39:52.399+0000 7ff194b6df00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-02T23:39:52.404 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:39:52.401+0000 7fd303959640 1 Processor -- start 2024-08-02T23:39:52.405 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:39:52.401+0000 7fd303959640 1 -- start start 2024-08-02T23:39:52.405 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:39:52.401+0000 7fd303959640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd2fc0a22f0 0x7fd2fc11f3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-02T23:39:52.405 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:39:52.401+0000 7fd303959640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 -- 0x7fd2fc11f030 con 0x7fd2fc0a22f0 2024-08-02T23:39:52.405 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:39:52.401+0000 7fd3016f5640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd2fc0a22f0 0x7fd2fc11f3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-02T23:39:52.405 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:39:52.402+0000 7fd3016f5640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd2fc0a22f0 0x7fd2fc11f3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:51188/0 (socket says 172.21.15.73:51188) 2024-08-02T23:39:52.405 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:39:52.402+0000 7fd3016f5640 1 -- 172.21.15.73:0/1439001265 learned_addr learned my addr 172.21.15.73:0/1439001265 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-08-02T23:39:52.405 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:39:52.402+0000 7fd3016f5640 1 -- 172.21.15.73:0/1439001265 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2fc1205e0 con 0x7fd2fc0a22f0 2024-08-02T23:39:52.405 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:39:52.402+0000 7fd3016f5640 1 --2- 172.21.15.73:0/1439001265 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd2fc0a22f0 0x7fd2fc11f3f0 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-02T23:39:52.405 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:39:52.402+0000 7fd2f1ffb640 1 -- 172.21.15.73:0/1439001265 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fd2f80059d0 con 0x7fd2fc0a22f0 2024-08-02T23:39:52.405 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:39:52.402+0000 7fd2f1ffb640 1 -- 172.21.15.73:0/1439001265 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7fd2f8005b30 con 0x7fd2fc0a22f0 2024-08-02T23:39:52.405 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:39:52.402+0000 7fd303959640 1 -- 172.21.15.73:0/1439001265 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2fc121400 con 0x7fd2fc0a22f0 2024-08-02T23:39:52.406 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:39:52.402+0000 7fd2f1ffb640 1 -- 172.21.15.73:0/1439001265 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fd2f8005dd0 con 0x7fd2fc0a22f0 2024-08-02T23:39:52.406 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:39:52.402+0000 7fd303959640 1 -- 172.21.15.73:0/1439001265 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2fc11e980 con 0x7fd2fc0a22f0 2024-08-02T23:39:52.406 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:39:52.402+0000 7fd2f1ffb640 1 -- 172.21.15.73:0/1439001265 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (crc 0 0 0) 0x7fd2f8007440 con 0x7fd2fc0a22f0 2024-08-02T23:39:52.406 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:39:52.402+0000 7fd2f1ffb640 1 -- 172.21.15.73:0/1439001265 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (crc 0 0 0) 0x7fd2f800e960 con 0x7fd2fc0a22f0 2024-08-02T23:39:52.406 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:39:52.403+0000 7fd303959640 1 -- 172.21.15.73:0/1439001265 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd2fc069280 con 0x7fd2fc0a22f0 2024-08-02T23:39:52.407 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:39:52.405+0000 7fd2f1ffb640 1 -- 172.21.15.73:0/1439001265 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73482 (crc 0 0 0) 0x7fd2f800eb20 con 0x7fd2fc0a22f0 2024-08-02T23:39:52.470 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:39:52.468+0000 7fd303959640 1 -- 172.21.15.73:0/1439001265 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7fd2fc0a2700 con 0x7fd2fc0a22f0 2024-08-02T23:39:52.471 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:39:52.469+0000 7fd2f1ffb640 1 -- 172.21.15.73:0/1439001265 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) ==== 124+0+0 (crc 0 0 0) 0x7fd2f800ecc0 con 0x7fd2fc0a22f0 2024-08-02T23:39:52.471 INFO:teuthology.orchestra.run.smithi073.stderr:adjusted tunables profile to default 2024-08-02T23:39:52.472 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:39:52.470+0000 7fd303959640 1 -- 172.21.15.73:0/1439001265 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd2fc0a22f0 msgr2=0x7fd2fc11f3f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-02T23:39:52.472 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:39:52.470+0000 7fd303959640 1 --2- 172.21.15.73:0/1439001265 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd2fc0a22f0 0x7fd2fc11f3f0 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:39:52.472 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:39:52.470+0000 7fd303959640 1 -- 172.21.15.73:0/1439001265 shutdown_connections 2024-08-02T23:39:52.472 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:39:52.470+0000 7fd303959640 1 --2- 172.21.15.73:0/1439001265 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd2fc0a22f0 0x7fd2fc11f3f0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:39:52.473 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:39:52.470+0000 7fd303959640 1 -- 172.21.15.73:0/1439001265 >> 172.21.15.73:0/1439001265 conn(0x7fd2fc06cf80 msgr2=0x7fd2fc1340b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-02T23:39:52.473 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:39:52.470+0000 7fd303959640 1 -- 172.21.15.73:0/1439001265 shutdown_connections 2024-08-02T23:39:52.473 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:39:52.470+0000 7fd303959640 1 -- 172.21.15.73:0/1439001265 wait complete. 2024-08-02T23:39:52.482 INFO:tasks.ceph:check_enable_crimson: True 2024-08-02T23:39:52.483 INFO:tasks.ceph:check_enable_crimson: setting set-allow-crimson 2024-08-02T23:39:52.483 DEBUG:teuthology.orchestra.run.smithi073:> sudo ceph --cluster ceph osd set-allow-crimson --yes-i-really-mean-it 2024-08-02T23:39:52.506 INFO:tasks.ceph.mgr.x.smithi131.stderr:2024-08-02T23:39:52.504+0000 7ff194b6df00 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-08-02T23:39:52.584 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:39:52.578+0000 7f1cbcf99640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-02T23:39:52.585 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:39:52.583+0000 7f1cbcf99640 1 Processor -- start 2024-08-02T23:39:52.585 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:39:52.583+0000 7f1cbcf99640 1 -- start start 2024-08-02T23:39:52.586 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:39:52.584+0000 7f1cbcf99640 1 --2- >> v2:172.21.15.73:3300/0 conn(0x7f1cb812d690 0x7f1cb812daa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-02T23:39:52.586 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:39:52.584+0000 7f1cbcf99640 1 -- --> v1:172.21.15.73:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1cb806e4f0 con 0x7f1cb81306a0 2024-08-02T23:39:52.586 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:39:52.584+0000 7f1cbcf99640 1 -- --> v2:172.21.15.73:3300/0 -- mon_getmap magic: 0 -- 0x7f1cb812d240 con 0x7f1cb812d690 2024-08-02T23:39:52.586 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:39:52.584+0000 7f1cb659c640 1 --2- >> v2:172.21.15.73:3300/0 conn(0x7f1cb812d690 0x7f1cb812daa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-02T23:39:52.587 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:39:52.584+0000 7f1cadd9b640 1 --1- >> v1:172.21.15.73:6789/0 conn(0x7f1cb81306a0 0x7f1cb8130ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.73:6789/0 says I am v1:172.21.15.73:44780/0 (socket says 172.21.15.73:44780) 2024-08-02T23:39:52.587 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:39:52.584+0000 7f1cadd9b640 1 -- 172.21.15.73:0/2851007700 learned_addr learned my addr 172.21.15.73:0/2851007700 (peer_addr_for_me v1:172.21.15.73:0/0) 2024-08-02T23:39:52.587 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:39:52.584+0000 7f1cb659c640 1 -- 172.21.15.73:0/2851007700 >> v1:172.21.15.73:6789/0 conn(0x7f1cb81306a0 legacy=0x7f1cb8130ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-02T23:39:52.587 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:39:52.584+0000 7f1cb659c640 1 -- 172.21.15.73:0/2851007700 --> v2:172.21.15.73:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1cb809e870 con 0x7f1cb812d690 2024-08-02T23:39:52.587 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:39:52.584+0000 7f1cb659c640 1 --2- 172.21.15.73:0/2851007700 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1cb812d690 0x7f1cb812daa0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a7844bfd1f6658 server_cookie=0 in_seq=0 out_seq=0 2024-08-02T23:39:52.587 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:39:52.584+0000 7f1cb5d9b640 1 -- 172.21.15.73:0/2851007700 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f1ca8009ce0 con 0x7f1cb812d690 2024-08-02T23:39:52.587 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:39:52.584+0000 7f1cb5d9b640 1 -- 172.21.15.73:0/2851007700 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f1ca8009e40 con 0x7f1cb812d690 2024-08-02T23:39:52.587 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:39:52.584+0000 7f1cb5d9b640 1 -- 172.21.15.73:0/2851007700 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f1ca800f470 con 0x7f1cb812d690 2024-08-02T23:39:52.588 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:39:52.585+0000 7f1cbcf99640 1 -- 172.21.15.73:0/2851007700 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1cb812d690 msgr2=0x7f1cb812daa0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-02T23:39:52.588 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:39:52.585+0000 7f1cbcf99640 1 --2- 172.21.15.73:0/2851007700 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1cb812d690 0x7f1cb812daa0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:39:52.588 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:39:52.585+0000 7f1cbcf99640 1 -- 172.21.15.73:0/2851007700 shutdown_connections 2024-08-02T23:39:52.589 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:39:52.585+0000 7f1cbcf99640 1 --2- 172.21.15.73:0/2851007700 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1cb812d690 0x7f1cb812daa0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:39:52.589 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:39:52.585+0000 7f1cbcf99640 1 -- 172.21.15.73:0/2851007700 >> 172.21.15.73:0/2851007700 conn(0x7f1cb80a0590 msgr2=0x7f1cb80a09a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-02T23:39:52.589 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:39:52.585+0000 7f1cbcf99640 1 -- 172.21.15.73:0/2851007700 shutdown_connections 2024-08-02T23:39:52.589 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:39:52.585+0000 7f1cbcf99640 1 -- 172.21.15.73:0/2851007700 wait complete. 2024-08-02T23:39:52.589 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:39:52.585+0000 7f1cbcf99640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-02T23:39:52.589 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:39:52.586+0000 7f1cbcf99640 1 Processor -- start 2024-08-02T23:39:52.589 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:39:52.586+0000 7f1cbcf99640 1 -- start start 2024-08-02T23:39:52.591 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:39:52.587+0000 7f1cbcf99640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1cb812d690 0x7f1cb812aa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-02T23:39:52.591 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:39:52.587+0000 7f1cbcf99640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 -- 0x7f1cb811fb80 con 0x7f1cb812d690 2024-08-02T23:39:52.591 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:39:52.587+0000 7f1cb659c640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1cb812d690 0x7f1cb812aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-02T23:39:52.591 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:39:52.587+0000 7f1cb659c640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1cb812d690 0x7f1cb812aa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:51204/0 (socket says 172.21.15.73:51204) 2024-08-02T23:39:52.591 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:39:52.587+0000 7f1cb659c640 1 -- 172.21.15.73:0/2492253825 learned_addr learned my addr 172.21.15.73:0/2492253825 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-08-02T23:39:52.591 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:39:52.588+0000 7f1cb659c640 1 -- 172.21.15.73:0/2492253825 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1cb811f080 con 0x7f1cb812d690 2024-08-02T23:39:52.591 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:39:52.588+0000 7f1cb659c640 1 --2- 172.21.15.73:0/2492253825 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1cb812d690 0x7f1cb812aa70 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-02T23:39:52.591 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:39:52.588+0000 7f1caf7fe640 1 -- 172.21.15.73:0/2492253825 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f1ca8013dc0 con 0x7f1cb812d690 2024-08-02T23:39:52.591 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:39:52.588+0000 7f1caf7fe640 1 -- 172.21.15.73:0/2492253825 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f1ca8011420 con 0x7f1cb812d690 2024-08-02T23:39:52.591 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:39:52.588+0000 7f1cbcf99640 1 -- 172.21.15.73:0/2492253825 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1cb811d690 con 0x7f1cb812d690 2024-08-02T23:39:52.592 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:39:52.588+0000 7f1caf7fe640 1 -- 172.21.15.73:0/2492253825 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f1ca80116c0 con 0x7f1cb812d690 2024-08-02T23:39:52.592 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:39:52.588+0000 7f1cbcf99640 1 -- 172.21.15.73:0/2492253825 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1cb811ed70 con 0x7f1cb812d690 2024-08-02T23:39:52.592 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:39:52.588+0000 7f1caf7fe640 1 -- 172.21.15.73:0/2492253825 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (crc 0 0 0) 0x7f1ca8011c00 con 0x7f1cb812d690 2024-08-02T23:39:52.592 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:39:52.588+0000 7f1caf7fe640 1 -- 172.21.15.73:0/2492253825 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 725+0+0 (crc 0 0 0) 0x7f1ca8018040 con 0x7f1cb812d690 2024-08-02T23:39:52.592 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:39:52.589+0000 7f1cbcf99640 1 -- 172.21.15.73:0/2492253825 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1cb811e5f0 con 0x7f1cb812d690 2024-08-02T23:39:52.592 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:39:52.591+0000 7f1caf7fe640 1 -- 172.21.15.73:0/2492253825 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73482 (crc 0 0 0) 0x7f1ca80103b0 con 0x7f1cb812d690 2024-08-02T23:39:52.665 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:39:52.663+0000 7f1cbcf99640 1 -- 172.21.15.73:0/2492253825 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "osd set-allow-crimson", "yes_i_really_mean_it": true} v 0) -- 0x7f1cb80a2320 con 0x7f1cb812d690 2024-08-02T23:39:52.666 INFO:tasks.ceph.mon.a.smithi073.stderr:2024-08-02T23:39:52.663+0000 7fd36c7f1640 -1 WARNING: experimental feature 'crimson' is enabled 2024-08-02T23:39:52.666 INFO:tasks.ceph.mon.a.smithi073.stderr:Please be aware that this feature is experimental, untested, 2024-08-02T23:39:52.666 INFO:tasks.ceph.mon.a.smithi073.stderr:unsupported, and may result in data corruption, data loss, 2024-08-02T23:39:52.666 INFO:tasks.ceph.mon.a.smithi073.stderr:and/or irreparable damage to your cluster. Do not use 2024-08-02T23:39:52.666 INFO:tasks.ceph.mon.a.smithi073.stderr:feature with important data. 2024-08-02T23:39:52.667 INFO:tasks.ceph.mon.a.smithi073.stderr: 2024-08-02T23:39:52.669 INFO:tasks.ceph.mgr.x.smithi131.stderr:2024-08-02T23:39:52.667+0000 7ff194b6df00 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-08-02T23:39:52.723 INFO:tasks.ceph.mgr.x.smithi131.stderr:2024-08-02T23:39:52.720+0000 7ff194b6df00 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-08-02T23:39:52.973 INFO:tasks.ceph.mgr.x.smithi131.stderr:2024-08-02T23:39:52.970+0000 7ff194b6df00 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-08-02T23:39:53.019 INFO:tasks.ceph.mgr.x.smithi131.stderr:2024-08-02T23:39:53.016+0000 7ff194b6df00 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-08-02T23:39:53.472 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:39:53.470+0000 7f1caf7fe640 1 -- 172.21.15.73:0/2492253825 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "osd set-allow-crimson", "yes_i_really_mean_it": true}]=0 v3) ==== 99+0+0 (crc 0 0 0) 0x7f1ca8010550 con 0x7f1cb812d690 2024-08-02T23:39:53.473 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:39:53.471+0000 7f1cbcf99640 1 -- 172.21.15.73:0/2492253825 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1cb812d690 msgr2=0x7f1cb812aa70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-02T23:39:53.473 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:39:53.471+0000 7f1cbcf99640 1 --2- 172.21.15.73:0/2492253825 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1cb812d690 0x7f1cb812aa70 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:39:53.473 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:39:53.471+0000 7f1cbcf99640 1 -- 172.21.15.73:0/2492253825 shutdown_connections 2024-08-02T23:39:53.473 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:39:53.471+0000 7f1cbcf99640 1 --2- 172.21.15.73:0/2492253825 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1cb812d690 0x7f1cb812aa70 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:39:53.473 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:39:53.471+0000 7f1cbcf99640 1 -- 172.21.15.73:0/2492253825 >> 172.21.15.73:0/2492253825 conn(0x7f1cb80a0590 msgr2=0x7f1cb809ca00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-02T23:39:53.473 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:39:53.471+0000 7f1cbcf99640 1 -- 172.21.15.73:0/2492253825 shutdown_connections 2024-08-02T23:39:53.474 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:39:53.471+0000 7f1cbcf99640 1 -- 172.21.15.73:0/2492253825 wait complete. 2024-08-02T23:39:53.485 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-08-02T23:39:53.486 DEBUG:teuthology.orchestra.run.smithi073:> set -ex 2024-08-02T23:39:53.486 DEBUG:teuthology.orchestra.run.smithi073:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-08-02T23:39:53.512 DEBUG:teuthology.orchestra.run.smithi073:> set -ex 2024-08-02T23:39:53.512 DEBUG:teuthology.orchestra.run.smithi073:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-08-02T23:39:53.581 DEBUG:teuthology.orchestra.run.smithi131:> set -ex 2024-08-02T23:39:53.581 DEBUG:teuthology.orchestra.run.smithi131:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-08-02T23:39:53.606 DEBUG:teuthology.orchestra.run.smithi131:> set -ex 2024-08-02T23:39:53.606 DEBUG:teuthology.orchestra.run.smithi131:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-08-02T23:39:53.671 DEBUG:teuthology.orchestra.run.smithi131:> sudo ceph --cluster ceph osd new bd37ae5d-55cc-4a6d-b943-8de7e40fd1bb 0 2024-08-02T23:39:53.768 INFO:tasks.ceph.mgr.x.smithi131.stderr:2024-08-02T23:39:53.765+0000 7ff194b6df00 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-08-02T23:39:53.816 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:53.808+0000 7fbd23559640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-02T23:39:53.817 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:53.814+0000 7fbd23559640 1 Processor -- start 2024-08-02T23:39:53.817 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:53.814+0000 7fbd23559640 1 -- start start 2024-08-02T23:39:53.817 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:53.815+0000 7fbd23559640 1 --2- >> v2:172.21.15.73:3300/0 conn(0x7fbd1c11bfc0 0x7fbd1c11e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-02T23:39:53.818 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:53.815+0000 7fbd23559640 1 -- --> v1:172.21.15.73:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbd1c11e9b0 con 0x7fbd1c1192e0 2024-08-02T23:39:53.818 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:53.815+0000 7fbd23559640 1 -- --> v2:172.21.15.73:3300/0 -- mon_getmap magic: 0 -- 0x7fbd1c11fb70 con 0x7fbd1c11bfc0 2024-08-02T23:39:53.818 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:53.815+0000 7fbd20af4640 1 --2- >> v2:172.21.15.73:3300/0 conn(0x7fbd1c11bfc0 0x7fbd1c11e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-02T23:39:53.818 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:53.815+0000 7fbd212f5640 1 --1- >> v1:172.21.15.73:6789/0 conn(0x7fbd1c1192e0 0x7fbd1c1196f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.73:6789/0 says I am v1:172.21.15.131:55898/0 (socket says 172.21.15.131:55898) 2024-08-02T23:39:53.818 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:53.815+0000 7fbd20af4640 1 --2- >> v2:172.21.15.73:3300/0 conn(0x7fbd1c11bfc0 0x7fbd1c11e3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.131:60482/0 (socket says 172.21.15.131:60482) 2024-08-02T23:39:53.818 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:53.815+0000 7fbd212f5640 1 -- 172.21.15.131:0/3171221595 learned_addr learned my addr 172.21.15.131:0/3171221595 (peer_addr_for_me v1:172.21.15.131:0/0) 2024-08-02T23:39:53.819 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:53.815+0000 7fbd20af4640 1 -- 172.21.15.131:0/3171221595 >> v1:172.21.15.73:6789/0 conn(0x7fbd1c1192e0 legacy=0x7fbd1c1196f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-02T23:39:53.819 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:53.815+0000 7fbd20af4640 1 -- 172.21.15.131:0/3171221595 --> v2:172.21.15.73:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd1c11fce0 con 0x7fbd1c11bfc0 2024-08-02T23:39:53.819 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:53.815+0000 7fbd20af4640 1 --2- 172.21.15.131:0/3171221595 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fbd1c11bfc0 0x7fbd1c11e3e0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4d1e52fccf6e89b server_cookie=0 in_seq=0 out_seq=0 2024-08-02T23:39:53.820 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:53.816+0000 7fbd13fff640 1 -- 172.21.15.131:0/3171221595 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fbd0c009ce0 con 0x7fbd1c11bfc0 2024-08-02T23:39:53.820 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:53.816+0000 7fbd13fff640 1 -- 172.21.15.131:0/3171221595 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7fbd0c009e40 con 0x7fbd1c11bfc0 2024-08-02T23:39:53.820 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:53.816+0000 7fbd13fff640 1 -- 172.21.15.131:0/3171221595 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fbd0c00e360 con 0x7fbd1c11bfc0 2024-08-02T23:39:53.820 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:53.816+0000 7fbd23559640 1 -- 172.21.15.131:0/3171221595 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fbd1c11bfc0 msgr2=0x7fbd1c11e3e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-02T23:39:53.820 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:53.816+0000 7fbd23559640 1 --2- 172.21.15.131:0/3171221595 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fbd1c11bfc0 0x7fbd1c11e3e0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:39:53.820 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:53.816+0000 7fbd23559640 1 -- 172.21.15.131:0/3171221595 shutdown_connections 2024-08-02T23:39:53.820 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:53.816+0000 7fbd23559640 1 --2- 172.21.15.131:0/3171221595 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fbd1c11bfc0 0x7fbd1c11e3e0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:39:53.820 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:53.816+0000 7fbd23559640 1 -- 172.21.15.131:0/3171221595 >> 172.21.15.131:0/3171221595 conn(0x7fbd1c06eb80 msgr2=0x7fbd1c06ef90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-02T23:39:53.820 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:53.816+0000 7fbd23559640 1 -- 172.21.15.131:0/3171221595 shutdown_connections 2024-08-02T23:39:53.820 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:53.816+0000 7fbd23559640 1 -- 172.21.15.131:0/3171221595 wait complete. 2024-08-02T23:39:53.820 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:53.817+0000 7fbd23559640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-02T23:39:53.822 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:53.817+0000 7fbd23559640 1 Processor -- start 2024-08-02T23:39:53.822 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:53.817+0000 7fbd23559640 1 -- start start 2024-08-02T23:39:53.823 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:53.817+0000 7fbd23559640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fbd1c1192e0 0x7fbd1c1adac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-02T23:39:53.823 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:53.817+0000 7fbd23559640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 -- 0x7fbd1c1ae090 con 0x7fbd1c1192e0 2024-08-02T23:39:53.823 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:53.818+0000 7fbd212f5640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fbd1c1192e0 0x7fbd1c1adac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-02T23:39:53.823 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:53.818+0000 7fbd212f5640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fbd1c1192e0 0x7fbd1c1adac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.131:60494/0 (socket says 172.21.15.131:60494) 2024-08-02T23:39:53.823 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:53.818+0000 7fbd212f5640 1 -- 172.21.15.131:0/41663084 learned_addr learned my addr 172.21.15.131:0/41663084 (peer_addr_for_me v2:172.21.15.131:0/0) 2024-08-02T23:39:53.823 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:53.818+0000 7fbd212f5640 1 -- 172.21.15.131:0/41663084 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd1c1ae1d0 con 0x7fbd1c1192e0 2024-08-02T23:39:53.823 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:53.818+0000 7fbd212f5640 1 --2- 172.21.15.131:0/41663084 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fbd1c1192e0 0x7fbd1c1adac0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-02T23:39:53.823 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:53.818+0000 7fbd11ffb640 1 -- 172.21.15.131:0/41663084 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fbd180059d0 con 0x7fbd1c1192e0 2024-08-02T23:39:53.823 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:53.818+0000 7fbd11ffb640 1 -- 172.21.15.131:0/41663084 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7fbd18005b30 con 0x7fbd1c1192e0 2024-08-02T23:39:53.824 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:53.818+0000 7fbd23559640 1 -- 172.21.15.131:0/41663084 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbd1c1ae420 con 0x7fbd1c1192e0 2024-08-02T23:39:53.824 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:53.818+0000 7fbd11ffb640 1 -- 172.21.15.131:0/41663084 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fbd18005dd0 con 0x7fbd1c1192e0 2024-08-02T23:39:53.824 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:53.819+0000 7fbd23559640 1 -- 172.21.15.131:0/41663084 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbd1c1a7e20 con 0x7fbd1c1192e0 2024-08-02T23:39:53.824 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:53.819+0000 7fbd11ffb640 1 -- 172.21.15.131:0/41663084 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (crc 0 0 0) 0x7fbd18007440 con 0x7fbd1c1192e0 2024-08-02T23:39:53.824 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:53.819+0000 7fbd11ffb640 1 -- 172.21.15.131:0/41663084 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 747+0+0 (crc 0 0 0) 0x7fbd1800ea10 con 0x7fbd1c1192e0 2024-08-02T23:39:53.824 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:53.819+0000 7fbd23559640 1 -- 172.21.15.131:0/41663084 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbd1c11f5a0 con 0x7fbd1c1192e0 2024-08-02T23:39:53.825 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:53.822+0000 7fbd11ffb640 1 -- 172.21.15.131:0/41663084 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73482 (crc 0 0 0) 0x7fbd1800e1c0 con 0x7fbd1c1192e0 2024-08-02T23:39:53.860 INFO:tasks.ceph.mgr.x.smithi131.stderr:2024-08-02T23:39:53.858+0000 7ff194b6df00 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-08-02T23:39:53.896 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:53.893+0000 7fbd23559640 1 -- 172.21.15.131:0/41663084 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "osd new", "uuid": "bd37ae5d-55cc-4a6d-b943-8de7e40fd1bb", "id": 0} v 0) -- 0x7fbd1c11c060 con 0x7fbd1c1192e0 2024-08-02T23:39:53.897 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:53.895+0000 7fbd11ffb640 1 -- 172.21.15.131:0/41663084 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "bd37ae5d-55cc-4a6d-b943-8de7e40fd1bb", "id": 0}]=0 v4) ==== 112+0+2 (crc 0 0 0) 0x7fbd1800eca0 con 0x7fbd1c1192e0 2024-08-02T23:39:53.897 INFO:teuthology.orchestra.run.smithi131.stdout:0 2024-08-02T23:39:53.899 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:53.896+0000 7fbd23559640 1 -- 172.21.15.131:0/41663084 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fbd1c1192e0 msgr2=0x7fbd1c1adac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-02T23:39:53.899 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:53.896+0000 7fbd23559640 1 --2- 172.21.15.131:0/41663084 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fbd1c1192e0 0x7fbd1c1adac0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:39:53.899 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:53.896+0000 7fbd23559640 1 -- 172.21.15.131:0/41663084 shutdown_connections 2024-08-02T23:39:53.899 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:53.896+0000 7fbd23559640 1 --2- 172.21.15.131:0/41663084 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fbd1c1192e0 0x7fbd1c1adac0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:39:53.899 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:53.896+0000 7fbd23559640 1 -- 172.21.15.131:0/41663084 >> 172.21.15.131:0/41663084 conn(0x7fbd1c06eb80 msgr2=0x7fbd1c111d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-02T23:39:53.899 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:53.896+0000 7fbd23559640 1 -- 172.21.15.131:0/41663084 shutdown_connections 2024-08-02T23:39:53.899 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:53.896+0000 7fbd23559640 1 -- 172.21.15.131:0/41663084 wait complete. 2024-08-02T23:39:53.904 INFO:tasks.ceph.mgr.x.smithi131.stderr:2024-08-02T23:39:53.901+0000 7ff194b6df00 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-08-02T23:39:53.909 DEBUG:teuthology.orchestra.run.smithi131:> sudo ceph --cluster ceph osd new 6bf29a4c-3240-4fce-8803-66926d0df50c 1 2024-08-02T23:39:54.012 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:54.004+0000 7f9462599640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-02T23:39:54.013 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:54.010+0000 7f9462599640 1 Processor -- start 2024-08-02T23:39:54.013 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:54.010+0000 7f9462599640 1 -- start start 2024-08-02T23:39:54.013 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:54.010+0000 7f9462599640 1 --2- >> v2:172.21.15.73:3300/0 conn(0x7f945c11c040 0x7f945c11e460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-02T23:39:54.013 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:54.010+0000 7f9462599640 1 -- --> v1:172.21.15.73:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f945c11ea60 con 0x7f945c1192a0 2024-08-02T23:39:54.013 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:54.010+0000 7f9462599640 1 -- --> v2:172.21.15.73:3300/0 -- mon_getmap magic: 0 -- 0x7f945c11fc20 con 0x7f945c11c040 2024-08-02T23:39:54.016 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:54.011+0000 7f945b7fe640 1 --2- >> v2:172.21.15.73:3300/0 conn(0x7f945c11c040 0x7f945c11e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-02T23:39:54.016 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:54.011+0000 7f945bfff640 1 --1- >> v1:172.21.15.73:6789/0 conn(0x7f945c1192a0 0x7f945c1196b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.73:6789/0 says I am v1:172.21.15.131:55900/0 (socket says 172.21.15.131:55900) 2024-08-02T23:39:54.016 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:54.011+0000 7f945b7fe640 1 --2- >> v2:172.21.15.73:3300/0 conn(0x7f945c11c040 0x7f945c11e460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.131:60504/0 (socket says 172.21.15.131:60504) 2024-08-02T23:39:54.016 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:54.011+0000 7f945b7fe640 1 -- 172.21.15.131:0/472467781 learned_addr learned my addr 172.21.15.131:0/472467781 (peer_addr_for_me v2:172.21.15.131:0/0) 2024-08-02T23:39:54.016 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:54.011+0000 7f945b7fe640 1 -- 172.21.15.131:0/472467781 >> v1:172.21.15.73:6789/0 conn(0x7f945c1192a0 legacy=0x7f945c1196b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-02T23:39:54.016 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:54.011+0000 7f945b7fe640 1 -- 172.21.15.131:0/472467781 --> v2:172.21.15.73:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f945c11fd90 con 0x7f945c11c040 2024-08-02T23:39:54.016 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:54.011+0000 7f945b7fe640 1 --2- 172.21.15.131:0/472467781 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f945c11c040 0x7f945c11e460 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24e91960735e608e server_cookie=0 in_seq=0 out_seq=0 2024-08-02T23:39:54.016 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:54.011+0000 7f945affd640 1 -- 172.21.15.131:0/472467781 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f944c009dd0 con 0x7f945c11c040 2024-08-02T23:39:54.016 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:54.011+0000 7f945affd640 1 -- 172.21.15.131:0/472467781 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f944c00e230 con 0x7f945c11c040 2024-08-02T23:39:54.016 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:54.011+0000 7f945affd640 1 -- 172.21.15.131:0/472467781 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f944c00e4d0 con 0x7f945c11c040 2024-08-02T23:39:54.017 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:54.012+0000 7f9462599640 1 -- 172.21.15.131:0/472467781 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f945c11c040 msgr2=0x7f945c11e460 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-02T23:39:54.017 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:54.012+0000 7f9462599640 1 --2- 172.21.15.131:0/472467781 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f945c11c040 0x7f945c11e460 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:39:54.017 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:54.012+0000 7f9462599640 1 -- 172.21.15.131:0/472467781 shutdown_connections 2024-08-02T23:39:54.017 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:54.012+0000 7f9462599640 1 --2- 172.21.15.131:0/472467781 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f945c11c040 0x7f945c11e460 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:39:54.017 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:54.012+0000 7f9462599640 1 -- 172.21.15.131:0/472467781 >> 172.21.15.131:0/472467781 conn(0x7f945c06e9a0 msgr2=0x7f945c06edb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-02T23:39:54.017 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:54.012+0000 7f9462599640 1 -- 172.21.15.131:0/472467781 shutdown_connections 2024-08-02T23:39:54.017 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:54.012+0000 7f9462599640 1 -- 172.21.15.131:0/472467781 wait complete. 2024-08-02T23:39:54.017 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:54.012+0000 7f9462599640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-02T23:39:54.017 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:54.013+0000 7f9462599640 1 Processor -- start 2024-08-02T23:39:54.017 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:54.013+0000 7f9462599640 1 -- start start 2024-08-02T23:39:54.019 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:54.013+0000 7f9462599640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f945c1192a0 0x7f945c1adaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-02T23:39:54.019 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:54.013+0000 7f9462599640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 -- 0x7f945c1ae0c0 con 0x7f945c1192a0 2024-08-02T23:39:54.019 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:54.014+0000 7f945bfff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f945c1192a0 0x7f945c1adaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-02T23:39:54.019 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:54.014+0000 7f945bfff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f945c1192a0 0x7f945c1adaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.131:60520/0 (socket says 172.21.15.131:60520) 2024-08-02T23:39:54.019 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:54.014+0000 7f945bfff640 1 -- 172.21.15.131:0/1968943012 learned_addr learned my addr 172.21.15.131:0/1968943012 (peer_addr_for_me v2:172.21.15.131:0/0) 2024-08-02T23:39:54.019 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:54.014+0000 7f945bfff640 1 -- 172.21.15.131:0/1968943012 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f945c1ae200 con 0x7f945c1192a0 2024-08-02T23:39:54.019 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:54.014+0000 7f945bfff640 1 --2- 172.21.15.131:0/1968943012 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f945c1192a0 0x7f945c1adaf0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-02T23:39:54.019 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:54.014+0000 7f9458ff9640 1 -- 172.21.15.131:0/1968943012 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f94440071e0 con 0x7f945c1192a0 2024-08-02T23:39:54.019 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:54.014+0000 7f9458ff9640 1 -- 172.21.15.131:0/1968943012 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f9444007340 con 0x7f945c1192a0 2024-08-02T23:39:54.019 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:54.014+0000 7f9462599640 1 -- 172.21.15.131:0/1968943012 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f945c1ae450 con 0x7f945c1192a0 2024-08-02T23:39:54.020 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:54.014+0000 7f9458ff9640 1 -- 172.21.15.131:0/1968943012 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f94440075e0 con 0x7f945c1192a0 2024-08-02T23:39:54.020 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:54.014+0000 7f9462599640 1 -- 172.21.15.131:0/1968943012 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f945c1a7e50 con 0x7f945c1192a0 2024-08-02T23:39:54.020 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:54.014+0000 7f9458ff9640 1 -- 172.21.15.131:0/1968943012 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (crc 0 0 0) 0x7f9444005b70 con 0x7f945c1192a0 2024-08-02T23:39:54.020 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:54.014+0000 7f9458ff9640 1 -- 172.21.15.131:0/1968943012 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 862+0+0 (crc 0 0 0) 0x7f944400fa00 con 0x7f945c1192a0 2024-08-02T23:39:54.020 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:54.016+0000 7f9462599640 1 -- 172.21.15.131:0/1968943012 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f945c11f650 con 0x7f945c1192a0 2024-08-02T23:39:54.020 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:54.018+0000 7f9458ff9640 1 -- 172.21.15.131:0/1968943012 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73482 (crc 0 0 0) 0x7f9444005e20 con 0x7f945c1192a0 2024-08-02T23:39:54.086 INFO:tasks.ceph.mgr.x.smithi131.stderr:2024-08-02T23:39:54.084+0000 7ff194b6df00 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-08-02T23:39:54.102 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:54.099+0000 7f9462599640 1 -- 172.21.15.131:0/1968943012 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "osd new", "uuid": "6bf29a4c-3240-4fce-8803-66926d0df50c", "id": 1} v 0) -- 0x7f945c11c0e0 con 0x7f945c1192a0 2024-08-02T23:39:54.103 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:54.101+0000 7f9458ff9640 1 -- 172.21.15.131:0/1968943012 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "6bf29a4c-3240-4fce-8803-66926d0df50c", "id": 1}]=0 v5) ==== 112+0+2 (crc 0 0 0) 0x7f944400cde0 con 0x7f945c1192a0 2024-08-02T23:39:54.103 INFO:teuthology.orchestra.run.smithi131.stdout:1 2024-08-02T23:39:54.104 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:54.101+0000 7f9462599640 1 -- 172.21.15.131:0/1968943012 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f945c1192a0 msgr2=0x7f945c1adaf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-02T23:39:54.105 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:54.101+0000 7f9462599640 1 --2- 172.21.15.131:0/1968943012 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f945c1192a0 0x7f945c1adaf0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:39:54.105 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:54.102+0000 7f9462599640 1 -- 172.21.15.131:0/1968943012 shutdown_connections 2024-08-02T23:39:54.105 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:54.102+0000 7f9462599640 1 --2- 172.21.15.131:0/1968943012 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f945c1192a0 0x7f945c1adaf0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:39:54.105 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:54.102+0000 7f9462599640 1 -- 172.21.15.131:0/1968943012 >> 172.21.15.131:0/1968943012 conn(0x7f945c06e9a0 msgr2=0x7f945c111d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-02T23:39:54.105 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:54.102+0000 7f9462599640 1 -- 172.21.15.131:0/1968943012 shutdown_connections 2024-08-02T23:39:54.105 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:54.102+0000 7f9462599640 1 -- 172.21.15.131:0/1968943012 wait complete. 2024-08-02T23:39:54.115 DEBUG:teuthology.orchestra.run.smithi131:> sudo ceph --cluster ceph osd new 7d855822-09de-49f2-8811-c47826c38d5d 2 2024-08-02T23:39:54.144 INFO:tasks.ceph.mgr.x.smithi131.stderr:2024-08-02T23:39:54.141+0000 7ff194b6df00 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-08-02T23:39:54.200 INFO:tasks.ceph.mgr.x.smithi131.stderr:2024-08-02T23:39:54.197+0000 7ff194b6df00 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-08-02T23:39:54.219 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:54.210+0000 7f9c5126d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-02T23:39:54.219 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:54.216+0000 7f9c5126d640 1 Processor -- start 2024-08-02T23:39:54.219 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:54.216+0000 7f9c5126d640 1 -- start start 2024-08-02T23:39:54.221 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:54.216+0000 7f9c5126d640 1 --2- >> v2:172.21.15.73:3300/0 conn(0x7f9c4c11bfc0 0x7f9c4c11e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-02T23:39:54.221 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:54.217+0000 7f9c5126d640 1 -- --> v1:172.21.15.73:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9c4c11e9b0 con 0x7f9c4c1192e0 2024-08-02T23:39:54.221 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:54.217+0000 7f9c5126d640 1 -- --> v2:172.21.15.73:3300/0 -- mon_getmap magic: 0 -- 0x7f9c4c11fb70 con 0x7f9c4c11bfc0 2024-08-02T23:39:54.222 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:54.217+0000 7f9c4a7fc640 1 --2- >> v2:172.21.15.73:3300/0 conn(0x7f9c4c11bfc0 0x7f9c4c11e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-02T23:39:54.222 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:54.217+0000 7f9c4affd640 1 --1- >> v1:172.21.15.73:6789/0 conn(0x7f9c4c1192e0 0x7f9c4c1196f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.73:6789/0 says I am v1:172.21.15.131:55908/0 (socket says 172.21.15.131:55908) 2024-08-02T23:39:54.222 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:54.217+0000 7f9c4a7fc640 1 --2- >> v2:172.21.15.73:3300/0 conn(0x7f9c4c11bfc0 0x7f9c4c11e3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.131:60526/0 (socket says 172.21.15.131:60526) 2024-08-02T23:39:54.222 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:54.217+0000 7f9c4affd640 1 -- 172.21.15.131:0/3666601536 learned_addr learned my addr 172.21.15.131:0/3666601536 (peer_addr_for_me v1:172.21.15.131:0/0) 2024-08-02T23:39:54.222 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:54.217+0000 7f9c4a7fc640 1 -- 172.21.15.131:0/3666601536 >> v1:172.21.15.73:6789/0 conn(0x7f9c4c1192e0 legacy=0x7f9c4c1196f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-02T23:39:54.222 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:54.217+0000 7f9c49ffb640 1 -- 172.21.15.131:0/3666601536 <== mon.0 v1:172.21.15.73:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3736411955 0 0) 0x7f9c4c11e9b0 con 0x7f9c4c1192e0 2024-08-02T23:39:54.222 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:54.217+0000 7f9c4a7fc640 1 -- 172.21.15.131:0/3666601536 --> v2:172.21.15.73:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c4c11fce0 con 0x7f9c4c11bfc0 2024-08-02T23:39:54.222 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:54.217+0000 7f9c4a7fc640 1 --2- 172.21.15.131:0/3666601536 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9c4c11bfc0 0x7f9c4c11e3e0 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ea27c5480999c30 server_cookie=0 in_seq=0 out_seq=0 2024-08-02T23:39:54.222 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:54.218+0000 7f9c49ffb640 1 -- 172.21.15.131:0/3666601536 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f9c3c009ce0 con 0x7f9c4c11bfc0 2024-08-02T23:39:54.222 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:54.218+0000 7f9c49ffb640 1 -- 172.21.15.131:0/3666601536 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f9c3c009e40 con 0x7f9c4c11bfc0 2024-08-02T23:39:54.223 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:54.218+0000 7f9c49ffb640 1 -- 172.21.15.131:0/3666601536 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f9c3c00e360 con 0x7f9c4c11bfc0 2024-08-02T23:39:54.223 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:54.218+0000 7f9c5126d640 1 -- 172.21.15.131:0/3666601536 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9c4c11bfc0 msgr2=0x7f9c4c11e3e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-02T23:39:54.223 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:54.218+0000 7f9c5126d640 1 --2- 172.21.15.131:0/3666601536 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9c4c11bfc0 0x7f9c4c11e3e0 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:39:54.223 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:54.218+0000 7f9c5126d640 1 -- 172.21.15.131:0/3666601536 shutdown_connections 2024-08-02T23:39:54.223 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:54.218+0000 7f9c5126d640 1 --2- 172.21.15.131:0/3666601536 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9c4c11bfc0 0x7f9c4c11e3e0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:39:54.223 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:54.218+0000 7f9c5126d640 1 -- 172.21.15.131:0/3666601536 >> 172.21.15.131:0/3666601536 conn(0x7f9c4c06eb80 msgr2=0x7f9c4c06ef90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-02T23:39:54.223 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:54.218+0000 7f9c5126d640 1 -- 172.21.15.131:0/3666601536 shutdown_connections 2024-08-02T23:39:54.223 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:54.218+0000 7f9c5126d640 1 -- 172.21.15.131:0/3666601536 wait complete. 2024-08-02T23:39:54.223 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:54.219+0000 7f9c5126d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-02T23:39:54.225 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:54.219+0000 7f9c5126d640 1 Processor -- start 2024-08-02T23:39:54.225 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:54.219+0000 7f9c5126d640 1 -- start start 2024-08-02T23:39:54.226 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:54.220+0000 7f9c5126d640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9c4c1192e0 0x7f9c4c1a7d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-02T23:39:54.226 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:54.220+0000 7f9c5126d640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 -- 0x7f9c4c1a8300 con 0x7f9c4c1192e0 2024-08-02T23:39:54.226 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:54.220+0000 7f9c4affd640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9c4c1192e0 0x7f9c4c1a7d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-02T23:39:54.226 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:54.220+0000 7f9c4affd640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9c4c1192e0 0x7f9c4c1a7d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.131:60540/0 (socket says 172.21.15.131:60540) 2024-08-02T23:39:54.226 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:54.220+0000 7f9c4affd640 1 -- 172.21.15.131:0/1365807966 learned_addr learned my addr 172.21.15.131:0/1365807966 (peer_addr_for_me v2:172.21.15.131:0/0) 2024-08-02T23:39:54.226 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:54.220+0000 7f9c4affd640 1 -- 172.21.15.131:0/1365807966 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c4c1a8440 con 0x7f9c4c1192e0 2024-08-02T23:39:54.226 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:54.220+0000 7f9c4affd640 1 --2- 172.21.15.131:0/1365807966 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9c4c1192e0 0x7f9c4c1a7d30 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-02T23:39:54.226 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:54.221+0000 7f9c437fe640 1 -- 172.21.15.131:0/1365807966 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f9c34004bd0 con 0x7f9c4c1192e0 2024-08-02T23:39:54.226 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:54.221+0000 7f9c437fe640 1 -- 172.21.15.131:0/1365807966 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f9c34004d30 con 0x7f9c4c1192e0 2024-08-02T23:39:54.226 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:54.221+0000 7f9c5126d640 1 -- 172.21.15.131:0/1365807966 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9c4c1a86f0 con 0x7f9c4c1192e0 2024-08-02T23:39:54.226 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:54.221+0000 7f9c437fe640 1 -- 172.21.15.131:0/1365807966 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f9c34007120 con 0x7f9c4c1192e0 2024-08-02T23:39:54.227 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:54.221+0000 7f9c5126d640 1 -- 172.21.15.131:0/1365807966 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9c4c1ac5d0 con 0x7f9c4c1192e0 2024-08-02T23:39:54.227 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:54.221+0000 7f9c437fe640 1 -- 172.21.15.131:0/1365807966 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (crc 0 0 0) 0x7f9c3400c580 con 0x7f9c4c1192e0 2024-08-02T23:39:54.227 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:54.221+0000 7f9c437fe640 1 -- 172.21.15.131:0/1365807966 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 977+0+0 (crc 0 0 0) 0x7f9c3400f230 con 0x7f9c4c1192e0 2024-08-02T23:39:54.227 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:54.222+0000 7f9c5126d640 1 -- 172.21.15.131:0/1365807966 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9c4c1a88e0 con 0x7f9c4c1192e0 2024-08-02T23:39:54.227 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:54.224+0000 7f9c437fe640 1 -- 172.21.15.131:0/1365807966 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73482 (crc 0 0 0) 0x7f9c3400c830 con 0x7f9c4c1192e0 2024-08-02T23:39:54.297 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:54.295+0000 7f9c5126d640 1 -- 172.21.15.131:0/1365807966 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "osd new", "uuid": "7d855822-09de-49f2-8811-c47826c38d5d", "id": 2} v 0) -- 0x7f9c4c11bfc0 con 0x7f9c4c1192e0 2024-08-02T23:39:54.298 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:54.296+0000 7f9c437fe640 1 -- 172.21.15.131:0/1365807966 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "7d855822-09de-49f2-8811-c47826c38d5d", "id": 2}]=0 v6) ==== 112+0+2 (crc 0 0 0) 0x7f9c3400ca10 con 0x7f9c4c1192e0 2024-08-02T23:39:54.299 INFO:teuthology.orchestra.run.smithi131.stdout:2 2024-08-02T23:39:54.300 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:54.297+0000 7f9c5126d640 1 -- 172.21.15.131:0/1365807966 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9c4c1192e0 msgr2=0x7f9c4c1a7d30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-02T23:39:54.300 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:54.297+0000 7f9c5126d640 1 --2- 172.21.15.131:0/1365807966 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9c4c1192e0 0x7f9c4c1a7d30 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:39:54.300 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:54.297+0000 7f9c5126d640 1 -- 172.21.15.131:0/1365807966 shutdown_connections 2024-08-02T23:39:54.300 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:54.297+0000 7f9c5126d640 1 --2- 172.21.15.131:0/1365807966 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9c4c1192e0 0x7f9c4c1a7d30 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:39:54.300 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:54.297+0000 7f9c5126d640 1 -- 172.21.15.131:0/1365807966 >> 172.21.15.131:0/1365807966 conn(0x7f9c4c06eb80 msgr2=0x7f9c4c111d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-02T23:39:54.300 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:54.297+0000 7f9c5126d640 1 -- 172.21.15.131:0/1365807966 shutdown_connections 2024-08-02T23:39:54.300 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:54.297+0000 7f9c5126d640 1 -- 172.21.15.131:0/1365807966 wait complete. 2024-08-02T23:39:54.310 DEBUG:teuthology.orchestra.run.smithi131:> sudo ceph --cluster ceph osd new d6e6bd88-9a90-441c-8bbd-8b8d4a2b4f9f 3 2024-08-02T23:39:54.316 INFO:tasks.ceph.mgr.x.smithi131.stderr:2024-08-02T23:39:54.314+0000 7ff194b6df00 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-08-02T23:39:54.420 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:54.412+0000 7f4d9f020640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-02T23:39:54.421 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:54.418+0000 7f4d9f020640 1 Processor -- start 2024-08-02T23:39:54.421 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:54.418+0000 7f4d9f020640 1 -- start start 2024-08-02T23:39:54.421 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:54.418+0000 7f4d9f020640 1 --2- >> v2:172.21.15.73:3300/0 conn(0x7f4d9811bfc0 0x7f4d9811e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-02T23:39:54.421 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:54.418+0000 7f4d9f020640 1 -- --> v1:172.21.15.73:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4d9811e9b0 con 0x7f4d981192e0 2024-08-02T23:39:54.421 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:54.418+0000 7f4d9f020640 1 -- --> v2:172.21.15.73:3300/0 -- mon_getmap magic: 0 -- 0x7f4d9811fb70 con 0x7f4d9811bfc0 2024-08-02T23:39:54.423 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:54.418+0000 7f4d97fff640 1 --2- >> v2:172.21.15.73:3300/0 conn(0x7f4d9811bfc0 0x7f4d9811e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-02T23:39:54.423 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:54.419+0000 7f4d97fff640 1 --2- >> v2:172.21.15.73:3300/0 conn(0x7f4d9811bfc0 0x7f4d9811e3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.131:60554/0 (socket says 172.21.15.131:60554) 2024-08-02T23:39:54.424 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:54.419+0000 7f4d9cdbc640 1 --1- >> v1:172.21.15.73:6789/0 conn(0x7f4d981192e0 0x7f4d981196f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.73:6789/0 says I am v1:172.21.15.131:55914/0 (socket says 172.21.15.131:55914) 2024-08-02T23:39:54.424 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:54.419+0000 7f4d97fff640 1 -- 172.21.15.131:0/974886472 learned_addr learned my addr 172.21.15.131:0/974886472 (peer_addr_for_me v2:172.21.15.131:0/0) 2024-08-02T23:39:54.424 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:54.419+0000 7f4d97fff640 1 -- 172.21.15.131:0/974886472 >> v1:172.21.15.73:6789/0 conn(0x7f4d981192e0 legacy=0x7f4d981196f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-02T23:39:54.424 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:54.419+0000 7f4d97fff640 1 -- 172.21.15.131:0/974886472 --> v2:172.21.15.73:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d9811fce0 con 0x7f4d9811bfc0 2024-08-02T23:39:54.424 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:54.419+0000 7f4d97fff640 1 --2- 172.21.15.131:0/974886472 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f4d9811bfc0 0x7f4d9811e3e0 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=675545cd015fc67f server_cookie=0 in_seq=0 out_seq=0 2024-08-02T23:39:54.424 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:54.419+0000 7f4d977fe640 1 -- 172.21.15.131:0/974886472 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f4d84009cc0 con 0x7f4d9811bfc0 2024-08-02T23:39:54.424 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:54.419+0000 7f4d977fe640 1 -- 172.21.15.131:0/974886472 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f4d84009e20 con 0x7f4d9811bfc0 2024-08-02T23:39:54.424 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:54.419+0000 7f4d977fe640 1 -- 172.21.15.131:0/974886472 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f4d8400e360 con 0x7f4d9811bfc0 2024-08-02T23:39:54.424 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:54.420+0000 7f4d9f020640 1 -- 172.21.15.131:0/974886472 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f4d9811bfc0 msgr2=0x7f4d9811e3e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-02T23:39:54.424 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:54.420+0000 7f4d9f020640 1 --2- 172.21.15.131:0/974886472 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f4d9811bfc0 0x7f4d9811e3e0 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:39:54.424 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:54.420+0000 7f4d9f020640 1 -- 172.21.15.131:0/974886472 shutdown_connections 2024-08-02T23:39:54.425 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:54.420+0000 7f4d9f020640 1 --2- 172.21.15.131:0/974886472 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f4d9811bfc0 0x7f4d9811e3e0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:39:54.425 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:54.420+0000 7f4d9f020640 1 -- 172.21.15.131:0/974886472 >> 172.21.15.131:0/974886472 conn(0x7f4d9806eb80 msgr2=0x7f4d9806ef90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-02T23:39:54.425 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:54.420+0000 7f4d9f020640 1 -- 172.21.15.131:0/974886472 shutdown_connections 2024-08-02T23:39:54.425 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:54.420+0000 7f4d9f020640 1 -- 172.21.15.131:0/974886472 wait complete. 2024-08-02T23:39:54.425 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:54.420+0000 7f4d9f020640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-02T23:39:54.425 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:54.421+0000 7f4d9f020640 1 Processor -- start 2024-08-02T23:39:54.425 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:54.421+0000 7f4d9f020640 1 -- start start 2024-08-02T23:39:54.427 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:54.421+0000 7f4d9f020640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f4d981192e0 0x7f4d981ada00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-02T23:39:54.427 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:54.421+0000 7f4d9f020640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 -- 0x7f4d981adfd0 con 0x7f4d981192e0 2024-08-02T23:39:54.427 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:54.421+0000 7f4d9cdbc640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f4d981192e0 0x7f4d981ada00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-02T23:39:54.427 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:54.421+0000 7f4d9cdbc640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f4d981192e0 0x7f4d981ada00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.131:60558/0 (socket says 172.21.15.131:60558) 2024-08-02T23:39:54.427 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:54.421+0000 7f4d9cdbc640 1 -- 172.21.15.131:0/2198022355 learned_addr learned my addr 172.21.15.131:0/2198022355 (peer_addr_for_me v2:172.21.15.131:0/0) 2024-08-02T23:39:54.427 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:54.422+0000 7f4d9cdbc640 1 -- 172.21.15.131:0/2198022355 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d981ae110 con 0x7f4d981192e0 2024-08-02T23:39:54.427 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:54.422+0000 7f4d9cdbc640 1 --2- 172.21.15.131:0/2198022355 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f4d981192e0 0x7f4d981ada00 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-02T23:39:54.427 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:54.422+0000 7f4d957fa640 1 -- 172.21.15.131:0/2198022355 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f4d8c00caf0 con 0x7f4d981192e0 2024-08-02T23:39:54.427 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:54.422+0000 7f4d957fa640 1 -- 172.21.15.131:0/2198022355 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f4d8c00cc50 con 0x7f4d981192e0 2024-08-02T23:39:54.428 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:54.422+0000 7f4d9f020640 1 -- 172.21.15.131:0/2198022355 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d981ae360 con 0x7f4d981192e0 2024-08-02T23:39:54.428 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:54.422+0000 7f4d957fa640 1 -- 172.21.15.131:0/2198022355 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f4d8c007120 con 0x7f4d981192e0 2024-08-02T23:39:54.428 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:54.422+0000 7f4d9f020640 1 -- 172.21.15.131:0/2198022355 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d981a7de0 con 0x7f4d981192e0 2024-08-02T23:39:54.428 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:54.422+0000 7f4d957fa640 1 -- 172.21.15.131:0/2198022355 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (crc 0 0 0) 0x7f4d8c0047b0 con 0x7f4d981192e0 2024-08-02T23:39:54.428 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:54.422+0000 7f4d957fa640 1 -- 172.21.15.131:0/2198022355 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(6..6 src has 1..6) ==== 1092+0+0 (crc 0 0 0) 0x7f4d8c00f230 con 0x7f4d981192e0 2024-08-02T23:39:54.428 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:54.423+0000 7f4d9f020640 1 -- 172.21.15.131:0/2198022355 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4d981ae550 con 0x7f4d981192e0 2024-08-02T23:39:54.428 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:54.425+0000 7f4d957fa640 1 -- 172.21.15.131:0/2198022355 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73482 (crc 0 0 0) 0x7f4d8c00f3f0 con 0x7f4d981192e0 2024-08-02T23:39:54.500 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:54.497+0000 7f4d9f020640 1 -- 172.21.15.131:0/2198022355 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "osd new", "uuid": "d6e6bd88-9a90-441c-8bbd-8b8d4a2b4f9f", "id": 3} v 0) -- 0x7f4d9811c060 con 0x7f4d981192e0 2024-08-02T23:39:54.502 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:54.499+0000 7f4d957fa640 1 -- 172.21.15.131:0/2198022355 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "d6e6bd88-9a90-441c-8bbd-8b8d4a2b4f9f", "id": 3}]=0 v7) ==== 112+0+2 (crc 0 0 0) 0x7f4d8c004a60 con 0x7f4d981192e0 2024-08-02T23:39:54.502 INFO:teuthology.orchestra.run.smithi131.stdout:3 2024-08-02T23:39:54.503 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:54.500+0000 7f4d9f020640 1 -- 172.21.15.131:0/2198022355 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f4d981192e0 msgr2=0x7f4d981ada00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-02T23:39:54.503 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:54.500+0000 7f4d9f020640 1 --2- 172.21.15.131:0/2198022355 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f4d981192e0 0x7f4d981ada00 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:39:54.503 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:54.500+0000 7f4d9f020640 1 -- 172.21.15.131:0/2198022355 shutdown_connections 2024-08-02T23:39:54.504 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:54.500+0000 7f4d9f020640 1 --2- 172.21.15.131:0/2198022355 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f4d981192e0 0x7f4d981ada00 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:39:54.504 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:54.500+0000 7f4d9f020640 1 -- 172.21.15.131:0/2198022355 >> 172.21.15.131:0/2198022355 conn(0x7f4d9806eb80 msgr2=0x7f4d98111d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-02T23:39:54.504 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:54.500+0000 7f4d9f020640 1 -- 172.21.15.131:0/2198022355 shutdown_connections 2024-08-02T23:39:54.504 INFO:teuthology.orchestra.run.smithi131.stderr:2024-08-02T23:39:54.500+0000 7f4d9f020640 1 -- 172.21.15.131:0/2198022355 wait complete. 2024-08-02T23:39:54.514 DEBUG:teuthology.orchestra.run.smithi073:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.0.log 2024-08-02T23:39:54.549 INFO:tasks.ceph.osd.0:Restarting daemon 2024-08-02T23:39:54.550 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2>> /var/log/ceph/ceph-osd.0.log 2024-08-02T23:39:54.592 INFO:tasks.ceph.osd.0:Started 2024-08-02T23:39:54.593 DEBUG:teuthology.orchestra.run.smithi073:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.1.log 2024-08-02T23:39:54.627 INFO:tasks.ceph.osd.1:Restarting daemon 2024-08-02T23:39:54.627 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2>> /var/log/ceph/ceph-osd.1.log 2024-08-02T23:39:54.670 INFO:tasks.ceph.osd.1:Started 2024-08-02T23:39:54.670 DEBUG:teuthology.orchestra.run.smithi131:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.2.log 2024-08-02T23:39:54.701 INFO:tasks.ceph.osd.2:Restarting daemon 2024-08-02T23:39:54.701 DEBUG:teuthology.orchestra.run.smithi131:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2>> /var/log/ceph/ceph-osd.2.log 2024-08-02T23:39:54.709 INFO:tasks.ceph.mgr.x.smithi131.stderr:2024-08-02T23:39:54.706+0000 7ff194b6df00 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-08-02T23:39:54.710 INFO:tasks.ceph.osd.2:Started 2024-08-02T23:39:54.711 DEBUG:teuthology.orchestra.run.smithi131:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.3.log 2024-08-02T23:39:54.739 INFO:tasks.ceph.osd.3:Restarting daemon 2024-08-02T23:39:54.796 DEBUG:teuthology.orchestra.run.smithi131:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2>> /var/log/ceph/ceph-osd.3.log 2024-08-02T23:39:54.799 INFO:tasks.ceph.osd.3:Started 2024-08-02T23:39:54.799 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-08-02T23:39:54.921 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:39:54.915+0000 7f741b599640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-02T23:39:54.922 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:39:54.920+0000 7f741b599640 1 Processor -- start 2024-08-02T23:39:54.922 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:39:54.920+0000 7f741b599640 1 -- start start 2024-08-02T23:39:54.924 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:39:54.921+0000 7f741b599640 1 --2- >> v2:172.21.15.73:3300/0 conn(0x7f74140a4770 0x7f74140a4b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-02T23:39:54.924 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:39:54.921+0000 7f741b599640 1 -- --> v1:172.21.15.73:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f741406e030 con 0x7f74140a50c0 2024-08-02T23:39:54.924 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:39:54.921+0000 7f741b599640 1 -- --> v2:172.21.15.73:3300/0 -- mon_getmap magic: 0 -- 0x7f74140a0ed0 con 0x7f74140a4770 2024-08-02T23:39:54.924 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:39:54.921+0000 7f7419335640 1 --2- >> v2:172.21.15.73:3300/0 conn(0x7f74140a4770 0x7f74140a4b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-02T23:39:54.924 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:39:54.921+0000 7f7418b34640 1 --1- >> v1:172.21.15.73:6789/0 conn(0x7f74140a50c0 0x7f7414131000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.73:6789/0 says I am v1:172.21.15.73:44792/0 (socket says 172.21.15.73:44792) 2024-08-02T23:39:54.924 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:39:54.921+0000 7f7418b34640 1 -- 172.21.15.73:0/107948568 learned_addr learned my addr 172.21.15.73:0/107948568 (peer_addr_for_me v1:172.21.15.73:0/0) 2024-08-02T23:39:54.924 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:39:54.922+0000 7f7419335640 1 -- 172.21.15.73:0/107948568 >> v1:172.21.15.73:6789/0 conn(0x7f74140a50c0 legacy=0x7f7414131000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-02T23:39:54.925 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:39:54.922+0000 7f7419335640 1 -- 172.21.15.73:0/107948568 --> v2:172.21.15.73:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f741406e260 con 0x7f74140a4770 2024-08-02T23:39:54.925 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:39:54.922+0000 7f7419335640 1 --2- 172.21.15.73:0/107948568 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f74140a4770 0x7f74140a4b80 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=820d76807b8a5397 server_cookie=0 in_seq=0 out_seq=0 2024-08-02T23:39:54.925 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:39:54.922+0000 7f740bfff640 1 -- 172.21.15.73:0/107948568 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f7410009ce0 con 0x7f74140a4770 2024-08-02T23:39:54.925 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:39:54.922+0000 7f740bfff640 1 -- 172.21.15.73:0/107948568 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f7410009e40 con 0x7f74140a4770 2024-08-02T23:39:54.925 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:39:54.922+0000 7f740bfff640 1 -- 172.21.15.73:0/107948568 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f741000e360 con 0x7f74140a4770 2024-08-02T23:39:54.926 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:39:54.923+0000 7f741b599640 1 -- 172.21.15.73:0/107948568 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f74140a4770 msgr2=0x7f74140a4b80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-02T23:39:54.926 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:39:54.923+0000 7f741b599640 1 --2- 172.21.15.73:0/107948568 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f74140a4770 0x7f74140a4b80 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:39:54.926 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:39:54.923+0000 7f741b599640 1 -- 172.21.15.73:0/107948568 shutdown_connections 2024-08-02T23:39:54.926 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:39:54.923+0000 7f741b599640 1 --2- 172.21.15.73:0/107948568 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f74140a4770 0x7f74140a4b80 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:39:54.926 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:39:54.923+0000 7f741b599640 1 -- 172.21.15.73:0/107948568 >> 172.21.15.73:0/107948568 conn(0x7f74140a3510 msgr2=0x7f74140a3920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-02T23:39:54.927 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:39:54.923+0000 7f741b599640 1 -- 172.21.15.73:0/107948568 shutdown_connections 2024-08-02T23:39:54.927 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:39:54.923+0000 7f741b599640 1 -- 172.21.15.73:0/107948568 wait complete. 2024-08-02T23:39:54.927 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:39:54.923+0000 7f741b599640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-02T23:39:54.927 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:39:54.924+0000 7f741b599640 1 Processor -- start 2024-08-02T23:39:54.927 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:39:54.924+0000 7f741b599640 1 -- start start 2024-08-02T23:39:54.929 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:39:54.925+0000 7f741b599640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f74140a4770 0x7f741411db50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-02T23:39:54.929 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:39:54.925+0000 7f741b599640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 -- 0x7f7414133660 con 0x7f74140a4770 2024-08-02T23:39:54.929 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:39:54.925+0000 7f7419335640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f74140a4770 0x7f741411db50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-02T23:39:54.929 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:39:54.925+0000 7f7419335640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f74140a4770 0x7f741411db50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:51232/0 (socket says 172.21.15.73:51232) 2024-08-02T23:39:54.929 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:39:54.925+0000 7f7419335640 1 -- 172.21.15.73:0/1463001728 learned_addr learned my addr 172.21.15.73:0/1463001728 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-08-02T23:39:54.929 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:39:54.926+0000 7f7419335640 1 -- 172.21.15.73:0/1463001728 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7414120630 con 0x7f74140a4770 2024-08-02T23:39:54.929 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:39:54.926+0000 7f7419335640 1 --2- 172.21.15.73:0/1463001728 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f74140a4770 0x7f741411db50 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-02T23:39:54.929 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:39:54.926+0000 7f7409ffb640 1 -- 172.21.15.73:0/1463001728 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f7410014620 con 0x7f74140a4770 2024-08-02T23:39:54.929 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:39:54.926+0000 7f7409ffb640 1 -- 172.21.15.73:0/1463001728 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f7410014780 con 0x7f74140a4770 2024-08-02T23:39:54.929 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:39:54.926+0000 7f7409ffb640 1 -- 172.21.15.73:0/1463001728 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f7410014a20 con 0x7f74140a4770 2024-08-02T23:39:54.930 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:39:54.926+0000 7f741b599640 1 -- 172.21.15.73:0/1463001728 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7414121050 con 0x7f74140a4770 2024-08-02T23:39:54.930 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:39:54.926+0000 7f7409ffb640 1 -- 172.21.15.73:0/1463001728 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (crc 0 0 0) 0x7f7410011380 con 0x7f74140a4770 2024-08-02T23:39:54.930 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:39:54.926+0000 7f741b599640 1 -- 172.21.15.73:0/1463001728 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f741411ffb0 con 0x7f74140a4770 2024-08-02T23:39:54.930 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:39:54.927+0000 7f7409ffb640 1 -- 172.21.15.73:0/1463001728 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(7..7 src has 1..7) ==== 1207+0+0 (crc 0 0 0) 0x7f7410010310 con 0x7f74140a4770 2024-08-02T23:39:54.930 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:39:54.927+0000 7f741b599640 1 -- 172.21.15.73:0/1463001728 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f741409aba0 con 0x7f74140a4770 2024-08-02T23:39:54.931 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:39:54.930+0000 7f7409ffb640 1 -- 172.21.15.73:0/1463001728 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73482 (crc 0 0 0) 0x7f74100118b0 con 0x7f74140a4770 2024-08-02T23:39:54.995 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:39:54.993+0000 7f741b599640 1 -- 172.21.15.73:0/1463001728 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7414073220 con 0x7f74140a4770 2024-08-02T23:39:54.995 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:39:54.993+0000 7f7409ffb640 1 -- 172.21.15.73:0/1463001728 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v7) ==== 74+0+4152 (crc 0 0 0) 0x7f7410011b00 con 0x7f74140a4770 2024-08-02T23:39:54.996 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-08-02T23:39:54.996 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":7,"fsid":"7caeb70d-fd31-4bdf-94e1-d0f19ea8e3c6","created":"2024-08-02T23:39:52.389003+0000","modified":"2024-08-02T23:39:54.500028+0000","last_up_change":"0.000000","last_in_change":"2024-08-02T23:39:54.500028+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"bd37ae5d-55cc-4a6d-b943-8de7e40fd1bb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6bf29a4c-3240-4fce-8803-66926d0df50c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7d855822-09de-49f2-8811-c47826c38d5d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d6e6bd88-9a90-441c-8bbd-8b8d4a2b4f9f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-02T23:39:54.997 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:39:54.994+0000 7f741b599640 1 -- 172.21.15.73:0/1463001728 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f74140a4770 msgr2=0x7f741411db50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-02T23:39:54.997 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:39:54.994+0000 7f741b599640 1 --2- 172.21.15.73:0/1463001728 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f74140a4770 0x7f741411db50 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:39:54.997 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:39:54.995+0000 7f741b599640 1 -- 172.21.15.73:0/1463001728 shutdown_connections 2024-08-02T23:39:54.997 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:39:54.995+0000 7f741b599640 1 --2- 172.21.15.73:0/1463001728 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f74140a4770 0x7f741411db50 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:39:54.997 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:39:54.995+0000 7f741b599640 1 -- 172.21.15.73:0/1463001728 >> 172.21.15.73:0/1463001728 conn(0x7f74140a3510 msgr2=0x7f74140a12e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-02T23:39:54.997 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:39:54.995+0000 7f741b599640 1 -- 172.21.15.73:0/1463001728 shutdown_connections 2024-08-02T23:39:54.997 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:39:54.995+0000 7f741b599640 1 -- 172.21.15.73:0/1463001728 wait complete. 2024-08-02T23:39:55.009 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-08-02T23:39:55.009 INFO:tasks.ceph:Waiting for OSDs to come up 2024-08-02T23:39:55.010 INFO:tasks.ceph.mgr.x.smithi131.stderr:2024-08-02T23:39:55.006+0000 7ff194b6df00 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-08-02T23:39:55.072 INFO:tasks.ceph.mgr.x.smithi131.stderr:2024-08-02T23:39:55.069+0000 7ff194b6df00 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-08-02T23:39:55.207 INFO:tasks.ceph.mgr.x.smithi131.stderr:2024-08-02T23:39:55.205+0000 7ff194b6df00 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-08-02T23:39:55.412 DEBUG:teuthology.orchestra.run.smithi073:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-08-02T23:39:55.513 INFO:teuthology.misc.health.smithi073.stderr:2024-08-02T23:39:55.508+0000 7fb99066d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-02T23:39:55.514 INFO:teuthology.misc.health.smithi073.stderr:2024-08-02T23:39:55.512+0000 7fb99066d640 1 Processor -- start 2024-08-02T23:39:55.514 INFO:teuthology.misc.health.smithi073.stderr:2024-08-02T23:39:55.512+0000 7fb99066d640 1 -- start start 2024-08-02T23:39:55.516 INFO:teuthology.misc.health.smithi073.stderr:2024-08-02T23:39:55.513+0000 7fb99066d640 1 --2- >> v2:172.21.15.73:3300/0 conn(0x7fb9880a2d00 0x7fb988139820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-02T23:39:55.516 INFO:teuthology.misc.health.smithi073.stderr:2024-08-02T23:39:55.513+0000 7fb99066d640 1 -- --> v1:172.21.15.73:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb9880a0c80 con 0x7fb9880a21e0 2024-08-02T23:39:55.516 INFO:teuthology.misc.health.smithi073.stderr:2024-08-02T23:39:55.513+0000 7fb99066d640 1 -- --> v2:172.21.15.73:3300/0 -- mon_getmap magic: 0 -- 0x7fb9880a0eb0 con 0x7fb9880a2d00 2024-08-02T23:39:55.516 INFO:teuthology.misc.health.smithi073.stderr:2024-08-02T23:39:55.513+0000 7fb98dc08640 1 --2- >> v2:172.21.15.73:3300/0 conn(0x7fb9880a2d00 0x7fb988139820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-02T23:39:55.516 INFO:teuthology.misc.health.smithi073.stderr:2024-08-02T23:39:55.513+0000 7fb98e409640 1 --1- >> v1:172.21.15.73:6789/0 conn(0x7fb9880a21e0 0x7fb9880a25f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.73:6789/0 says I am v1:172.21.15.73:44802/0 (socket says 172.21.15.73:44802) 2024-08-02T23:39:55.516 INFO:teuthology.misc.health.smithi073.stderr:2024-08-02T23:39:55.513+0000 7fb98e409640 1 -- 172.21.15.73:0/3114066042 learned_addr learned my addr 172.21.15.73:0/3114066042 (peer_addr_for_me v1:172.21.15.73:0/0) 2024-08-02T23:39:55.516 INFO:teuthology.misc.health.smithi073.stderr:2024-08-02T23:39:55.514+0000 7fb98dc08640 1 -- 172.21.15.73:0/3114066042 >> v1:172.21.15.73:6789/0 conn(0x7fb9880a21e0 legacy=0x7fb9880a25f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-02T23:39:55.516 INFO:teuthology.misc.health.smithi073.stderr:2024-08-02T23:39:55.514+0000 7fb98dc08640 1 -- 172.21.15.73:0/3114066042 --> v2:172.21.15.73:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb98806e0e0 con 0x7fb9880a2d00 2024-08-02T23:39:55.516 INFO:teuthology.misc.health.smithi073.stderr:2024-08-02T23:39:55.514+0000 7fb98dc08640 1 --2- 172.21.15.73:0/3114066042 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb9880a2d00 0x7fb988139820 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86982e8a47d0677e server_cookie=0 in_seq=0 out_seq=0 2024-08-02T23:39:55.516 INFO:teuthology.misc.health.smithi073.stderr:2024-08-02T23:39:55.514+0000 7fb98d407640 1 -- 172.21.15.73:0/3114066042 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fb978009ce0 con 0x7fb9880a2d00 2024-08-02T23:39:55.516 INFO:teuthology.misc.health.smithi073.stderr:2024-08-02T23:39:55.514+0000 7fb98d407640 1 -- 172.21.15.73:0/3114066042 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7fb978009e40 con 0x7fb9880a2d00 2024-08-02T23:39:55.517 INFO:teuthology.misc.health.smithi073.stderr:2024-08-02T23:39:55.514+0000 7fb98d407640 1 -- 172.21.15.73:0/3114066042 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fb97800e360 con 0x7fb9880a2d00 2024-08-02T23:39:55.518 INFO:teuthology.misc.health.smithi073.stderr:2024-08-02T23:39:55.515+0000 7fb99066d640 1 -- 172.21.15.73:0/3114066042 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb9880a2d00 msgr2=0x7fb988139820 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-02T23:39:55.518 INFO:teuthology.misc.health.smithi073.stderr:2024-08-02T23:39:55.515+0000 7fb99066d640 1 --2- 172.21.15.73:0/3114066042 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb9880a2d00 0x7fb988139820 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:39:55.518 INFO:teuthology.misc.health.smithi073.stderr:2024-08-02T23:39:55.515+0000 7fb99066d640 1 -- 172.21.15.73:0/3114066042 shutdown_connections 2024-08-02T23:39:55.518 INFO:teuthology.misc.health.smithi073.stderr:2024-08-02T23:39:55.515+0000 7fb99066d640 1 --2- 172.21.15.73:0/3114066042 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb9880a2d00 0x7fb988139820 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:39:55.518 INFO:teuthology.misc.health.smithi073.stderr:2024-08-02T23:39:55.515+0000 7fb99066d640 1 -- 172.21.15.73:0/3114066042 >> 172.21.15.73:0/3114066042 conn(0x7fb98806cea0 msgr2=0x7fb98806d2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-02T23:39:55.518 INFO:teuthology.misc.health.smithi073.stderr:2024-08-02T23:39:55.515+0000 7fb99066d640 1 -- 172.21.15.73:0/3114066042 shutdown_connections 2024-08-02T23:39:55.518 INFO:teuthology.misc.health.smithi073.stderr:2024-08-02T23:39:55.515+0000 7fb99066d640 1 -- 172.21.15.73:0/3114066042 wait complete. 2024-08-02T23:39:55.518 INFO:teuthology.misc.health.smithi073.stderr:2024-08-02T23:39:55.515+0000 7fb99066d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-02T23:39:55.518 INFO:teuthology.misc.health.smithi073.stderr:2024-08-02T23:39:55.516+0000 7fb99066d640 1 Processor -- start 2024-08-02T23:39:55.518 INFO:teuthology.misc.health.smithi073.stderr:2024-08-02T23:39:55.516+0000 7fb99066d640 1 -- start start 2024-08-02T23:39:55.520 INFO:teuthology.misc.health.smithi073.stderr:2024-08-02T23:39:55.517+0000 7fb99066d640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb9880a21e0 0x7fb9881c91a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-02T23:39:55.520 INFO:teuthology.misc.health.smithi073.stderr:2024-08-02T23:39:55.517+0000 7fb99066d640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 -- 0x7fb9881c8f10 con 0x7fb9880a21e0 2024-08-02T23:39:55.520 INFO:teuthology.misc.health.smithi073.stderr:2024-08-02T23:39:55.517+0000 7fb98e409640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb9880a21e0 0x7fb9881c91a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-02T23:39:55.520 INFO:teuthology.misc.health.smithi073.stderr:2024-08-02T23:39:55.517+0000 7fb98e409640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb9880a21e0 0x7fb9881c91a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:51250/0 (socket says 172.21.15.73:51250) 2024-08-02T23:39:55.520 INFO:teuthology.misc.health.smithi073.stderr:2024-08-02T23:39:55.517+0000 7fb98e409640 1 -- 172.21.15.73:0/3153343389 learned_addr learned my addr 172.21.15.73:0/3153343389 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-08-02T23:39:55.521 INFO:teuthology.misc.health.smithi073.stderr:2024-08-02T23:39:55.517+0000 7fb98e409640 1 -- 172.21.15.73:0/3153343389 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb988133ea0 con 0x7fb9880a21e0 2024-08-02T23:39:55.521 INFO:teuthology.misc.health.smithi073.stderr:2024-08-02T23:39:55.517+0000 7fb98e409640 1 --2- 172.21.15.73:0/3153343389 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb9880a21e0 0x7fb9881c91a0 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-02T23:39:55.521 INFO:teuthology.misc.health.smithi073.stderr:2024-08-02T23:39:55.517+0000 7fb97effd640 1 -- 172.21.15.73:0/3153343389 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fb9840059d0 con 0x7fb9880a21e0 2024-08-02T23:39:55.521 INFO:teuthology.misc.health.smithi073.stderr:2024-08-02T23:39:55.517+0000 7fb97effd640 1 -- 172.21.15.73:0/3153343389 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7fb984005b30 con 0x7fb9880a21e0 2024-08-02T23:39:55.521 INFO:teuthology.misc.health.smithi073.stderr:2024-08-02T23:39:55.517+0000 7fb99066d640 1 -- 172.21.15.73:0/3153343389 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb9880a04c0 con 0x7fb9880a21e0 2024-08-02T23:39:55.521 INFO:teuthology.misc.health.smithi073.stderr:2024-08-02T23:39:55.517+0000 7fb97effd640 1 -- 172.21.15.73:0/3153343389 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fb984005dd0 con 0x7fb9880a21e0 2024-08-02T23:39:55.521 INFO:teuthology.misc.health.smithi073.stderr:2024-08-02T23:39:55.517+0000 7fb99066d640 1 -- 172.21.15.73:0/3153343389 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9880a0760 con 0x7fb9880a21e0 2024-08-02T23:39:55.521 INFO:teuthology.misc.health.smithi073.stderr:2024-08-02T23:39:55.518+0000 7fb97effd640 1 -- 172.21.15.73:0/3153343389 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (crc 0 0 0) 0x7fb984007440 con 0x7fb9880a21e0 2024-08-02T23:39:55.521 INFO:teuthology.misc.health.smithi073.stderr:2024-08-02T23:39:55.518+0000 7fb97effd640 1 -- 172.21.15.73:0/3153343389 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(7..7 src has 1..7) ==== 1207+0+0 (crc 0 0 0) 0x7fb98400fb10 con 0x7fb9880a21e0 2024-08-02T23:39:55.521 INFO:teuthology.misc.health.smithi073.stderr:2024-08-02T23:39:55.518+0000 7fb99066d640 1 -- 172.21.15.73:0/3153343389 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb988069280 con 0x7fb9880a21e0 2024-08-02T23:39:55.522 INFO:teuthology.misc.health.smithi073.stderr:2024-08-02T23:39:55.520+0000 7fb97effd640 1 -- 172.21.15.73:0/3153343389 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73482 (crc 0 0 0) 0x7fb98400fd70 con 0x7fb9880a21e0 2024-08-02T23:39:55.582 INFO:teuthology.misc.health.smithi073.stderr:2024-08-02T23:39:55.580+0000 7fb99066d640 1 -- 172.21.15.73:0/3153343389 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb9880a0950 con 0x7fb9880a21e0 2024-08-02T23:39:55.582 INFO:teuthology.misc.health.smithi073.stderr:2024-08-02T23:39:55.581+0000 7fb97effd640 1 -- 172.21.15.73:0/3153343389 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v7) ==== 74+0+4152 (crc 0 0 0) 0x7fb98400f5f0 con 0x7fb9880a21e0 2024-08-02T23:39:55.583 INFO:teuthology.misc.health.smithi073.stdout: 2024-08-02T23:39:55.583 INFO:teuthology.misc.health.smithi073.stdout:{"epoch":7,"fsid":"7caeb70d-fd31-4bdf-94e1-d0f19ea8e3c6","created":"2024-08-02T23:39:52.389003+0000","modified":"2024-08-02T23:39:54.500028+0000","last_up_change":"0.000000","last_in_change":"2024-08-02T23:39:54.500028+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"bd37ae5d-55cc-4a6d-b943-8de7e40fd1bb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6bf29a4c-3240-4fce-8803-66926d0df50c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7d855822-09de-49f2-8811-c47826c38d5d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d6e6bd88-9a90-441c-8bbd-8b8d4a2b4f9f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-02T23:39:55.584 INFO:teuthology.misc.health.smithi073.stderr:2024-08-02T23:39:55.582+0000 7fb99066d640 1 -- 172.21.15.73:0/3153343389 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb9880a21e0 msgr2=0x7fb9881c91a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-02T23:39:55.584 INFO:teuthology.misc.health.smithi073.stderr:2024-08-02T23:39:55.582+0000 7fb99066d640 1 --2- 172.21.15.73:0/3153343389 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb9880a21e0 0x7fb9881c91a0 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:39:55.584 INFO:teuthology.misc.health.smithi073.stderr:2024-08-02T23:39:55.582+0000 7fb99066d640 1 -- 172.21.15.73:0/3153343389 shutdown_connections 2024-08-02T23:39:55.584 INFO:teuthology.misc.health.smithi073.stderr:2024-08-02T23:39:55.582+0000 7fb99066d640 1 --2- 172.21.15.73:0/3153343389 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb9880a21e0 0x7fb9881c91a0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:39:55.584 INFO:teuthology.misc.health.smithi073.stderr:2024-08-02T23:39:55.582+0000 7fb99066d640 1 -- 172.21.15.73:0/3153343389 >> 172.21.15.73:0/3153343389 conn(0x7fb98806cea0 msgr2=0x7fb988132180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-02T23:39:55.584 INFO:teuthology.misc.health.smithi073.stderr:2024-08-02T23:39:55.582+0000 7fb99066d640 1 -- 172.21.15.73:0/3153343389 shutdown_connections 2024-08-02T23:39:55.584 INFO:teuthology.misc.health.smithi073.stderr:2024-08-02T23:39:55.582+0000 7fb99066d640 1 -- 172.21.15.73:0/3153343389 wait complete. 2024-08-02T23:39:55.593 DEBUG:teuthology.misc:0 of 4 OSDs are up 2024-08-02T23:39:55.652 INFO:tasks.ceph.mgr.x.smithi131.stderr:2024-08-02T23:39:55.649+0000 7ff194b6df00 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-08-02T23:39:55.731 INFO:tasks.ceph.mgr.x.smithi131.stderr:2024-08-02T23:39:55.729+0000 7ff194b6df00 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-08-02T23:39:55.988 INFO:tasks.ceph.mgr.x.smithi131.stderr:2024-08-02T23:39:55.986+0000 7ff194b6df00 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-08-02T23:39:56.193 INFO:tasks.ceph.mgr.x.smithi131.stderr:2024-08-02T23:39:56.191+0000 7ff194b6df00 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-08-02T23:39:56.274 INFO:tasks.ceph.mgr.x.smithi131.stderr:2024-08-02T23:39:56.271+0000 7ff194b6df00 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-08-02T23:39:56.392 INFO:tasks.ceph.mgr.x.smithi131.stderr:2024-08-02T23:39:56.389+0000 7ff194b6df00 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-08-02T23:39:56.464 INFO:tasks.ceph.mgr.x.smithi131.stderr:2024-08-02T23:39:56.461+0000 7ff194b6df00 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-08-02T23:39:57.164 INFO:tasks.ceph.mgr.x.smithi131.stderr:2024-08-02T23:39:57.161+0000 7ff194b6df00 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-08-02T23:39:57.362 INFO:tasks.ceph.mgr.x.smithi131.stderr:2024-08-02T23:39:57.359+0000 7ff194b6df00 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-08-02T23:39:58.371 INFO:tasks.ceph.mgr.x.smithi131.stderr:2024-08-02T23:39:58.368+0000 7ff133772640 -1 mgr.server handle_report got status from non-daemon mon.a 2024-08-02T23:40:01.997 DEBUG:teuthology.orchestra.run.smithi073:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-08-02T23:40:02.097 INFO:teuthology.misc.health.smithi073.stderr:2024-08-02T23:40:02.092+0000 7f859cb99640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-02T23:40:02.098 INFO:teuthology.misc.health.smithi073.stderr:2024-08-02T23:40:02.096+0000 7f859cb99640 1 Processor -- start 2024-08-02T23:40:02.098 INFO:teuthology.misc.health.smithi073.stderr:2024-08-02T23:40:02.096+0000 7f859cb99640 1 -- start start 2024-08-02T23:40:02.098 INFO:teuthology.misc.health.smithi073.stderr:2024-08-02T23:40:02.097+0000 7f859cb99640 1 --2- >> v2:172.21.15.73:3300/0 conn(0x7f8598137620 0x7f8598139a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-02T23:40:02.099 INFO:teuthology.misc.health.smithi073.stderr:2024-08-02T23:40:02.097+0000 7f859cb99640 1 -- --> v1:172.21.15.73:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f859809aec0 con 0x7f85981347d0 2024-08-02T23:40:02.099 INFO:teuthology.misc.health.smithi073.stderr:2024-08-02T23:40:02.097+0000 7f859cb99640 1 -- --> v2:172.21.15.73:3300/0 -- mon_getmap magic: 0 -- 0x7f8598134310 con 0x7f8598137620 2024-08-02T23:40:02.100 INFO:teuthology.misc.health.smithi073.stderr:2024-08-02T23:40:02.097+0000 7f8595d9b640 1 --2- >> v2:172.21.15.73:3300/0 conn(0x7f8598137620 0x7f8598139a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-02T23:40:02.100 INFO:teuthology.misc.health.smithi073.stderr:2024-08-02T23:40:02.097+0000 7f859659c640 1 --1- >> v1:172.21.15.73:6789/0 conn(0x7f85981347d0 0x7f8598134be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.73:6789/0 says I am v1:172.21.15.73:40110/0 (socket says 172.21.15.73:40110) 2024-08-02T23:40:02.100 INFO:teuthology.misc.health.smithi073.stderr:2024-08-02T23:40:02.097+0000 7f859659c640 1 -- 172.21.15.73:0/971557820 learned_addr learned my addr 172.21.15.73:0/971557820 (peer_addr_for_me v1:172.21.15.73:0/0) 2024-08-02T23:40:02.100 INFO:teuthology.misc.health.smithi073.stderr:2024-08-02T23:40:02.098+0000 7f8595d9b640 1 -- 172.21.15.73:0/971557820 >> v1:172.21.15.73:6789/0 conn(0x7f85981347d0 legacy=0x7f8598134be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-02T23:40:02.100 INFO:teuthology.misc.health.smithi073.stderr:2024-08-02T23:40:02.098+0000 7f859559a640 1 -- 172.21.15.73:0/971557820 <== mon.0 v1:172.21.15.73:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1509535441 0 0) 0x7f859809aec0 con 0x7f85981347d0 2024-08-02T23:40:02.100 INFO:teuthology.misc.health.smithi073.stderr:2024-08-02T23:40:02.098+0000 7f8595d9b640 1 -- 172.21.15.73:0/971557820 --> v2:172.21.15.73:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85980a0d10 con 0x7f8598137620 2024-08-02T23:40:02.100 INFO:teuthology.misc.health.smithi073.stderr:2024-08-02T23:40:02.098+0000 7f8595d9b640 1 --2- 172.21.15.73:0/971557820 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8598137620 0x7f8598139a40 crc :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=837e25668f9c958d server_cookie=0 in_seq=0 out_seq=0 2024-08-02T23:40:02.101 INFO:teuthology.misc.health.smithi073.stderr:2024-08-02T23:40:02.098+0000 7f859559a640 1 -- 172.21.15.73:0/971557820 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f8588009ce0 con 0x7f8598137620 2024-08-02T23:40:02.101 INFO:teuthology.misc.health.smithi073.stderr:2024-08-02T23:40:02.098+0000 7f859559a640 1 -- 172.21.15.73:0/971557820 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f8588009e40 con 0x7f8598137620 2024-08-02T23:40:02.101 INFO:teuthology.misc.health.smithi073.stderr:2024-08-02T23:40:02.098+0000 7f859559a640 1 -- 172.21.15.73:0/971557820 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f858800e360 con 0x7f8598137620 2024-08-02T23:40:02.102 INFO:teuthology.misc.health.smithi073.stderr:2024-08-02T23:40:02.099+0000 7f859cb99640 1 -- 172.21.15.73:0/971557820 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8598137620 msgr2=0x7f8598139a40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-02T23:40:02.102 INFO:teuthology.misc.health.smithi073.stderr:2024-08-02T23:40:02.099+0000 7f859cb99640 1 --2- 172.21.15.73:0/971557820 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8598137620 0x7f8598139a40 crc :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:40:02.102 INFO:teuthology.misc.health.smithi073.stderr:2024-08-02T23:40:02.099+0000 7f859cb99640 1 -- 172.21.15.73:0/971557820 shutdown_connections 2024-08-02T23:40:02.102 INFO:teuthology.misc.health.smithi073.stderr:2024-08-02T23:40:02.099+0000 7f859cb99640 1 --2- 172.21.15.73:0/971557820 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8598137620 0x7f8598139a40 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:40:02.102 INFO:teuthology.misc.health.smithi073.stderr:2024-08-02T23:40:02.099+0000 7f859cb99640 1 -- 172.21.15.73:0/971557820 >> 172.21.15.73:0/971557820 conn(0x7f859806cea0 msgr2=0x7f859806d2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-02T23:40:02.102 INFO:teuthology.misc.health.smithi073.stderr:2024-08-02T23:40:02.099+0000 7f859cb99640 1 -- 172.21.15.73:0/971557820 shutdown_connections 2024-08-02T23:40:02.102 INFO:teuthology.misc.health.smithi073.stderr:2024-08-02T23:40:02.099+0000 7f859cb99640 1 -- 172.21.15.73:0/971557820 wait complete. 2024-08-02T23:40:02.103 INFO:teuthology.misc.health.smithi073.stderr:2024-08-02T23:40:02.099+0000 7f859cb99640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-02T23:40:02.103 INFO:teuthology.misc.health.smithi073.stderr:2024-08-02T23:40:02.100+0000 7f859cb99640 1 Processor -- start 2024-08-02T23:40:02.103 INFO:teuthology.misc.health.smithi073.stderr:2024-08-02T23:40:02.100+0000 7f859cb99640 1 -- start start 2024-08-02T23:40:02.105 INFO:teuthology.misc.health.smithi073.stderr:2024-08-02T23:40:02.101+0000 7f859cb99640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f85981347d0 0x7f859811efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-02T23:40:02.105 INFO:teuthology.misc.health.smithi073.stderr:2024-08-02T23:40:02.101+0000 7f859cb99640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 -- 0x7f85981202e0 con 0x7f85981347d0 2024-08-02T23:40:02.105 INFO:teuthology.misc.health.smithi073.stderr:2024-08-02T23:40:02.101+0000 7f859659c640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f85981347d0 0x7f859811efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-02T23:40:02.105 INFO:teuthology.misc.health.smithi073.stderr:2024-08-02T23:40:02.101+0000 7f859659c640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f85981347d0 0x7f859811efe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:39610/0 (socket says 172.21.15.73:39610) 2024-08-02T23:40:02.105 INFO:teuthology.misc.health.smithi073.stderr:2024-08-02T23:40:02.101+0000 7f859659c640 1 -- 172.21.15.73:0/2841049662 learned_addr learned my addr 172.21.15.73:0/2841049662 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-08-02T23:40:02.105 INFO:teuthology.misc.health.smithi073.stderr:2024-08-02T23:40:02.101+0000 7f859659c640 1 -- 172.21.15.73:0/2841049662 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8598120b50 con 0x7f85981347d0 2024-08-02T23:40:02.105 INFO:teuthology.misc.health.smithi073.stderr:2024-08-02T23:40:02.101+0000 7f859659c640 1 --2- 172.21.15.73:0/2841049662 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f85981347d0 0x7f859811efe0 crc :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-02T23:40:02.105 INFO:teuthology.misc.health.smithi073.stderr:2024-08-02T23:40:02.101+0000 7f8586ffd640 1 -- 172.21.15.73:0/2841049662 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f858c004af0 con 0x7f85981347d0 2024-08-02T23:40:02.105 INFO:teuthology.misc.health.smithi073.stderr:2024-08-02T23:40:02.101+0000 7f8586ffd640 1 -- 172.21.15.73:0/2841049662 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f858c004c50 con 0x7f85981347d0 2024-08-02T23:40:02.105 INFO:teuthology.misc.health.smithi073.stderr:2024-08-02T23:40:02.101+0000 7f859cb99640 1 -- 172.21.15.73:0/2841049662 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f859811ecd0 con 0x7f85981347d0 2024-08-02T23:40:02.106 INFO:teuthology.misc.health.smithi073.stderr:2024-08-02T23:40:02.101+0000 7f8586ffd640 1 -- 172.21.15.73:0/2841049662 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f858c007120 con 0x7f85981347d0 2024-08-02T23:40:02.106 INFO:teuthology.misc.health.smithi073.stderr:2024-08-02T23:40:02.101+0000 7f859cb99640 1 -- 172.21.15.73:0/2841049662 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f859811e750 con 0x7f85981347d0 2024-08-02T23:40:02.106 INFO:teuthology.misc.health.smithi073.stderr:2024-08-02T23:40:02.102+0000 7f8586ffd640 1 -- 172.21.15.73:0/2841049662 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 5) ==== 45563+0+0 (crc 0 0 0) 0x7f858c0072d0 con 0x7f85981347d0 2024-08-02T23:40:02.106 INFO:teuthology.misc.health.smithi073.stderr:2024-08-02T23:40:02.102+0000 7f859cb99640 1 -- 172.21.15.73:0/2841049662 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f859809bff0 con 0x7f85981347d0 2024-08-02T23:40:02.106 INFO:teuthology.misc.health.smithi073.stderr:2024-08-02T23:40:02.103+0000 7f8586ffd640 1 --2- 172.21.15.73:0/2841049662 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7f8578038390 0x7f857803a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-02T23:40:02.106 INFO:teuthology.misc.health.smithi073.stderr:2024-08-02T23:40:02.103+0000 7f8586ffd640 1 -- 172.21.15.73:0/2841049662 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(12..12 src has 1..12) ==== 2319+0+0 (crc 0 0 0) 0x7f858c047a60 con 0x7f85981347d0 2024-08-02T23:40:02.106 INFO:teuthology.misc.health.smithi073.stderr:2024-08-02T23:40:02.103+0000 7f8595d9b640 1 --2- 172.21.15.73:0/2841049662 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7f8578038390 0x7f857803a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-02T23:40:02.106 INFO:teuthology.misc.health.smithi073.stderr:2024-08-02T23:40:02.103+0000 7f8595d9b640 1 --2- 172.21.15.73:0/2841049662 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7f8578038390 0x7f857803a8a0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-02T23:40:02.109 INFO:teuthology.misc.health.smithi073.stderr:2024-08-02T23:40:02.108+0000 7f8586ffd640 1 -- 172.21.15.73:0/2841049662 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186747 (crc 0 0 0) 0x7f858c00a9e0 con 0x7f85981347d0 2024-08-02T23:40:02.293 INFO:teuthology.misc.health.smithi073.stderr:2024-08-02T23:40:02.291+0000 7f859cb99640 1 -- 172.21.15.73:0/2841049662 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f8598121720 con 0x7f85981347d0 2024-08-02T23:40:02.293 INFO:teuthology.misc.health.smithi073.stderr:2024-08-02T23:40:02.292+0000 7f8586ffd640 1 -- 172.21.15.73:0/2841049662 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v12) ==== 74+0+6939 (crc 0 0 0) 0x7f858c00a5d0 con 0x7f85981347d0 2024-08-02T23:40:02.293 INFO:teuthology.misc.health.smithi073.stdout: 2024-08-02T23:40:02.294 INFO:teuthology.misc.health.smithi073.stdout:{"epoch":12,"fsid":"7caeb70d-fd31-4bdf-94e1-d0f19ea8e3c6","created":"2024-08-02T23:39:52.389003+0000","modified":"2024-08-02T23:40:01.493682+0000","last_up_change":"2024-08-02T23:39:59.483999+0000","last_in_change":"2024-08-02T23:39:54.500028+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-02T23:39:59.404407+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bd37ae5d-55cc-4a6d-b943-8de7e40fd1bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6801","nonce":4128208498}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6800","nonce":4128208498}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6808","nonce":4128208498}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6805","nonce":4128208498}]},"public_addr":"172.21.15.73:6801/4128208498","cluster_addr":"172.21.15.73:6800/4128208498","heartbeat_back_addr":"172.21.15.73:6808/4128208498","heartbeat_front_addr":"172.21.15.73:6805/4128208498","state":["exists","up"]},{"osd":1,"uuid":"6bf29a4c-3240-4fce-8803-66926d0df50c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6804","nonce":3612805766}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6803","nonce":3612805766}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6807","nonce":3612805766}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6806","nonce":3612805766}]},"public_addr":"172.21.15.73:6804/3612805766","cluster_addr":"172.21.15.73:6803/3612805766","heartbeat_back_addr":"172.21.15.73:6807/3612805766","heartbeat_front_addr":"172.21.15.73:6806/3612805766","state":["exists","up"]},{"osd":2,"uuid":"7d855822-09de-49f2-8811-c47826c38d5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6802","nonce":3558187202}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6803","nonce":3558187202}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6811","nonce":3558187202}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6810","nonce":3558187202}]},"public_addr":"172.21.15.131:6802/3558187202","cluster_addr":"172.21.15.131:6803/3558187202","heartbeat_back_addr":"172.21.15.131:6811/3558187202","heartbeat_front_addr":"172.21.15.131:6810/3558187202","state":["exists","up"]},{"osd":3,"uuid":"d6e6bd88-9a90-441c-8bbd-8b8d4a2b4f9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6809","nonce":1209903557}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6808","nonce":1209903557}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6813","nonce":1209903557}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6812","nonce":1209903557}]},"public_addr":"172.21.15.131:6809/1209903557","cluster_addr":"172.21.15.131:6808/1209903557","heartbeat_back_addr":"172.21.15.131:6813/1209903557","heartbeat_front_addr":"172.21.15.131:6812/1209903557","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-02T23:40:02.296 INFO:teuthology.misc.health.smithi073.stderr:2024-08-02T23:40:02.294+0000 7f859cb99640 1 -- 172.21.15.73:0/2841049662 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7f8578038390 msgr2=0x7f857803a8a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-02T23:40:02.296 INFO:teuthology.misc.health.smithi073.stderr:2024-08-02T23:40:02.294+0000 7f859cb99640 1 --2- 172.21.15.73:0/2841049662 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7f8578038390 0x7f857803a8a0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:40:02.296 INFO:teuthology.misc.health.smithi073.stderr:2024-08-02T23:40:02.294+0000 7f859cb99640 1 -- 172.21.15.73:0/2841049662 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f85981347d0 msgr2=0x7f859811efe0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-02T23:40:02.296 INFO:teuthology.misc.health.smithi073.stderr:2024-08-02T23:40:02.294+0000 7f859cb99640 1 --2- 172.21.15.73:0/2841049662 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f85981347d0 0x7f859811efe0 crc :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:40:02.296 INFO:teuthology.misc.health.smithi073.stderr:2024-08-02T23:40:02.294+0000 7f859cb99640 1 -- 172.21.15.73:0/2841049662 shutdown_connections 2024-08-02T23:40:02.297 INFO:teuthology.misc.health.smithi073.stderr:2024-08-02T23:40:02.294+0000 7f859cb99640 1 --2- 172.21.15.73:0/2841049662 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7f8578038390 0x7f857803a8a0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:40:02.297 INFO:teuthology.misc.health.smithi073.stderr:2024-08-02T23:40:02.294+0000 7f859cb99640 1 --2- 172.21.15.73:0/2841049662 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f85981347d0 0x7f859811efe0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:40:02.297 INFO:teuthology.misc.health.smithi073.stderr:2024-08-02T23:40:02.294+0000 7f859cb99640 1 -- 172.21.15.73:0/2841049662 >> 172.21.15.73:0/2841049662 conn(0x7f859806cea0 msgr2=0x7f85980a1dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-02T23:40:02.297 INFO:teuthology.misc.health.smithi073.stderr:2024-08-02T23:40:02.294+0000 7f859cb99640 1 -- 172.21.15.73:0/2841049662 shutdown_connections 2024-08-02T23:40:02.297 INFO:teuthology.misc.health.smithi073.stderr:2024-08-02T23:40:02.294+0000 7f859cb99640 1 -- 172.21.15.73:0/2841049662 wait complete. 2024-08-02T23:40:02.305 DEBUG:teuthology.misc:4 of 4 OSDs are up 2024-08-02T23:40:02.305 INFO:tasks.ceph:Creating RBD pool 2024-08-02T23:40:02.306 DEBUG:teuthology.orchestra.run.smithi073:> sudo ceph --cluster ceph osd pool create rbd 8 2024-08-02T23:40:02.412 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:02.404+0000 7fcca64ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-02T23:40:02.413 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:02.411+0000 7fcca64ad640 1 Processor -- start 2024-08-02T23:40:02.413 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:02.411+0000 7fcca64ad640 1 -- start start 2024-08-02T23:40:02.414 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:02.412+0000 7fcca64ad640 1 --2- >> v2:172.21.15.73:3300/0 conn(0x7fcca0134ed0 0x7fcca01398e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-02T23:40:02.415 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:02.412+0000 7fcca64ad640 1 -- --> v1:172.21.15.73:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcca009ab80 con 0x7fcca01343b0 2024-08-02T23:40:02.415 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:02.412+0000 7fcca64ad640 1 -- --> v2:172.21.15.73:3300/0 -- mon_getmap magic: 0 -- 0x7fcca00a4870 con 0x7fcca0134ed0 2024-08-02T23:40:02.415 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:02.412+0000 7fcc9f7fe640 1 --2- >> v2:172.21.15.73:3300/0 conn(0x7fcca0134ed0 0x7fcca01398e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-02T23:40:02.415 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:02.412+0000 7fcc9ffff640 1 --1- >> v1:172.21.15.73:6789/0 conn(0x7fcca01343b0 0x7fcca01347c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.73:6789/0 says I am v1:172.21.15.73:40116/0 (socket says 172.21.15.73:40116) 2024-08-02T23:40:02.415 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:02.412+0000 7fcc9ffff640 1 -- 172.21.15.73:0/2941684962 learned_addr learned my addr 172.21.15.73:0/2941684962 (peer_addr_for_me v1:172.21.15.73:0/0) 2024-08-02T23:40:02.415 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:02.412+0000 7fcc9f7fe640 1 -- 172.21.15.73:0/2941684962 >> v1:172.21.15.73:6789/0 conn(0x7fcca01343b0 legacy=0x7fcca01347c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-02T23:40:02.415 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:02.412+0000 7fcc9f7fe640 1 -- 172.21.15.73:0/2941684962 --> v2:172.21.15.73:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcca00a4b10 con 0x7fcca0134ed0 2024-08-02T23:40:02.415 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:02.413+0000 7fcc9f7fe640 1 --2- 172.21.15.73:0/2941684962 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcca0134ed0 0x7fcca01398e0 crc :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92c818d7d290188b server_cookie=0 in_seq=0 out_seq=0 2024-08-02T23:40:02.415 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:02.413+0000 7fcc9effd640 1 -- 172.21.15.73:0/2941684962 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fcc90009cf0 con 0x7fcca0134ed0 2024-08-02T23:40:02.415 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:02.413+0000 7fcc9effd640 1 -- 172.21.15.73:0/2941684962 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7fcc90009e50 con 0x7fcca0134ed0 2024-08-02T23:40:02.415 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:02.413+0000 7fcc9effd640 1 -- 172.21.15.73:0/2941684962 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fcc90005a90 con 0x7fcca0134ed0 2024-08-02T23:40:02.416 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:02.414+0000 7fcca64ad640 1 -- 172.21.15.73:0/2941684962 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcca0134ed0 msgr2=0x7fcca01398e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-02T23:40:02.416 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:02.414+0000 7fcca64ad640 1 --2- 172.21.15.73:0/2941684962 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcca0134ed0 0x7fcca01398e0 crc :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:40:02.417 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:02.414+0000 7fcca64ad640 1 -- 172.21.15.73:0/2941684962 shutdown_connections 2024-08-02T23:40:02.417 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:02.414+0000 7fcca64ad640 1 --2- 172.21.15.73:0/2941684962 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcca0134ed0 0x7fcca01398e0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:40:02.417 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:02.414+0000 7fcca64ad640 1 -- 172.21.15.73:0/2941684962 >> 172.21.15.73:0/2941684962 conn(0x7fcca00a39e0 msgr2=0x7fcca00a3df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-02T23:40:02.417 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:02.414+0000 7fcca64ad640 1 -- 172.21.15.73:0/2941684962 shutdown_connections 2024-08-02T23:40:02.417 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:02.414+0000 7fcca64ad640 1 -- 172.21.15.73:0/2941684962 wait complete. 2024-08-02T23:40:02.417 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:02.414+0000 7fcca64ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-02T23:40:02.417 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:02.416+0000 7fcca64ad640 1 Processor -- start 2024-08-02T23:40:02.418 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:02.416+0000 7fcca64ad640 1 -- start start 2024-08-02T23:40:02.420 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:02.417+0000 7fcca64ad640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcca01343b0 0x7fcca011dd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-02T23:40:02.420 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:02.417+0000 7fcca64ad640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 -- 0x7fcca0131fc0 con 0x7fcca01343b0 2024-08-02T23:40:02.420 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:02.417+0000 7fcc9ffff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcca01343b0 0x7fcca011dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-02T23:40:02.420 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:02.417+0000 7fcc9ffff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcca01343b0 0x7fcca011dd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:39624/0 (socket says 172.21.15.73:39624) 2024-08-02T23:40:02.421 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:02.417+0000 7fcc9ffff640 1 -- 172.21.15.73:0/4240554663 learned_addr learned my addr 172.21.15.73:0/4240554663 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-08-02T23:40:02.421 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:02.418+0000 7fcc9ffff640 1 -- 172.21.15.73:0/4240554663 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcca011f0f0 con 0x7fcca01343b0 2024-08-02T23:40:02.421 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:02.418+0000 7fcc9ffff640 1 --2- 172.21.15.73:0/4240554663 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcca01343b0 0x7fcca011dd30 crc :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-02T23:40:02.421 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:02.418+0000 7fcc9cff9640 1 -- 172.21.15.73:0/4240554663 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fcc880058a0 con 0x7fcca01343b0 2024-08-02T23:40:02.421 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:02.418+0000 7fcc9cff9640 1 -- 172.21.15.73:0/4240554663 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7fcc88005a00 con 0x7fcca01343b0 2024-08-02T23:40:02.421 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:02.418+0000 7fcca64ad640 1 -- 172.21.15.73:0/4240554663 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcca01207e0 con 0x7fcca01343b0 2024-08-02T23:40:02.421 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:02.418+0000 7fcc9cff9640 1 -- 172.21.15.73:0/4240554663 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fcc88005ca0 con 0x7fcca01343b0 2024-08-02T23:40:02.421 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:02.418+0000 7fcca64ad640 1 -- 172.21.15.73:0/4240554663 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcca0121d90 con 0x7fcca01343b0 2024-08-02T23:40:02.422 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:02.419+0000 7fcc9cff9640 1 -- 172.21.15.73:0/4240554663 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 5) ==== 45563+0+0 (crc 0 0 0) 0x7fcc880070b0 con 0x7fcca01343b0 2024-08-02T23:40:02.422 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:02.419+0000 7fcca64ad640 1 -- 172.21.15.73:0/4240554663 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcca0120180 con 0x7fcca01343b0 2024-08-02T23:40:02.422 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:02.419+0000 7fcc9cff9640 1 --2- 172.21.15.73:0/4240554663 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7fcc80038390 0x7fcc8003a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-02T23:40:02.422 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:02.419+0000 7fcc9cff9640 1 -- 172.21.15.73:0/4240554663 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(12..12 src has 1..12) ==== 2319+0+0 (crc 0 0 0) 0x7fcc880478e0 con 0x7fcca01343b0 2024-08-02T23:40:02.423 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:02.420+0000 7fcc9f7fe640 1 --2- 172.21.15.73:0/4240554663 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7fcc80038390 0x7fcc8003a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-02T23:40:02.423 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:02.420+0000 7fcc9f7fe640 1 --2- 172.21.15.73:0/4240554663 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7fcc80038390 0x7fcc8003a8a0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-02T23:40:02.426 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:02.424+0000 7fcc9cff9640 1 -- 172.21.15.73:0/4240554663 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186747 (crc 0 0 0) 0x7fcc8800da70 con 0x7fcca01343b0 2024-08-02T23:40:02.612 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:02.610+0000 7fcca64ad640 1 -- 172.21.15.73:0/4240554663 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) -- 0x7fcca01347c0 con 0x7fcca01343b0 2024-08-02T23:40:03.504 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:03.502+0000 7fcc9cff9640 1 -- 172.21.15.73:0/4240554663 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]=0 pool 'rbd' created v13) ==== 109+0+0 (crc 0 0 0) 0x7fcc8800d660 con 0x7fcca01343b0 2024-08-02T23:40:03.504 INFO:teuthology.orchestra.run.smithi073.stderr:pool 'rbd' created 2024-08-02T23:40:03.506 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:03.504+0000 7fcca64ad640 1 -- 172.21.15.73:0/4240554663 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7fcc80038390 msgr2=0x7fcc8003a8a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-02T23:40:03.506 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:03.504+0000 7fcca64ad640 1 --2- 172.21.15.73:0/4240554663 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7fcc80038390 0x7fcc8003a8a0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:40:03.506 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:03.504+0000 7fcca64ad640 1 -- 172.21.15.73:0/4240554663 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcca01343b0 msgr2=0x7fcca011dd30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-02T23:40:03.507 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:03.504+0000 7fcca64ad640 1 --2- 172.21.15.73:0/4240554663 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcca01343b0 0x7fcca011dd30 crc :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:40:03.507 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:03.504+0000 7fcca64ad640 1 -- 172.21.15.73:0/4240554663 shutdown_connections 2024-08-02T23:40:03.507 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:03.504+0000 7fcca64ad640 1 --2- 172.21.15.73:0/4240554663 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7fcc80038390 0x7fcc8003a8a0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:40:03.507 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:03.504+0000 7fcca64ad640 1 --2- 172.21.15.73:0/4240554663 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcca01343b0 0x7fcca011dd30 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:40:03.507 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:03.504+0000 7fcca64ad640 1 -- 172.21.15.73:0/4240554663 >> 172.21.15.73:0/4240554663 conn(0x7fcca00a39e0 msgr2=0x7fcca0132320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-02T23:40:03.507 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:03.504+0000 7fcca64ad640 1 -- 172.21.15.73:0/4240554663 shutdown_connections 2024-08-02T23:40:03.507 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:03.504+0000 7fcca64ad640 1 -- 172.21.15.73:0/4240554663 wait complete. 2024-08-02T23:40:03.522 DEBUG:teuthology.orchestra.run.smithi073:> rbd --cluster ceph pool init rbd 2024-08-02T23:40:03.587 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:03.585+0000 7f22a088cd00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-02T23:40:03.587 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:03.585+0000 7f22a088cd00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-02T23:40:03.587 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:03.586+0000 7f22a088cd00 1 Processor -- start 2024-08-02T23:40:03.587 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:03.586+0000 7f22a088cd00 1 -- start start 2024-08-02T23:40:03.589 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:03.586+0000 7f22a088cd00 1 --2- >> v2:172.21.15.73:3300/0 conn(0x55912f63a1c0 0x55912f63a5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-02T23:40:03.589 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:03.586+0000 7f22a088cd00 1 -- --> v1:172.21.15.73:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x55912f4cb080 con 0x55912f63ab10 2024-08-02T23:40:03.589 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:03.586+0000 7f22a088cd00 1 -- --> v2:172.21.15.73:3300/0 -- mon_getmap magic: 0 -- 0x55912f514030 con 0x55912f63a1c0 2024-08-02T23:40:03.589 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:03.587+0000 7f2299c48640 1 --2- >> v2:172.21.15.73:3300/0 conn(0x55912f63a1c0 0x55912f63a5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-02T23:40:03.589 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:03.587+0000 7f2299447640 1 --1- >> v1:172.21.15.73:6789/0 conn(0x55912f63ab10 0x55912f6c8ad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.73:6789/0 says I am v1:172.21.15.73:40120/0 (socket says 172.21.15.73:40120) 2024-08-02T23:40:03.589 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:03.587+0000 7f2299c48640 1 --2- >> v2:172.21.15.73:3300/0 conn(0x55912f63a1c0 0x55912f63a5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:39626/0 (socket says 172.21.15.73:39626) 2024-08-02T23:40:03.589 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:03.587+0000 7f2299447640 1 -- 172.21.15.73:0/4177765130 learned_addr learned my addr 172.21.15.73:0/4177765130 (peer_addr_for_me v1:172.21.15.73:0/0) 2024-08-02T23:40:03.589 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:03.587+0000 7f2299c48640 1 -- 172.21.15.73:0/4177765130 >> v1:172.21.15.73:6789/0 conn(0x55912f63ab10 legacy=0x55912f6c8ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-02T23:40:03.589 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:03.587+0000 7f2299c48640 1 -- 172.21.15.73:0/4177765130 --> v2:172.21.15.73:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x55912f4a33f0 con 0x55912f63a1c0 2024-08-02T23:40:03.590 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:03.588+0000 7f2299c48640 1 --2- 172.21.15.73:0/4177765130 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55912f63a1c0 0x55912f63a5d0 crc :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4628bf72a1c03d29 server_cookie=0 in_seq=0 out_seq=0 2024-08-02T23:40:03.590 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:03.588+0000 7f2298c46640 1 -- 172.21.15.73:0/4177765130 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f22900090b0 con 0x55912f63a1c0 2024-08-02T23:40:03.590 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:03.588+0000 7f2298c46640 1 -- 172.21.15.73:0/4177765130 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f2290009210 con 0x55912f63a1c0 2024-08-02T23:40:03.590 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:03.588+0000 7f2298c46640 1 -- 172.21.15.73:0/4177765130 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f2290007490 con 0x55912f63a1c0 2024-08-02T23:40:03.592 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:03.589+0000 7f22a088cd00 1 -- 172.21.15.73:0/4177765130 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55912f63a1c0 msgr2=0x55912f63a5d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-02T23:40:03.592 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:03.589+0000 7f22a088cd00 1 --2- 172.21.15.73:0/4177765130 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55912f63a1c0 0x55912f63a5d0 crc :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:40:03.592 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:03.589+0000 7f22a088cd00 1 -- 172.21.15.73:0/4177765130 shutdown_connections 2024-08-02T23:40:03.592 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:03.589+0000 7f22a088cd00 1 --2- 172.21.15.73:0/4177765130 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55912f63a1c0 0x55912f63a5d0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:40:03.592 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:03.589+0000 7f22a088cd00 1 -- 172.21.15.73:0/4177765130 >> 172.21.15.73:0/4177765130 conn(0x55912f482ea0 msgr2=0x55912f637d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-02T23:40:03.592 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:03.589+0000 7f22a088cd00 1 -- 172.21.15.73:0/4177765130 shutdown_connections 2024-08-02T23:40:03.592 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:03.589+0000 7f22a088cd00 1 -- 172.21.15.73:0/4177765130 wait complete. 2024-08-02T23:40:03.592 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:03.589+0000 7f22a088cd00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-02T23:40:03.592 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:03.590+0000 7f22a088cd00 1 Processor -- start 2024-08-02T23:40:03.592 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:03.590+0000 7f22a088cd00 1 -- start start 2024-08-02T23:40:03.594 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:03.591+0000 7f22a088cd00 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55912f63a1c0 0x55912f6d6fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-02T23:40:03.594 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:03.591+0000 7f22a088cd00 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 -- 0x55912f511560 con 0x55912f63a1c0 2024-08-02T23:40:03.595 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:03.591+0000 7f2299c48640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55912f63a1c0 0x55912f6d6fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-02T23:40:03.595 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:03.591+0000 7f2299c48640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55912f63a1c0 0x55912f6d6fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:39636/0 (socket says 172.21.15.73:39636) 2024-08-02T23:40:03.595 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:03.591+0000 7f2299c48640 1 -- 172.21.15.73:0/1760051023 learned_addr learned my addr 172.21.15.73:0/1760051023 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-08-02T23:40:03.595 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:03.591+0000 7f2299c48640 1 -- 172.21.15.73:0/1760051023 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55912f4b05d0 con 0x55912f63a1c0 2024-08-02T23:40:03.595 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:03.591+0000 7f2299c48640 1 --2- 172.21.15.73:0/1760051023 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55912f63a1c0 0x55912f6d6fd0 crc :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-02T23:40:03.595 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:03.591+0000 7f2289ffb640 1 -- 172.21.15.73:0/1760051023 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f2290007490 con 0x55912f63a1c0 2024-08-02T23:40:03.595 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:03.591+0000 7f2289ffb640 1 -- 172.21.15.73:0/1760051023 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f2290009380 con 0x55912f63a1c0 2024-08-02T23:40:03.595 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:03.591+0000 7f22a088cd00 1 -- 172.21.15.73:0/1760051023 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55912f4a3780 con 0x55912f63a1c0 2024-08-02T23:40:03.595 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:03.591+0000 7f2289ffb640 1 -- 172.21.15.73:0/1760051023 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f2290036560 con 0x55912f63a1c0 2024-08-02T23:40:03.595 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:03.591+0000 7f22a088cd00 1 -- 172.21.15.73:0/1760051023 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55912f4a31e0 con 0x55912f63a1c0 2024-08-02T23:40:03.595 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:03.592+0000 7f2289ffb640 1 -- 172.21.15.73:0/1760051023 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 5) ==== 45563+0+0 (crc 0 0 0) 0x7f22900366c0 con 0x55912f63a1c0 2024-08-02T23:40:03.595 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:03.592+0000 7f2289ffb640 1 --2- 172.21.15.73:0/1760051023 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7f2278036cb0 0x7f22780391c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-02T23:40:03.596 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:03.593+0000 7f2289ffb640 1 -- 172.21.15.73:0/1760051023 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(13..13 src has 1..13) ==== 2661+0+0 (crc 0 0 0) 0x7f229002e040 con 0x55912f63a1c0 2024-08-02T23:40:03.596 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:03.593+0000 7f22a088cd00 1 -- 172.21.15.73:0/1760051023 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "osd pool application enable","pool": "rbd","app": "rbd"} v 0) -- 0x55912f6b7d80 con 0x55912f63a1c0 2024-08-02T23:40:03.596 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:03.593+0000 7f2299447640 1 --2- 172.21.15.73:0/1760051023 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7f2278036cb0 0x7f22780391c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-02T23:40:03.596 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:03.593+0000 7f2299447640 1 --2- 172.21.15.73:0/1760051023 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7f2278036cb0 0x7f22780391c0 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-02T23:40:04.512 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:04.510+0000 7f2289ffb640 1 -- 172.21.15.73:0/1760051023 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}]=0 enabled application 'rbd' on pool 'rbd' v14) ==== 141+0+0 (crc 0 0 0) 0x7f22900353c0 con 0x55912f63a1c0 2024-08-02T23:40:04.512 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:04.510+0000 7f22a088cd00 1 -- 172.21.15.73:0/1760051023 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x55912f4a4900 con 0x55912f63a1c0 2024-08-02T23:40:04.516 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:04.514+0000 7f2289ffb640 1 -- 172.21.15.73:0/1760051023 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_get_version_reply(handle=1 version=14) ==== 24+0+0 (crc 0 0 0) 0x7f22900353c0 con 0x55912f63a1c0 2024-08-02T23:40:04.516 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:04.514+0000 7f228bfff640 1 -- 172.21.15.73:0/1760051023 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=14}) -- 0x7f2274002ae0 con 0x55912f63a1c0 2024-08-02T23:40:04.517 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:04.515+0000 7f2289ffb640 1 -- 172.21.15.73:0/1760051023 <== mon.0 v2:172.21.15.73:3300/0 8 ==== osd_map(14..14 src has 1..14) ==== 658+0+0 (crc 0 0 0) 0x7f22900304b0 con 0x55912f63a1c0 2024-08-02T23:40:04.517 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:04.515+0000 7f22a088cd00 1 --2- 172.21.15.73:0/1760051023 >> v2:172.21.15.73:6804/3612805766 conn(0x55912f63ab10 0x55912f6b3d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-02T23:40:04.517 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:04.515+0000 7f22a088cd00 1 -- 172.21.15.73:0/1760051023 --> v2:172.21.15.73:6804/3612805766 -- osd_op(unknown.0.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14) -- 0x55912f6b4270 con 0x55912f63ab10 2024-08-02T23:40:04.519 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:04.517+0000 7f229aeaa640 1 --2- 172.21.15.73:0/1760051023 >> v2:172.21.15.73:6804/3612805766 conn(0x55912f63ab10 0x55912f6b3d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-08-02T23:40:04.525 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:04.523+0000 7f229aeaa640 1 --2- 172.21.15.73:0/1760051023 >> v2:172.21.15.73:6804/3612805766 conn(0x55912f63ab10 0x55912f6b3d30 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-02T23:40:04.577 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:04.575+0000 7f229aeaa640 1 -- 172.21.15.73:0/1760051023 <== osd.1 v2:172.21.15.73:6804/3612805766 1 ==== osd_op_reply(1 rbd_trash [create] v14'1 uv1 ondisk = 0) ==== 153+0+0 (crc 0 0 0) 0x7f228c00a4b0 con 0x55912f63ab10 2024-08-02T23:40:04.580 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:04.578+0000 7f22a088cd00 1 --2- 172.21.15.73:0/1760051023 >> v2:172.21.15.73:6801/4128208498 conn(0x55912f72d060 0x55912f72f540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-02T23:40:04.580 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:04.578+0000 7f22a088cd00 1 -- 172.21.15.73:0/1760051023 --> v2:172.21.15.73:6801/4128208498 -- osd_op(unknown.0.0:2 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=33b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e14) -- 0x55912f72fb10 con 0x55912f72d060 2024-08-02T23:40:04.581 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:04.579+0000 7f2299c48640 1 --2- 172.21.15.73:0/1760051023 >> v2:172.21.15.73:6801/4128208498 conn(0x55912f72d060 0x55912f72f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-08-02T23:40:04.582 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:04.581+0000 7f2299c48640 1 --2- 172.21.15.73:0/1760051023 >> v2:172.21.15.73:6801/4128208498 conn(0x55912f72d060 0x55912f72f540 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-02T23:40:04.586 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:04.584+0000 7f2299c48640 1 -- 172.21.15.73:0/1760051023 <== osd.0 v2:172.21.15.73:6801/4128208498 1 ==== osd_op_reply(2 rbd_info [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 152+0+0 (crc 0 0 0) 0x7f2290001bc0 con 0x55912f72d060 2024-08-02T23:40:04.586 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:04.584+0000 7f22a088cd00 1 -- 172.21.15.73:0/1760051023 --> v2:172.21.15.73:6801/4128208498 -- osd_op(unknown.0.0:3 2.2 2:5cea7035:::rbd_info:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e14) -- 0x55912f72c520 con 0x55912f72d060 2024-08-02T23:40:04.587 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:04.586+0000 7f2299c48640 1 -- 172.21.15.73:0/1760051023 <== osd.0 v2:172.21.15.73:6801/4128208498 2 ==== osd_op_reply(3 rbd_info [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 152+0+0 (crc 0 0 0) 0x7f2290001bc0 con 0x55912f72d060 2024-08-02T23:40:04.587 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:04.586+0000 7f228b7fe640 1 -- 172.21.15.73:0/1760051023 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- pool_op(create unmanaged snap pool 2 tid 4 name v0) -- 0x7f226c001f60 con 0x55912f63a1c0 2024-08-02T23:40:05.524 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:05.522+0000 7f2289ffb640 1 -- 172.21.15.73:0/1760051023 <== mon.0 v2:172.21.15.73:3300/0 9 ==== pool_op_reply(tid 4 (0) Success v15) ==== 55+0+0 (crc 0 0 0) 0x7f22900312c0 con 0x55912f63a1c0 2024-08-02T23:40:05.524 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:05.522+0000 7f2289ffb640 1 -- 172.21.15.73:0/1760051023 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=15}) -- 0x7f227803c050 con 0x55912f63a1c0 2024-08-02T23:40:05.524 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:05.522+0000 7f2289ffb640 1 -- 172.21.15.73:0/1760051023 <== mon.0 v2:172.21.15.73:3300/0 10 ==== osd_map(15..15 src has 1..15) ==== 626+0+0 (crc 0 0 0) 0x7f2290032ba0 con 0x55912f63a1c0 2024-08-02T23:40:05.524 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:05.523+0000 7f228bfff640 1 -- 172.21.15.73:0/1760051023 --> v2:172.21.15.73:6801/4128208498 -- osd_op(unknown.0.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15) -- 0x7f2274005660 con 0x55912f72d060 2024-08-02T23:40:05.551 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:05.549+0000 7f2299c48640 1 -- 172.21.15.73:0/1760051023 <== osd.0 v2:172.21.15.73:6801/4128208498 3 ==== osd_op_reply(5 rbd_info [create,write 0~8] v15'1 uv1 ondisk = 0) ==== 194+0+0 (crc 0 0 0) 0x7f2290001bc0 con 0x55912f72d060 2024-08-02T23:40:05.551 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:05.549+0000 7f228b7fe640 1 -- 172.21.15.73:0/1760051023 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- pool_op(delete unmanaged snap pool 2 tid 6 name v15) -- 0x7f226c003370 con 0x55912f63a1c0 2024-08-02T23:40:06.525 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:06.524+0000 7f2289ffb640 1 -- 172.21.15.73:0/1760051023 <== mon.0 v2:172.21.15.73:3300/0 11 ==== pool_op_reply(tid 6 (0) Success v16) ==== 43+0+0 (crc 0 0 0) 0x7f22900314a0 con 0x55912f63a1c0 2024-08-02T23:40:06.526 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:06.524+0000 7f2289ffb640 1 -- 172.21.15.73:0/1760051023 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=16}) -- 0x7f227803c290 con 0x55912f63a1c0 2024-08-02T23:40:06.526 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:06.524+0000 7f2289ffb640 1 -- 172.21.15.73:0/1760051023 <== mon.0 v2:172.21.15.73:3300/0 12 ==== osd_map(16..16 src has 1..16) ==== 654+0+0 (crc 0 0 0) 0x7f2290032d60 con 0x55912f63a1c0 2024-08-02T23:40:06.526 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:06.524+0000 7f228bfff640 1 -- 172.21.15.73:0/1760051023 --> v2:172.21.15.73:6801/4128208498 -- osd_op(unknown.0.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19 in=19b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16) -- 0x7f2274005a40 con 0x55912f72d060 2024-08-02T23:40:06.553 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:06.550+0000 7f2299c48640 1 -- 172.21.15.73:0/1760051023 <== osd.0 v2:172.21.15.73:6801/4128208498 4 ==== osd_op_reply(7 rbd_info [write 0~19] v16'2 uv2 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f2290001bc0 con 0x55912f72d060 2024-08-02T23:40:06.553 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:06.550+0000 7f22a088cd00 1 -- 172.21.15.73:0/1760051023 >> v2:172.21.15.73:6801/4128208498 conn(0x55912f72d060 msgr2=0x55912f72f540 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-02T23:40:06.553 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:06.550+0000 7f22a088cd00 1 --2- 172.21.15.73:0/1760051023 >> v2:172.21.15.73:6801/4128208498 conn(0x55912f72d060 0x55912f72f540 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:40:06.553 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:06.550+0000 7f22a088cd00 1 -- 172.21.15.73:0/1760051023 >> v2:172.21.15.73:6804/3612805766 conn(0x55912f63ab10 msgr2=0x55912f6b3d30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-02T23:40:06.554 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:06.550+0000 7f22a088cd00 1 --2- 172.21.15.73:0/1760051023 >> v2:172.21.15.73:6804/3612805766 conn(0x55912f63ab10 0x55912f6b3d30 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:40:06.554 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:06.550+0000 7f22a088cd00 1 -- 172.21.15.73:0/1760051023 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7f2278036cb0 msgr2=0x7f22780391c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-02T23:40:06.554 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:06.550+0000 7f22a088cd00 1 --2- 172.21.15.73:0/1760051023 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7f2278036cb0 0x7f22780391c0 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:40:06.554 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:06.550+0000 7f22a088cd00 1 -- 172.21.15.73:0/1760051023 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55912f63a1c0 msgr2=0x55912f6d6fd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-02T23:40:06.554 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:06.550+0000 7f22a088cd00 1 --2- 172.21.15.73:0/1760051023 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55912f63a1c0 0x55912f6d6fd0 crc :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:40:06.554 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:06.551+0000 7f22a088cd00 1 -- 172.21.15.73:0/1760051023 shutdown_connections 2024-08-02T23:40:06.554 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:06.551+0000 7f22a088cd00 1 --2- 172.21.15.73:0/1760051023 >> v2:172.21.15.73:6804/3612805766 conn(0x55912f63ab10 0x55912f6b3d30 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:40:06.554 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:06.551+0000 7f22a088cd00 1 --2- 172.21.15.73:0/1760051023 >> v2:172.21.15.73:6801/4128208498 conn(0x55912f72d060 0x55912f72f540 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:40:06.554 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:06.551+0000 7f22a088cd00 1 --2- 172.21.15.73:0/1760051023 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7f2278036cb0 0x7f22780391c0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:40:06.554 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:06.551+0000 7f22a088cd00 1 --2- 172.21.15.73:0/1760051023 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55912f63a1c0 0x55912f6d6fd0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:40:06.555 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:06.551+0000 7f22a088cd00 1 -- 172.21.15.73:0/1760051023 >> 172.21.15.73:0/1760051023 conn(0x55912f482ea0 msgr2=0x55912f635ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-02T23:40:06.555 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:06.551+0000 7f22a088cd00 1 -- 172.21.15.73:0/1760051023 shutdown_connections 2024-08-02T23:40:06.555 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:06.551+0000 7f22a088cd00 1 -- 172.21.15.73:0/1760051023 wait complete. 2024-08-02T23:40:06.557 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-08-02T23:40:06.557 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-08-02T23:40:06.557 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-08-02T23:40:06.558 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-08-02T23:40:06.558 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-08-02T23:40:06.670 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:06.665+0000 7fa9c5d03640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-02T23:40:06.671 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:06.669+0000 7fa9c5d03640 1 Processor -- start 2024-08-02T23:40:06.671 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:06.669+0000 7fa9c5d03640 1 -- start start 2024-08-02T23:40:06.672 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:06.670+0000 7fa9c5d03640 1 --2- >> v2:172.21.15.73:3300/0 conn(0x7fa9c006e880 0x7fa9c006ec90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-02T23:40:06.672 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:06.670+0000 7fa9c5d03640 1 -- --> v1:172.21.15.73:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa9c006fef0 con 0x7fa9c006f1d0 2024-08-02T23:40:06.672 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:06.670+0000 7fa9c5d03640 1 -- --> v2:172.21.15.73:3300/0 -- mon_getmap magic: 0 -- 0x7fa9c009e5d0 con 0x7fa9c006e880 2024-08-02T23:40:06.673 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:06.670+0000 7fa9bf7fe640 1 --2- >> v2:172.21.15.73:3300/0 conn(0x7fa9c006e880 0x7fa9c006ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-02T23:40:06.673 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:06.670+0000 7fa9beffd640 1 --1- >> v1:172.21.15.73:6789/0 conn(0x7fa9c006f1d0 0x7fa9c0131040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.73:6789/0 says I am v1:172.21.15.73:40132/0 (socket says 172.21.15.73:40132) 2024-08-02T23:40:06.673 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:06.670+0000 7fa9beffd640 1 -- 172.21.15.73:0/814509030 learned_addr learned my addr 172.21.15.73:0/814509030 (peer_addr_for_me v1:172.21.15.73:0/0) 2024-08-02T23:40:06.673 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:06.670+0000 7fa9bf7fe640 1 -- 172.21.15.73:0/814509030 >> v1:172.21.15.73:6789/0 conn(0x7fa9c006f1d0 legacy=0x7fa9c0131040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-02T23:40:06.673 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:06.670+0000 7fa9bf7fe640 1 -- 172.21.15.73:0/814509030 --> v2:172.21.15.73:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9c006e430 con 0x7fa9c006e880 2024-08-02T23:40:06.673 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:06.670+0000 7fa9be7fc640 1 -- 172.21.15.73:0/814509030 <== mon.0 v1:172.21.15.73:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2532538489 0 0) 0x7fa9c006fef0 con 0x7fa9c006f1d0 2024-08-02T23:40:06.673 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:06.670+0000 7fa9bf7fe640 1 --2- 172.21.15.73:0/814509030 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa9c006e880 0x7fa9c006ec90 crc :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1f8f56cf790ce51 server_cookie=0 in_seq=0 out_seq=0 2024-08-02T23:40:06.673 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:06.670+0000 7fa9be7fc640 1 -- 172.21.15.73:0/814509030 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fa9b000e040 con 0x7fa9c006e880 2024-08-02T23:40:06.673 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:06.670+0000 7fa9be7fc640 1 -- 172.21.15.73:0/814509030 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7fa9b0007670 con 0x7fa9c006e880 2024-08-02T23:40:06.673 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:06.670+0000 7fa9be7fc640 1 -- 172.21.15.73:0/814509030 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fa9b0007910 con 0x7fa9c006e880 2024-08-02T23:40:06.674 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:06.671+0000 7fa9c5d03640 1 -- 172.21.15.73:0/814509030 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa9c006e880 msgr2=0x7fa9c006ec90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-02T23:40:06.674 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:06.671+0000 7fa9c5d03640 1 --2- 172.21.15.73:0/814509030 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa9c006e880 0x7fa9c006ec90 crc :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:40:06.674 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:06.671+0000 7fa9c5d03640 1 -- 172.21.15.73:0/814509030 shutdown_connections 2024-08-02T23:40:06.674 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:06.671+0000 7fa9c5d03640 1 --2- 172.21.15.73:0/814509030 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa9c006e880 0x7fa9c006ec90 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:40:06.675 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:06.671+0000 7fa9c5d03640 1 -- 172.21.15.73:0/814509030 >> 172.21.15.73:0/814509030 conn(0x7fa9c00a0590 msgr2=0x7fa9c00a09a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-02T23:40:06.675 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:06.671+0000 7fa9c5d03640 1 -- 172.21.15.73:0/814509030 shutdown_connections 2024-08-02T23:40:06.675 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:06.671+0000 7fa9c5d03640 1 -- 172.21.15.73:0/814509030 wait complete. 2024-08-02T23:40:06.675 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:06.671+0000 7fa9c5d03640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-02T23:40:06.675 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:06.672+0000 7fa9c5d03640 1 Processor -- start 2024-08-02T23:40:06.675 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:06.673+0000 7fa9c5d03640 1 -- start start 2024-08-02T23:40:06.677 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:06.673+0000 7fa9c5d03640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa9c006e880 0x7fa9c011e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-02T23:40:06.677 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:06.673+0000 7fa9c5d03640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 -- 0x7fa9c0120120 con 0x7fa9c006e880 2024-08-02T23:40:06.677 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:06.673+0000 7fa9bf7fe640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa9c006e880 0x7fa9c011e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-02T23:40:06.677 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:06.673+0000 7fa9bf7fe640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa9c006e880 0x7fa9c011e050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:39642/0 (socket says 172.21.15.73:39642) 2024-08-02T23:40:06.677 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:06.673+0000 7fa9bf7fe640 1 -- 172.21.15.73:0/2622173738 learned_addr learned my addr 172.21.15.73:0/2622173738 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-08-02T23:40:06.677 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:06.673+0000 7fa9bf7fe640 1 -- 172.21.15.73:0/2622173738 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9c009de70 con 0x7fa9c006e880 2024-08-02T23:40:06.677 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:06.674+0000 7fa9bf7fe640 1 --2- 172.21.15.73:0/2622173738 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa9c006e880 0x7fa9c011e050 crc :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-02T23:40:06.677 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:06.674+0000 7fa9a7fff640 1 -- 172.21.15.73:0/2622173738 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fa9b000e040 con 0x7fa9c006e880 2024-08-02T23:40:06.677 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:06.674+0000 7fa9a7fff640 1 -- 172.21.15.73:0/2622173738 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7fa9b0013aa0 con 0x7fa9c006e880 2024-08-02T23:40:06.677 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:06.674+0000 7fa9c5d03640 1 -- 172.21.15.73:0/2622173738 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9c012e860 con 0x7fa9c006e880 2024-08-02T23:40:06.678 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:06.674+0000 7fa9a7fff640 1 -- 172.21.15.73:0/2622173738 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fa9b0013d40 con 0x7fa9c006e880 2024-08-02T23:40:06.678 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:06.674+0000 7fa9c5d03640 1 -- 172.21.15.73:0/2622173738 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9c009d570 con 0x7fa9c006e880 2024-08-02T23:40:06.678 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:06.675+0000 7fa9a7fff640 1 -- 172.21.15.73:0/2622173738 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 5) ==== 45563+0+0 (crc 0 0 0) 0x7fa9b0011040 con 0x7fa9c006e880 2024-08-02T23:40:06.678 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:06.675+0000 7fa9c5d03640 1 -- 172.21.15.73:0/2622173738 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa9c0120f60 con 0x7fa9c006e880 2024-08-02T23:40:06.678 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:06.675+0000 7fa9a7fff640 1 --2- 172.21.15.73:0/2622173738 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7fa9ac038390 0x7fa9ac03a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-02T23:40:06.678 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:06.675+0000 7fa9a7fff640 1 -- 172.21.15.73:0/2622173738 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 2728+0+0 (crc 0 0 0) 0x7fa9b0049380 con 0x7fa9c006e880 2024-08-02T23:40:06.678 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:06.675+0000 7fa9beffd640 1 --2- 172.21.15.73:0/2622173738 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7fa9ac038390 0x7fa9ac03a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-02T23:40:06.678 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:06.676+0000 7fa9beffd640 1 --2- 172.21.15.73:0/2622173738 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7fa9ac038390 0x7fa9ac03a8a0 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-02T23:40:06.681 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:06.680+0000 7fa9a7fff640 1 -- 172.21.15.73:0/2622173738 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186747 (crc 0 0 0) 0x7fa9b0016e40 con 0x7fa9c006e880 2024-08-02T23:40:06.903 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:06.901+0000 7fa9c5d03640 1 -- 172.21.15.73:0/2622173738 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7fa9c0071b50 con 0x7fa9c006e880 2024-08-02T23:40:06.904 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:06.903+0000 7fa9a7fff640 1 -- 172.21.15.73:0/2622173738 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) ==== 74+0+87205 (crc 0 0 0) 0x7fa9b0013ea0 con 0x7fa9c006e880 2024-08-02T23:40:06.906 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-08-02T23:40:06.907 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:06.905+0000 7fa9c5d03640 1 -- 172.21.15.73:0/2622173738 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7fa9ac038390 msgr2=0x7fa9ac03a8a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-02T23:40:06.907 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:06.905+0000 7fa9c5d03640 1 --2- 172.21.15.73:0/2622173738 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7fa9ac038390 0x7fa9ac03a8a0 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:40:06.907 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:06.905+0000 7fa9c5d03640 1 -- 172.21.15.73:0/2622173738 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa9c006e880 msgr2=0x7fa9c011e050 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-02T23:40:06.907 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:06.905+0000 7fa9c5d03640 1 --2- 172.21.15.73:0/2622173738 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa9c006e880 0x7fa9c011e050 crc :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:40:06.908 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:06.905+0000 7fa9c5d03640 1 -- 172.21.15.73:0/2622173738 shutdown_connections 2024-08-02T23:40:06.908 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:06.905+0000 7fa9c5d03640 1 --2- 172.21.15.73:0/2622173738 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7fa9ac038390 0x7fa9ac03a8a0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:40:06.908 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:06.905+0000 7fa9c5d03640 1 --2- 172.21.15.73:0/2622173738 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa9c006e880 0x7fa9c011e050 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:40:06.908 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:06.905+0000 7fa9c5d03640 1 -- 172.21.15.73:0/2622173738 >> 172.21.15.73:0/2622173738 conn(0x7fa9c00a0590 msgr2=0x7fa9c009a9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-02T23:40:06.908 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:06.905+0000 7fa9c5d03640 1 -- 172.21.15.73:0/2622173738 shutdown_connections 2024-08-02T23:40:06.908 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:06.905+0000 7fa9c5d03640 1 -- 172.21.15.73:0/2622173738 wait complete. 2024-08-02T23:40:06.918 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":5,"flags":0,"active_gid":4100,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6800","nonce":3351532440},{"type":"v1","addr":"172.21.15.131:6801","nonce":3351532440}]},"active_addr":"172.21.15.131:6801/3351532440","active_change":"2024-08-02T23:39:57.363339+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.131:0","nonce":78288276}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.131:0","nonce":3715600886}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.131:0","nonce":1055630449}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.131:0","nonce":3186087968}]}]} 2024-08-02T23:40:06.920 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-08-02T23:40:06.920 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-08-02T23:40:06.920 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-08-02T23:40:07.024 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.019+0000 7f05cae6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-02T23:40:07.024 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.023+0000 7f05cae6d640 1 Processor -- start 2024-08-02T23:40:07.025 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.023+0000 7f05cae6d640 1 -- start start 2024-08-02T23:40:07.026 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.024+0000 7f05cae6d640 1 --2- >> v2:172.21.15.73:3300/0 conn(0x7f05c4132c10 0x7f05c4137690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-02T23:40:07.026 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.024+0000 7f05cae6d640 1 -- --> v1:172.21.15.73:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f05c4073a40 con 0x7f05c41320f0 2024-08-02T23:40:07.026 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.024+0000 7f05cae6d640 1 -- --> v2:172.21.15.73:3300/0 -- mon_getmap magic: 0 -- 0x7f05c4073810 con 0x7f05c4132c10 2024-08-02T23:40:07.026 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.024+0000 7f05bbfff640 1 --2- >> v2:172.21.15.73:3300/0 conn(0x7f05c4132c10 0x7f05c4137690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-02T23:40:07.026 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.024+0000 7f05c8c09640 1 --1- >> v1:172.21.15.73:6789/0 conn(0x7f05c41320f0 0x7f05c4132500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.73:6789/0 says I am v1:172.21.15.73:43352/0 (socket says 172.21.15.73:43352) 2024-08-02T23:40:07.026 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.024+0000 7f05c8c09640 1 -- 172.21.15.73:0/37102738 learned_addr learned my addr 172.21.15.73:0/37102738 (peer_addr_for_me v1:172.21.15.73:0/0) 2024-08-02T23:40:07.027 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.024+0000 7f05bbfff640 1 -- 172.21.15.73:0/37102738 >> v1:172.21.15.73:6789/0 conn(0x7f05c41320f0 legacy=0x7f05c4132500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-02T23:40:07.027 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.024+0000 7f05bbfff640 1 -- 172.21.15.73:0/37102738 --> v2:172.21.15.73:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05c409dfe0 con 0x7f05c4132c10 2024-08-02T23:40:07.027 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.024+0000 7f05bbfff640 1 --2- 172.21.15.73:0/37102738 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f05c4132c10 0x7f05c4137690 crc :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c691f05c144900 server_cookie=0 in_seq=0 out_seq=0 2024-08-02T23:40:07.027 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.024+0000 7f05bb7fe640 1 -- 172.21.15.73:0/37102738 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f05b4009ce0 con 0x7f05c4132c10 2024-08-02T23:40:07.027 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.024+0000 7f05bb7fe640 1 -- 172.21.15.73:0/37102738 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f05b4009e40 con 0x7f05c4132c10 2024-08-02T23:40:07.027 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.024+0000 7f05bb7fe640 1 -- 172.21.15.73:0/37102738 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f05b400e360 con 0x7f05c4132c10 2024-08-02T23:40:07.028 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.025+0000 7f05cae6d640 1 -- 172.21.15.73:0/37102738 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f05c4132c10 msgr2=0x7f05c4137690 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-02T23:40:07.028 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.025+0000 7f05cae6d640 1 --2- 172.21.15.73:0/37102738 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f05c4132c10 0x7f05c4137690 crc :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:40:07.028 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.025+0000 7f05cae6d640 1 -- 172.21.15.73:0/37102738 shutdown_connections 2024-08-02T23:40:07.028 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.025+0000 7f05cae6d640 1 --2- 172.21.15.73:0/37102738 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f05c4132c10 0x7f05c4137690 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:40:07.028 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.025+0000 7f05cae6d640 1 -- 172.21.15.73:0/37102738 >> 172.21.15.73:0/37102738 conn(0x7f05c40a0590 msgr2=0x7f05c40a09a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-02T23:40:07.028 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.025+0000 7f05cae6d640 1 -- 172.21.15.73:0/37102738 shutdown_connections 2024-08-02T23:40:07.029 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.025+0000 7f05cae6d640 1 -- 172.21.15.73:0/37102738 wait complete. 2024-08-02T23:40:07.029 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.025+0000 7f05cae6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-02T23:40:07.029 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.026+0000 7f05cae6d640 1 Processor -- start 2024-08-02T23:40:07.029 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.026+0000 7f05cae6d640 1 -- start start 2024-08-02T23:40:07.031 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.027+0000 7f05cae6d640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f05c41320f0 0x7f05c411a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-02T23:40:07.031 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.027+0000 7f05cae6d640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 -- 0x7f05c411df30 con 0x7f05c41320f0 2024-08-02T23:40:07.031 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.027+0000 7f05c8c09640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f05c41320f0 0x7f05c411a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-02T23:40:07.031 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.027+0000 7f05c8c09640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f05c41320f0 0x7f05c411a8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:44142/0 (socket says 172.21.15.73:44142) 2024-08-02T23:40:07.031 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.027+0000 7f05c8c09640 1 -- 172.21.15.73:0/1074514930 learned_addr learned my addr 172.21.15.73:0/1074514930 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-08-02T23:40:07.031 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.027+0000 7f05c8c09640 1 -- 172.21.15.73:0/1074514930 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05c411ce60 con 0x7f05c41320f0 2024-08-02T23:40:07.031 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.027+0000 7f05c8c09640 1 --2- 172.21.15.73:0/1074514930 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f05c41320f0 0x7f05c411a8b0 crc :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-02T23:40:07.031 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.028+0000 7f05b97fa640 1 -- 172.21.15.73:0/1074514930 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f05c00058a0 con 0x7f05c41320f0 2024-08-02T23:40:07.031 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.028+0000 7f05b97fa640 1 -- 172.21.15.73:0/1074514930 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f05c0005a00 con 0x7f05c41320f0 2024-08-02T23:40:07.031 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.028+0000 7f05cae6d640 1 -- 172.21.15.73:0/1074514930 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f05c412f080 con 0x7f05c41320f0 2024-08-02T23:40:07.032 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.028+0000 7f05b97fa640 1 -- 172.21.15.73:0/1074514930 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f05c0005ca0 con 0x7f05c41320f0 2024-08-02T23:40:07.032 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.028+0000 7f05cae6d640 1 -- 172.21.15.73:0/1074514930 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f05c411dc20 con 0x7f05c41320f0 2024-08-02T23:40:07.032 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.029+0000 7f05b97fa640 1 -- 172.21.15.73:0/1074514930 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 5) ==== 45563+0+0 (crc 0 0 0) 0x7f05c00070b0 con 0x7f05c41320f0 2024-08-02T23:40:07.032 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.029+0000 7f05cae6d640 1 -- 172.21.15.73:0/1074514930 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f05c411c540 con 0x7f05c41320f0 2024-08-02T23:40:07.032 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.029+0000 7f05b97fa640 1 --2- 172.21.15.73:0/1074514930 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7f05b0038390 0x7f05b003a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-02T23:40:07.032 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.029+0000 7f05b97fa640 1 -- 172.21.15.73:0/1074514930 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 2728+0+0 (crc 0 0 0) 0x7f05c0047af0 con 0x7f05c41320f0 2024-08-02T23:40:07.032 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.029+0000 7f05bbfff640 1 --2- 172.21.15.73:0/1074514930 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7f05b0038390 0x7f05b003a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-02T23:40:07.033 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.030+0000 7f05bbfff640 1 --2- 172.21.15.73:0/1074514930 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7f05b0038390 0x7f05b003a8a0 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-02T23:40:07.035 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.034+0000 7f05b97fa640 1 -- 172.21.15.73:0/1074514930 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186747 (crc 0 0 0) 0x7f05c000da70 con 0x7f05c41320f0 2024-08-02T23:40:07.212 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.211+0000 7f05cae6d640 1 -- 172.21.15.73:0/1074514930 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f05c4132500 con 0x7f05c41320f0 2024-08-02T23:40:07.213 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.211+0000 7f05b97fa640 1 -- 172.21.15.73:0/1074514930 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) ==== 74+0+8836 (crc 0 0 0) 0x7f05c000d660 con 0x7f05c41320f0 2024-08-02T23:40:07.213 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-08-02T23:40:07.213 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":16,"fsid":"7caeb70d-fd31-4bdf-94e1-d0f19ea8e3c6","created":"2024-08-02T23:39:52.389003+0000","modified":"2024-08-02T23:40:06.519283+0000","last_up_change":"2024-08-02T23:39:59.483999+0000","last_in_change":"2024-08-02T23:39:54.500028+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-02T23:39:59.404407+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-08-02T23:40:02.628150+0000","flags":270369,"flags_names":"hashpspool,nopgchange,selfmanaged_snaps,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":16,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bd37ae5d-55cc-4a6d-b943-8de7e40fd1bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6801","nonce":4128208498}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6800","nonce":4128208498}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6808","nonce":4128208498}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6805","nonce":4128208498}]},"public_addr":"172.21.15.73:6801/4128208498","cluster_addr":"172.21.15.73:6800/4128208498","heartbeat_back_addr":"172.21.15.73:6808/4128208498","heartbeat_front_addr":"172.21.15.73:6805/4128208498","state":["exists","up"]},{"osd":1,"uuid":"6bf29a4c-3240-4fce-8803-66926d0df50c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6804","nonce":3612805766}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6803","nonce":3612805766}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6807","nonce":3612805766}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6806","nonce":3612805766}]},"public_addr":"172.21.15.73:6804/3612805766","cluster_addr":"172.21.15.73:6803/3612805766","heartbeat_back_addr":"172.21.15.73:6807/3612805766","heartbeat_front_addr":"172.21.15.73:6806/3612805766","state":["exists","up"]},{"osd":2,"uuid":"7d855822-09de-49f2-8811-c47826c38d5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6802","nonce":3558187202}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6803","nonce":3558187202}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6811","nonce":3558187202}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6810","nonce":3558187202}]},"public_addr":"172.21.15.131:6802/3558187202","cluster_addr":"172.21.15.131:6803/3558187202","heartbeat_back_addr":"172.21.15.131:6811/3558187202","heartbeat_front_addr":"172.21.15.131:6810/3558187202","state":["exists","up"]},{"osd":3,"uuid":"d6e6bd88-9a90-441c-8bbd-8b8d4a2b4f9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6809","nonce":1209903557}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6808","nonce":1209903557}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6813","nonce":1209903557}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6812","nonce":1209903557}]},"public_addr":"172.21.15.131:6809/1209903557","cluster_addr":"172.21.15.131:6808/1209903557","heartbeat_back_addr":"172.21.15.131:6813/1209903557","heartbeat_front_addr":"172.21.15.131:6812/1209903557","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-02T23:40:07.216 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.214+0000 7f05cae6d640 1 -- 172.21.15.73:0/1074514930 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7f05b0038390 msgr2=0x7f05b003a8a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-02T23:40:07.216 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.214+0000 7f05cae6d640 1 --2- 172.21.15.73:0/1074514930 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7f05b0038390 0x7f05b003a8a0 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:40:07.216 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.214+0000 7f05cae6d640 1 -- 172.21.15.73:0/1074514930 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f05c41320f0 msgr2=0x7f05c411a8b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-02T23:40:07.216 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.214+0000 7f05cae6d640 1 --2- 172.21.15.73:0/1074514930 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f05c41320f0 0x7f05c411a8b0 crc :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:40:07.216 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.214+0000 7f05cae6d640 1 -- 172.21.15.73:0/1074514930 shutdown_connections 2024-08-02T23:40:07.216 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.214+0000 7f05cae6d640 1 --2- 172.21.15.73:0/1074514930 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7f05b0038390 0x7f05b003a8a0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:40:07.216 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.214+0000 7f05cae6d640 1 --2- 172.21.15.73:0/1074514930 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f05c41320f0 0x7f05c411a8b0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:40:07.216 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.214+0000 7f05cae6d640 1 -- 172.21.15.73:0/1074514930 >> 172.21.15.73:0/1074514930 conn(0x7f05c40a0590 msgr2=0x7f05c41ba390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-02T23:40:07.217 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.214+0000 7f05cae6d640 1 -- 172.21.15.73:0/1074514930 shutdown_connections 2024-08-02T23:40:07.217 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.214+0000 7f05cae6d640 1 -- 172.21.15.73:0/1074514930 wait complete. 2024-08-02T23:40:07.226 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-08-02T23:40:07.226 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-08-02T23:40:07.331 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.326+0000 7fcadf421640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-02T23:40:07.332 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.330+0000 7fcadf421640 1 Processor -- start 2024-08-02T23:40:07.332 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.330+0000 7fcadf421640 1 -- start start 2024-08-02T23:40:07.333 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.331+0000 7fcadf421640 1 --2- >> v2:172.21.15.73:3300/0 conn(0x7fcad806e880 0x7fcad806ec90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-02T23:40:07.333 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.331+0000 7fcadf421640 1 -- --> v1:172.21.15.73:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcad806fef0 con 0x7fcad806f1d0 2024-08-02T23:40:07.333 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.331+0000 7fcadf421640 1 -- --> v2:172.21.15.73:3300/0 -- mon_getmap magic: 0 -- 0x7fcad809e5d0 con 0x7fcad806e880 2024-08-02T23:40:07.334 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.331+0000 7fcadd1bd640 1 --2- >> v2:172.21.15.73:3300/0 conn(0x7fcad806e880 0x7fcad806ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-02T23:40:07.334 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.331+0000 7fcadc9bc640 1 --1- >> v1:172.21.15.73:6789/0 conn(0x7fcad806f1d0 0x7fcad8131040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.73:6789/0 says I am v1:172.21.15.73:43368/0 (socket says 172.21.15.73:43368) 2024-08-02T23:40:07.334 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.331+0000 7fcadc9bc640 1 -- 172.21.15.73:0/1435912814 learned_addr learned my addr 172.21.15.73:0/1435912814 (peer_addr_for_me v1:172.21.15.73:0/0) 2024-08-02T23:40:07.334 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.331+0000 7fcacffff640 1 -- 172.21.15.73:0/1435912814 <== mon.0 v1:172.21.15.73:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 580820643 0 0) 0x7fcad806fef0 con 0x7fcad806f1d0 2024-08-02T23:40:07.334 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.331+0000 7fcacffff640 1 -- 172.21.15.73:0/1435912814 --> v1:172.21.15.73:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcac40034f0 con 0x7fcad806f1d0 2024-08-02T23:40:07.334 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.331+0000 7fcadd1bd640 1 -- 172.21.15.73:0/1435912814 >> v1:172.21.15.73:6789/0 conn(0x7fcad806f1d0 legacy=0x7fcad8131040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-02T23:40:07.334 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.331+0000 7fcadd1bd640 1 -- 172.21.15.73:0/1435912814 --> v2:172.21.15.73:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcad806e430 con 0x7fcad806e880 2024-08-02T23:40:07.334 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.331+0000 7fcadd1bd640 1 --2- 172.21.15.73:0/1435912814 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcad806e880 0x7fcad806ec90 crc :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78542ededc536d22 server_cookie=0 in_seq=0 out_seq=0 2024-08-02T23:40:07.334 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.331+0000 7fcacffff640 1 -- 172.21.15.73:0/1435912814 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fcad4009ce0 con 0x7fcad806e880 2024-08-02T23:40:07.334 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.332+0000 7fcacffff640 1 -- 172.21.15.73:0/1435912814 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7fcad4009e40 con 0x7fcad806e880 2024-08-02T23:40:07.334 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.332+0000 7fcacffff640 1 -- 172.21.15.73:0/1435912814 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fcad400e360 con 0x7fcad806e880 2024-08-02T23:40:07.335 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.332+0000 7fcadf421640 1 -- 172.21.15.73:0/1435912814 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcad806e880 msgr2=0x7fcad806ec90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-02T23:40:07.335 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.332+0000 7fcadf421640 1 --2- 172.21.15.73:0/1435912814 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcad806e880 0x7fcad806ec90 crc :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:40:07.335 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.332+0000 7fcadf421640 1 -- 172.21.15.73:0/1435912814 shutdown_connections 2024-08-02T23:40:07.336 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.332+0000 7fcadf421640 1 --2- 172.21.15.73:0/1435912814 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcad806e880 0x7fcad806ec90 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:40:07.336 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.332+0000 7fcadf421640 1 -- 172.21.15.73:0/1435912814 >> 172.21.15.73:0/1435912814 conn(0x7fcad80a0590 msgr2=0x7fcad80a09a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-02T23:40:07.336 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.332+0000 7fcadf421640 1 -- 172.21.15.73:0/1435912814 shutdown_connections 2024-08-02T23:40:07.336 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.332+0000 7fcadf421640 1 -- 172.21.15.73:0/1435912814 wait complete. 2024-08-02T23:40:07.336 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.333+0000 7fcadf421640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-02T23:40:07.336 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.333+0000 7fcadf421640 1 Processor -- start 2024-08-02T23:40:07.336 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.333+0000 7fcadf421640 1 -- start start 2024-08-02T23:40:07.338 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.334+0000 7fcadf421640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcad806e880 0x7fcad811ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-02T23:40:07.338 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.334+0000 7fcadf421640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 -- 0x7fcad812da80 con 0x7fcad806e880 2024-08-02T23:40:07.338 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.334+0000 7fcadd1bd640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcad806e880 0x7fcad811ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-02T23:40:07.338 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.334+0000 7fcadd1bd640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcad806e880 0x7fcad811ee40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:44158/0 (socket says 172.21.15.73:44158) 2024-08-02T23:40:07.338 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.334+0000 7fcadd1bd640 1 -- 172.21.15.73:0/1430221796 learned_addr learned my addr 172.21.15.73:0/1430221796 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-08-02T23:40:07.338 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.334+0000 7fcadd1bd640 1 -- 172.21.15.73:0/1430221796 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcad8120fc0 con 0x7fcad806e880 2024-08-02T23:40:07.338 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.334+0000 7fcadd1bd640 1 --2- 172.21.15.73:0/1430221796 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcad806e880 0x7fcad811ee40 crc :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-02T23:40:07.338 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.335+0000 7fcacdffb640 1 -- 172.21.15.73:0/1430221796 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fcad4014410 con 0x7fcad806e880 2024-08-02T23:40:07.338 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.335+0000 7fcacdffb640 1 -- 172.21.15.73:0/1430221796 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7fcad4014570 con 0x7fcad806e880 2024-08-02T23:40:07.338 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.335+0000 7fcacdffb640 1 -- 172.21.15.73:0/1430221796 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fcad4014810 con 0x7fcad806e880 2024-08-02T23:40:07.339 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.335+0000 7fcadf421640 1 -- 172.21.15.73:0/1430221796 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcad809de10 con 0x7fcad806e880 2024-08-02T23:40:07.339 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.335+0000 7fcadf421640 1 -- 172.21.15.73:0/1430221796 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcad812dc70 con 0x7fcad806e880 2024-08-02T23:40:07.339 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.336+0000 7fcacdffb640 1 -- 172.21.15.73:0/1430221796 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 5) ==== 45563+0+0 (crc 0 0 0) 0x7fcad40149c0 con 0x7fcad806e880 2024-08-02T23:40:07.339 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.336+0000 7fcacdffb640 1 --2- 172.21.15.73:0/1430221796 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7fcabc038390 0x7fcabc03a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-02T23:40:07.339 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.336+0000 7fcacdffb640 1 -- 172.21.15.73:0/1430221796 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 2728+0+0 (crc 0 0 0) 0x7fcad4049200 con 0x7fcad806e880 2024-08-02T23:40:07.339 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.336+0000 7fcadf421640 1 -- 172.21.15.73:0/1430221796 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcad806ec90 con 0x7fcad806e880 2024-08-02T23:40:07.339 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.336+0000 7fcadc9bc640 1 --2- 172.21.15.73:0/1430221796 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7fcabc038390 0x7fcabc03a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-02T23:40:07.339 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.337+0000 7fcadc9bc640 1 --2- 172.21.15.73:0/1430221796 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7fcabc038390 0x7fcabc03a8a0 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-02T23:40:07.342 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.341+0000 7fcacdffb640 1 -- 172.21.15.73:0/1430221796 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186747 (crc 0 0 0) 0x7fcad400f990 con 0x7fcad806e880 2024-08-02T23:40:07.518 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.517+0000 7fcadf421640 1 -- 172.21.15.73:0/1430221796 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fcad806bcc0 con 0x7fcad806e880 2024-08-02T23:40:07.519 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.517+0000 7fcacdffb640 1 -- 172.21.15.73:0/1430221796 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) ==== 74+0+8836 (crc 0 0 0) 0x7fcad400f2a0 con 0x7fcad806e880 2024-08-02T23:40:07.519 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-08-02T23:40:07.519 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":16,"fsid":"7caeb70d-fd31-4bdf-94e1-d0f19ea8e3c6","created":"2024-08-02T23:39:52.389003+0000","modified":"2024-08-02T23:40:06.519283+0000","last_up_change":"2024-08-02T23:39:59.483999+0000","last_in_change":"2024-08-02T23:39:54.500028+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-02T23:39:59.404407+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-08-02T23:40:02.628150+0000","flags":270369,"flags_names":"hashpspool,nopgchange,selfmanaged_snaps,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":16,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bd37ae5d-55cc-4a6d-b943-8de7e40fd1bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6801","nonce":4128208498}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6800","nonce":4128208498}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6808","nonce":4128208498}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6805","nonce":4128208498}]},"public_addr":"172.21.15.73:6801/4128208498","cluster_addr":"172.21.15.73:6800/4128208498","heartbeat_back_addr":"172.21.15.73:6808/4128208498","heartbeat_front_addr":"172.21.15.73:6805/4128208498","state":["exists","up"]},{"osd":1,"uuid":"6bf29a4c-3240-4fce-8803-66926d0df50c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6804","nonce":3612805766}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6803","nonce":3612805766}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6807","nonce":3612805766}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6806","nonce":3612805766}]},"public_addr":"172.21.15.73:6804/3612805766","cluster_addr":"172.21.15.73:6803/3612805766","heartbeat_back_addr":"172.21.15.73:6807/3612805766","heartbeat_front_addr":"172.21.15.73:6806/3612805766","state":["exists","up"]},{"osd":2,"uuid":"7d855822-09de-49f2-8811-c47826c38d5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6802","nonce":3558187202}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6803","nonce":3558187202}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6811","nonce":3558187202}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6810","nonce":3558187202}]},"public_addr":"172.21.15.131:6802/3558187202","cluster_addr":"172.21.15.131:6803/3558187202","heartbeat_back_addr":"172.21.15.131:6811/3558187202","heartbeat_front_addr":"172.21.15.131:6810/3558187202","state":["exists","up"]},{"osd":3,"uuid":"d6e6bd88-9a90-441c-8bbd-8b8d4a2b4f9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6809","nonce":1209903557}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6808","nonce":1209903557}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6813","nonce":1209903557}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6812","nonce":1209903557}]},"public_addr":"172.21.15.131:6809/1209903557","cluster_addr":"172.21.15.131:6808/1209903557","heartbeat_back_addr":"172.21.15.131:6813/1209903557","heartbeat_front_addr":"172.21.15.131:6812/1209903557","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-02T23:40:07.522 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.519+0000 7fcadf421640 1 -- 172.21.15.73:0/1430221796 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7fcabc038390 msgr2=0x7fcabc03a8a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-02T23:40:07.522 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.519+0000 7fcadf421640 1 --2- 172.21.15.73:0/1430221796 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7fcabc038390 0x7fcabc03a8a0 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:40:07.522 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.519+0000 7fcadf421640 1 -- 172.21.15.73:0/1430221796 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcad806e880 msgr2=0x7fcad811ee40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-02T23:40:07.522 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.519+0000 7fcadf421640 1 --2- 172.21.15.73:0/1430221796 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcad806e880 0x7fcad811ee40 crc :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:40:07.522 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.519+0000 7fcadf421640 1 -- 172.21.15.73:0/1430221796 shutdown_connections 2024-08-02T23:40:07.522 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.519+0000 7fcadf421640 1 --2- 172.21.15.73:0/1430221796 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7fcabc038390 0x7fcabc03a8a0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:40:07.522 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.519+0000 7fcadf421640 1 --2- 172.21.15.73:0/1430221796 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcad806e880 0x7fcad811ee40 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:40:07.522 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.519+0000 7fcadf421640 1 -- 172.21.15.73:0/1430221796 >> 172.21.15.73:0/1430221796 conn(0x7fcad80a0590 msgr2=0x7fcad809a9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-02T23:40:07.522 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.520+0000 7fcadf421640 1 -- 172.21.15.73:0/1430221796 shutdown_connections 2024-08-02T23:40:07.522 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.520+0000 7fcadf421640 1 -- 172.21.15.73:0/1430221796 wait complete. 2024-08-02T23:40:07.533 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-08-02T23:40:07.533 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-08-02T23:40:07.534 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-08-02T23:40:07.534 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-08-02T23:40:07.649 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.642+0000 7f2b9e26d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-02T23:40:07.649 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.642+0000 7fb8b4d99640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-02T23:40:07.649 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.642+0000 7f056ab59640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-02T23:40:07.650 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.643+0000 7f2489d99640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-02T23:40:07.650 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.647+0000 7fb8b4d99640 1 Processor -- start 2024-08-02T23:40:07.650 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.647+0000 7fb8b4d99640 1 -- start start 2024-08-02T23:40:07.650 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.648+0000 7f056ab59640 1 Processor -- start 2024-08-02T23:40:07.651 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.648+0000 7f2b9e26d640 1 Processor -- start 2024-08-02T23:40:07.651 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.648+0000 7f2b9e26d640 1 -- start start 2024-08-02T23:40:07.651 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.648+0000 7f056ab59640 1 -- start start 2024-08-02T23:40:07.651 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.648+0000 7f2489d99640 1 Processor -- start 2024-08-02T23:40:07.652 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.648+0000 7f2489d99640 1 -- start start 2024-08-02T23:40:07.652 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.648+0000 7f2b9e26d640 1 --2- >> v2:172.21.15.73:3300/0 conn(0x7f2b980a4770 0x7f2b980a4b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-02T23:40:07.652 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.648+0000 7f056ab59640 1 --2- >> v2:172.21.15.73:3300/0 conn(0x7f05641320f0 0x7f0564132500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-02T23:40:07.652 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.648+0000 7f056ab59640 1 -- --> v1:172.21.15.73:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0564073a40 con 0x7f0564132a40 2024-08-02T23:40:07.652 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.648+0000 7f056ab59640 1 -- --> v2:172.21.15.73:3300/0 -- mon_getmap magic: 0 -- 0x7f0564073810 con 0x7f05641320f0 2024-08-02T23:40:07.653 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.649+0000 7fb8b4d99640 1 --2- >> v2:172.21.15.73:3300/0 conn(0x7fb8b01339f0 0x7fb8b0135e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-02T23:40:07.653 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.649+0000 7f2b9e26d640 1 -- --> v1:172.21.15.73:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2b9806e030 con 0x7f2b980a50c0 2024-08-02T23:40:07.653 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.649+0000 7f2b9e26d640 1 -- --> v2:172.21.15.73:3300/0 -- mon_getmap magic: 0 -- 0x7f2b980a0ed0 con 0x7f2b980a4770 2024-08-02T23:40:07.653 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.649+0000 7f2b97fff640 1 --2- >> v2:172.21.15.73:3300/0 conn(0x7f2b980a4770 0x7f2b980a4b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-02T23:40:07.654 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.649+0000 7fb8b4d99640 1 -- --> v1:172.21.15.73:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb8b006e4f0 con 0x7fb8b012d690 2024-08-02T23:40:07.654 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.649+0000 7fb8b4d99640 1 -- --> v2:172.21.15.73:3300/0 -- mon_getmap magic: 0 -- 0x7fb8b012d240 con 0x7fb8b01339f0 2024-08-02T23:40:07.654 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.649+0000 7f2b977fe640 1 --1- >> v1:172.21.15.73:6789/0 conn(0x7f2b980a50c0 0x7f2b98131000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.73:6789/0 says I am v1:172.21.15.73:43380/0 (socket says 172.21.15.73:43380) 2024-08-02T23:40:07.654 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.649+0000 7f2b977fe640 1 -- 172.21.15.73:0/4265773825 learned_addr learned my addr 172.21.15.73:0/4265773825 (peer_addr_for_me v1:172.21.15.73:0/0) 2024-08-02T23:40:07.655 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.649+0000 7fb8ae59c640 1 --1- >> v1:172.21.15.73:6789/0 conn(0x7fb8b012d690 0x7fb8b012daa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.73:6789/0 says I am v1:172.21.15.73:43398/0 (socket says 172.21.15.73:43398) 2024-08-02T23:40:07.655 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.649+0000 7fb8ae59c640 1 -- 172.21.15.73:0/3792731387 learned_addr learned my addr 172.21.15.73:0/3792731387 (peer_addr_for_me v1:172.21.15.73:0/0) 2024-08-02T23:40:07.655 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.649+0000 7f05688f5640 1 --2- >> v2:172.21.15.73:3300/0 conn(0x7f05641320f0 0x7f0564132500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-02T23:40:07.655 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.649+0000 7f05688f5640 1 --2- >> v2:172.21.15.73:3300/0 conn(0x7f05641320f0 0x7f0564132500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:44164/0 (socket says 172.21.15.73:44164) 2024-08-02T23:40:07.655 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.649+0000 7f05688f5640 1 -- 172.21.15.73:0/1713003468 learned_addr learned my addr 172.21.15.73:0/1713003468 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-08-02T23:40:07.655 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.649+0000 7fb8add9b640 1 --2- 172.21.15.73:0/3792731387 >> v2:172.21.15.73:3300/0 conn(0x7fb8b01339f0 0x7fb8b0135e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-02T23:40:07.656 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.649+0000 7f2b97fff640 1 -- 172.21.15.73:0/4265773825 >> v1:172.21.15.73:6789/0 conn(0x7f2b980a50c0 legacy=0x7f2b98131000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-02T23:40:07.656 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.649+0000 7f2b96ffd640 1 -- 172.21.15.73:0/4265773825 <== mon.0 v1:172.21.15.73:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3944424843 0 0) 0x7f2b9806e030 con 0x7f2b980a50c0 2024-08-02T23:40:07.656 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.649+0000 7f2b97fff640 1 -- 172.21.15.73:0/4265773825 --> v2:172.21.15.73:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b9806e260 con 0x7f2b980a4770 2024-08-02T23:40:07.656 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.649+0000 7f2489d99640 1 --2- >> v2:172.21.15.73:3300/0 conn(0x7f2484131b90 0x7f2484139930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-02T23:40:07.657 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.649+0000 7fb8add9b640 1 -- 172.21.15.73:0/3792731387 >> v1:172.21.15.73:6789/0 conn(0x7fb8b012d690 legacy=0x7fb8b012daa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-02T23:40:07.657 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.649+0000 7f2b97fff640 1 --2- 172.21.15.73:0/4265773825 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2b980a4770 0x7f2b980a4b80 crc :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ce15e390c5dbcef server_cookie=0 in_seq=0 out_seq=0 2024-08-02T23:40:07.657 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.649+0000 7fb8ad59a640 1 -- 172.21.15.73:0/3792731387 <== mon.0 v1:172.21.15.73:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3075688744 0 0) 0x7fb8b006e4f0 con 0x7fb8b012d690 2024-08-02T23:40:07.657 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.649+0000 7fb8add9b640 1 -- 172.21.15.73:0/3792731387 --> v2:172.21.15.73:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8b009e870 con 0x7fb8b01339f0 2024-08-02T23:40:07.657 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.649+0000 7f2489d99640 1 -- --> v1:172.21.15.73:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f248406df40 con 0x7f2484131070 2024-08-02T23:40:07.658 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.649+0000 7fb8add9b640 1 --2- 172.21.15.73:0/3792731387 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb8b01339f0 0x7fb8b0135e10 crc :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83f9f7c0900b7210 server_cookie=0 in_seq=0 out_seq=0 2024-08-02T23:40:07.658 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.650+0000 7f2489d99640 1 -- --> v2:172.21.15.73:3300/0 -- mon_getmap magic: 0 -- 0x7f24840a0e40 con 0x7f2484131b90 2024-08-02T23:40:07.658 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.650+0000 7f2b96ffd640 1 -- 172.21.15.73:0/4265773825 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f2b88009ce0 con 0x7f2b980a4770 2024-08-02T23:40:07.659 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.650+0000 7f2482ffd640 1 --2- >> v2:172.21.15.73:3300/0 conn(0x7f2484131b90 0x7f2484139930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-02T23:40:07.659 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.650+0000 7f2482ffd640 1 --2- >> v2:172.21.15.73:3300/0 conn(0x7f2484131b90 0x7f2484139930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:44192/0 (socket says 172.21.15.73:44192) 2024-08-02T23:40:07.659 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.650+0000 7f05688f5640 1 -- 172.21.15.73:0/1713003468 >> v1:172.21.15.73:6789/0 conn(0x7f0564132a40 legacy=0x7f056412ee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-02T23:40:07.659 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.650+0000 7f05688f5640 1 -- 172.21.15.73:0/1713003468 --> v2:172.21.15.73:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f056409dfe0 con 0x7f05641320f0 2024-08-02T23:40:07.659 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.650+0000 7f2482ffd640 1 -- 172.21.15.73:0/3208843728 learned_addr learned my addr 172.21.15.73:0/3208843728 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-08-02T23:40:07.660 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.650+0000 7f2b96ffd640 1 -- 172.21.15.73:0/4265773825 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f2b88009e40 con 0x7f2b980a4770 2024-08-02T23:40:07.660 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.650+0000 7f2b96ffd640 1 -- 172.21.15.73:0/4265773825 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f2b8800e360 con 0x7f2b980a4770 2024-08-02T23:40:07.660 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.650+0000 7fb8ad59a640 1 -- 172.21.15.73:0/3792731387 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fb898009ce0 con 0x7fb8b01339f0 2024-08-02T23:40:07.660 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.650+0000 7fb8ad59a640 1 -- 172.21.15.73:0/3792731387 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7fb898009e40 con 0x7fb8b01339f0 2024-08-02T23:40:07.660 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.650+0000 7fb8ad59a640 1 -- 172.21.15.73:0/3792731387 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fb89800e360 con 0x7fb8b01339f0 2024-08-02T23:40:07.661 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.650+0000 7f05688f5640 1 --2- 172.21.15.73:0/1713003468 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f05641320f0 0x7f0564132500 crc :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cebc34039108380f server_cookie=0 in_seq=0 out_seq=0 2024-08-02T23:40:07.661 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.650+0000 7f2482ffd640 1 -- 172.21.15.73:0/3208843728 >> v1:172.21.15.73:6789/0 conn(0x7f2484131070 legacy=0x7f2484131480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-02T23:40:07.661 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.650+0000 7f2482ffd640 1 -- 172.21.15.73:0/3208843728 --> v2:172.21.15.73:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f248406dc30 con 0x7f2484131b90 2024-08-02T23:40:07.661 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.650+0000 7f05637fe640 1 -- 172.21.15.73:0/1713003468 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f0558009dd0 con 0x7f05641320f0 2024-08-02T23:40:07.661 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.650+0000 7f05637fe640 1 -- 172.21.15.73:0/1713003468 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f055800e230 con 0x7f05641320f0 2024-08-02T23:40:07.661 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.650+0000 7f05637fe640 1 -- 172.21.15.73:0/1713003468 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f055800e4d0 con 0x7f05641320f0 2024-08-02T23:40:07.662 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.650+0000 7f2482ffd640 1 --2- 172.21.15.73:0/3208843728 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2484131b90 0x7f2484139930 crc :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc4c75534e271719 server_cookie=0 in_seq=0 out_seq=0 2024-08-02T23:40:07.662 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.650+0000 7f24827fc640 1 -- 172.21.15.73:0/3208843728 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f2478009dd0 con 0x7f2484131b90 2024-08-02T23:40:07.662 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.650+0000 7f24827fc640 1 -- 172.21.15.73:0/3208843728 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f247800e230 con 0x7f2484131b90 2024-08-02T23:40:07.662 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.650+0000 7f24827fc640 1 -- 172.21.15.73:0/3208843728 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f247800e4d0 con 0x7f2484131b90 2024-08-02T23:40:07.663 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.651+0000 7f2b9e26d640 1 -- 172.21.15.73:0/4265773825 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2b980a4770 msgr2=0x7f2b980a4b80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-02T23:40:07.663 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.651+0000 7f2b9e26d640 1 --2- 172.21.15.73:0/4265773825 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2b980a4770 0x7f2b980a4b80 crc :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:40:07.663 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.651+0000 7f2b9e26d640 1 -- 172.21.15.73:0/4265773825 shutdown_connections 2024-08-02T23:40:07.663 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.651+0000 7f2b9e26d640 1 --2- 172.21.15.73:0/4265773825 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2b980a4770 0x7f2b980a4b80 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:40:07.663 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.651+0000 7f2b9e26d640 1 -- 172.21.15.73:0/4265773825 >> 172.21.15.73:0/4265773825 conn(0x7f2b980a3510 msgr2=0x7f2b980a3920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-02T23:40:07.663 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.651+0000 7fb8b4d99640 1 -- 172.21.15.73:0/3792731387 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb8b01339f0 msgr2=0x7fb8b0135e10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-02T23:40:07.663 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.651+0000 7fb8b4d99640 1 --2- 172.21.15.73:0/3792731387 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb8b01339f0 0x7fb8b0135e10 crc :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:40:07.664 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.651+0000 7f2b9e26d640 1 -- 172.21.15.73:0/4265773825 shutdown_connections 2024-08-02T23:40:07.664 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.651+0000 7f2b9e26d640 1 -- 172.21.15.73:0/4265773825 wait complete. 2024-08-02T23:40:07.664 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.651+0000 7f056ab59640 1 -- 172.21.15.73:0/1713003468 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f05641320f0 msgr2=0x7f0564132500 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-02T23:40:07.664 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.651+0000 7f056ab59640 1 --2- 172.21.15.73:0/1713003468 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f05641320f0 0x7f0564132500 crc :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:40:07.665 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.651+0000 7fb8b4d99640 1 -- 172.21.15.73:0/3792731387 shutdown_connections 2024-08-02T23:40:07.665 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.651+0000 7fb8b4d99640 1 --2- 172.21.15.73:0/3792731387 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb8b01339f0 0x7fb8b0135e10 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:40:07.665 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.651+0000 7f056ab59640 1 -- 172.21.15.73:0/1713003468 shutdown_connections 2024-08-02T23:40:07.665 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.651+0000 7f056ab59640 1 --2- 172.21.15.73:0/1713003468 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f05641320f0 0x7f0564132500 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:40:07.665 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.651+0000 7fb8b4d99640 1 -- 172.21.15.73:0/3792731387 >> 172.21.15.73:0/3792731387 conn(0x7fb8b00a0590 msgr2=0x7fb8b00a09a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-02T23:40:07.666 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.651+0000 7f056ab59640 1 -- 172.21.15.73:0/1713003468 >> 172.21.15.73:0/1713003468 conn(0x7f05640a0590 msgr2=0x7f05640a09a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-02T23:40:07.666 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.651+0000 7f056ab59640 1 -- 172.21.15.73:0/1713003468 shutdown_connections 2024-08-02T23:40:07.666 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.651+0000 7fb8b4d99640 1 -- 172.21.15.73:0/3792731387 shutdown_connections 2024-08-02T23:40:07.666 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.651+0000 7f056ab59640 1 -- 172.21.15.73:0/1713003468 wait complete. 2024-08-02T23:40:07.667 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.651+0000 7f2b9e26d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-02T23:40:07.667 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.651+0000 7fb8b4d99640 1 -- 172.21.15.73:0/3792731387 wait complete. 2024-08-02T23:40:07.667 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.651+0000 7f2489d99640 1 -- 172.21.15.73:0/3208843728 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2484131b90 msgr2=0x7f2484139930 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-02T23:40:07.667 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.651+0000 7f2489d99640 1 --2- 172.21.15.73:0/3208843728 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2484131b90 0x7f2484139930 crc :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:40:07.667 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.651+0000 7f056ab59640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-02T23:40:07.668 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.651+0000 7f2489d99640 1 -- 172.21.15.73:0/3208843728 shutdown_connections 2024-08-02T23:40:07.668 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.651+0000 7f2489d99640 1 --2- 172.21.15.73:0/3208843728 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2484131b90 0x7f2484139930 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:40:07.668 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.651+0000 7f2489d99640 1 -- 172.21.15.73:0/3208843728 >> 172.21.15.73:0/3208843728 conn(0x7f248406cf90 msgr2=0x7f248406d3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-02T23:40:07.668 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.651+0000 7f2489d99640 1 -- 172.21.15.73:0/3208843728 shutdown_connections 2024-08-02T23:40:07.668 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.651+0000 7f2489d99640 1 -- 172.21.15.73:0/3208843728 wait complete. 2024-08-02T23:40:07.669 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.652+0000 7fb8b4d99640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-02T23:40:07.669 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.652+0000 7f2489d99640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-02T23:40:07.669 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.652+0000 7f2b9e26d640 1 Processor -- start 2024-08-02T23:40:07.669 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.652+0000 7f056ab59640 1 Processor -- start 2024-08-02T23:40:07.669 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.652+0000 7f056ab59640 1 -- start start 2024-08-02T23:40:07.670 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.652+0000 7f2b9e26d640 1 -- start start 2024-08-02T23:40:07.670 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.653+0000 7f2489d99640 1 Processor -- start 2024-08-02T23:40:07.670 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.653+0000 7fb8b4d99640 1 Processor -- start 2024-08-02T23:40:07.671 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.653+0000 7f2489d99640 1 -- start start 2024-08-02T23:40:07.671 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.653+0000 7fb8b4d99640 1 -- start start 2024-08-02T23:40:07.671 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.653+0000 7f2b9e26d640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2b980a4770 0x7f2b98120b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-02T23:40:07.671 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.653+0000 7f2b9e26d640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 -- 0x7f2b98121f90 con 0x7f2b980a4770 2024-08-02T23:40:07.671 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.653+0000 7f056ab59640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f0564132a40 0x7f056411aad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-02T23:40:07.671 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.653+0000 7f056ab59640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 -- 0x7f056411d6e0 con 0x7f0564132a40 2024-08-02T23:40:07.672 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.653+0000 7f2b97fff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2b980a4770 0x7f2b98120b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-02T23:40:07.672 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.653+0000 7f2b97fff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2b980a4770 0x7f2b98120b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:44196/0 (socket says 172.21.15.73:44196) 2024-08-02T23:40:07.672 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.653+0000 7f2b97fff640 1 -- 172.21.15.73:0/918785841 learned_addr learned my addr 172.21.15.73:0/918785841 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-08-02T23:40:07.673 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.653+0000 7f05688f5640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f0564132a40 0x7f056411aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-02T23:40:07.673 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.653+0000 7f05688f5640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f0564132a40 0x7f056411aad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:44198/0 (socket says 172.21.15.73:44198) 2024-08-02T23:40:07.673 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.653+0000 7f05688f5640 1 -- 172.21.15.73:0/1256299241 learned_addr learned my addr 172.21.15.73:0/1256299241 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-08-02T23:40:07.673 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.653+0000 7f2489d99640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2484131070 0x7f248411dd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-02T23:40:07.673 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.653+0000 7f2489d99640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 -- 0x7f248411f910 con 0x7f2484131070 2024-08-02T23:40:07.673 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.654+0000 7fb8b4d99640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb8b012d690 0x7fb8b011b490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-02T23:40:07.674 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.654+0000 7fb8b4d99640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 -- 0x7fb8b006efd0 con 0x7fb8b012d690 2024-08-02T23:40:07.674 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.654+0000 7f24837fe640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2484131070 0x7f248411dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-02T23:40:07.674 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.654+0000 7f24837fe640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2484131070 0x7f248411dd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:44206/0 (socket says 172.21.15.73:44206) 2024-08-02T23:40:07.674 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.654+0000 7f24837fe640 1 -- 172.21.15.73:0/2552051713 learned_addr learned my addr 172.21.15.73:0/2552051713 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-08-02T23:40:07.674 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.654+0000 7f2b97fff640 1 -- 172.21.15.73:0/918785841 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b98124c30 con 0x7f2b980a4770 2024-08-02T23:40:07.675 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.654+0000 7fb8ae59c640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb8b012d690 0x7fb8b011b490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-02T23:40:07.675 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.654+0000 7fb8ae59c640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb8b012d690 0x7fb8b011b490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:44212/0 (socket says 172.21.15.73:44212) 2024-08-02T23:40:07.675 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.654+0000 7fb8ae59c640 1 -- 172.21.15.73:0/77904191 learned_addr learned my addr 172.21.15.73:0/77904191 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-08-02T23:40:07.675 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.654+0000 7f05688f5640 1 -- 172.21.15.73:0/1256299241 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f056411c730 con 0x7f0564132a40 2024-08-02T23:40:07.676 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.654+0000 7f2b97fff640 1 --2- 172.21.15.73:0/918785841 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2b980a4770 0x7f2b98120b60 crc :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-02T23:40:07.676 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.654+0000 7f24837fe640 1 -- 172.21.15.73:0/2552051713 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2484120ee0 con 0x7f2484131070 2024-08-02T23:40:07.676 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.654+0000 7f05688f5640 1 --2- 172.21.15.73:0/1256299241 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f0564132a40 0x7f056411aad0 crc :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-02T23:40:07.677 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.654+0000 7fb8ae59c640 1 -- 172.21.15.73:0/77904191 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8b011e760 con 0x7fb8b012d690 2024-08-02T23:40:07.677 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.654+0000 7f2b94ff9640 1 -- 172.21.15.73:0/918785841 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f2b88014540 con 0x7f2b980a4770 2024-08-02T23:40:07.677 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.654+0000 7f24837fe640 1 --2- 172.21.15.73:0/2552051713 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2484131070 0x7f248411dd40 crc :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-02T23:40:07.677 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.654+0000 7f2b94ff9640 1 -- 172.21.15.73:0/918785841 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f2b880146a0 con 0x7f2b980a4770 2024-08-02T23:40:07.678 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.654+0000 7f05617fa640 1 -- 172.21.15.73:0/1256299241 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f0558014410 con 0x7f0564132a40 2024-08-02T23:40:07.678 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.654+0000 7fb8ae59c640 1 --2- 172.21.15.73:0/77904191 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb8b012d690 0x7fb8b011b490 crc :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-02T23:40:07.678 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.654+0000 7f05617fa640 1 -- 172.21.15.73:0/1256299241 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f0558014570 con 0x7f0564132a40 2024-08-02T23:40:07.678 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.654+0000 7f056ab59640 1 -- 172.21.15.73:0/1256299241 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f056411d0e0 con 0x7f0564132a40 2024-08-02T23:40:07.679 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.654+0000 7f246bfff640 1 -- 172.21.15.73:0/2552051713 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f247400c7e0 con 0x7f2484131070 2024-08-02T23:40:07.679 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.655+0000 7f05617fa640 1 -- 172.21.15.73:0/1256299241 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f0558014810 con 0x7f0564132a40 2024-08-02T23:40:07.679 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.655+0000 7f246bfff640 1 -- 172.21.15.73:0/2552051713 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f247400c940 con 0x7f2484131070 2024-08-02T23:40:07.679 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.655+0000 7fb8a6ffd640 1 -- 172.21.15.73:0/77904191 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fb8a0004af0 con 0x7fb8b012d690 2024-08-02T23:40:07.680 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.655+0000 7f056ab59640 1 -- 172.21.15.73:0/1256299241 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f056411e6a0 con 0x7f0564132a40 2024-08-02T23:40:07.680 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.655+0000 7f246bfff640 1 -- 172.21.15.73:0/2552051713 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f247400cbe0 con 0x7f2484131070 2024-08-02T23:40:07.680 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.655+0000 7f2489d99640 1 -- 172.21.15.73:0/2552051713 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2484120870 con 0x7f2484131070 2024-08-02T23:40:07.680 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.655+0000 7fb8a6ffd640 1 -- 172.21.15.73:0/77904191 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7fb8a0004c50 con 0x7fb8b012d690 2024-08-02T23:40:07.681 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.655+0000 7f2489d99640 1 -- 172.21.15.73:0/2552051713 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f248411eff0 con 0x7f2484131070 2024-08-02T23:40:07.681 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.656+0000 7f2b94ff9640 1 -- 172.21.15.73:0/918785841 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f2b88014940 con 0x7f2b980a4770 2024-08-02T23:40:07.681 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.656+0000 7f2b9e26d640 1 -- 172.21.15.73:0/918785841 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b98122740 con 0x7f2b980a4770 2024-08-02T23:40:07.681 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.655+0000 7fb8a6ffd640 1 -- 172.21.15.73:0/77904191 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fb8a0007120 con 0x7fb8b012d690 2024-08-02T23:40:07.681 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.655+0000 7fb8b4d99640 1 -- 172.21.15.73:0/77904191 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb8b011ea00 con 0x7fb8b012d690 2024-08-02T23:40:07.682 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.655+0000 7fb8b4d99640 1 -- 172.21.15.73:0/77904191 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8b011d990 con 0x7fb8b012d690 2024-08-02T23:40:07.682 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.656+0000 7f2b9e26d640 1 -- 172.21.15.73:0/918785841 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b98121c80 con 0x7f2b980a4770 2024-08-02T23:40:07.682 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.656+0000 7f246bfff640 1 -- 172.21.15.73:0/2552051713 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 5) ==== 45563+0+0 (crc 0 0 0) 0x7f24740102a0 con 0x7f2484131070 2024-08-02T23:40:07.682 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.656+0000 7fb8a4ff9640 1 -- 172.21.15.73:0/77904191 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fb8b009eda0 con 0x7fb8b012d690 2024-08-02T23:40:07.683 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.656+0000 7f246bfff640 1 --2- 172.21.15.73:0/2552051713 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7f2470038390 0x7f247003a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-02T23:40:07.683 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.656+0000 7f056ab59640 1 -- 172.21.15.73:0/1256299241 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f05640a2320 con 0x7f0564132a40 2024-08-02T23:40:07.683 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.656+0000 7f246bfff640 1 -- 172.21.15.73:0/2552051713 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 2728+0+0 (crc 0 0 0) 0x7f247401f9f0 con 0x7f2484131070 2024-08-02T23:40:07.683 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.657+0000 7f2482ffd640 1 --2- 172.21.15.73:0/2552051713 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7f2470038390 0x7f247003a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-02T23:40:07.684 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.657+0000 7fb8a6ffd640 1 -- 172.21.15.73:0/77904191 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 5) ==== 45563+0+0 (crc 0 0 0) 0x7fb8a0007310 con 0x7fb8b012d690 2024-08-02T23:40:07.684 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.657+0000 7f2489d99640 1 --2- 172.21.15.73:0/2552051713 >> v2:172.21.15.131:6809/1209903557 conn(0x7f2484131b90 0x7f248411e280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-02T23:40:07.684 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.657+0000 7f2489d99640 1 -- 172.21.15.73:0/2552051713 --> v2:172.21.15.131:6809/1209903557 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f248409b880 con 0x7f2484131b90 2024-08-02T23:40:07.684 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.656+0000 7f05617fa640 1 -- 172.21.15.73:0/1256299241 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 5) ==== 45563+0+0 (crc 0 0 0) 0x7f0558014a10 con 0x7f0564132a40 2024-08-02T23:40:07.685 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.657+0000 7f05617fa640 1 --2- 172.21.15.73:0/1256299241 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7f0548038390 0x7f054803a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-02T23:40:07.685 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.657+0000 7f05617fa640 1 -- 172.21.15.73:0/1256299241 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 2728+0+0 (crc 0 0 0) 0x7f0558019040 con 0x7f0564132a40 2024-08-02T23:40:07.685 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.657+0000 7f05617fa640 1 --2- 172.21.15.73:0/1256299241 >> v2:172.21.15.131:6802/3558187202 conn(0x7f054803d390 0x7f054803f860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-02T23:40:07.685 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.657+0000 7f05617fa640 1 -- 172.21.15.73:0/1256299241 --> v2:172.21.15.131:6802/3558187202 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f054803fef0 con 0x7f054803d390 2024-08-02T23:40:07.685 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.657+0000 7f05617fa640 1 -- 172.21.15.73:0/1256299241 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_get_version_reply(handle=1 version=16) ==== 24+0+0 (crc 0 0 0) 0x7f055804b790 con 0x7f0564132a40 2024-08-02T23:40:07.685 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.657+0000 7fb8a6ffd640 1 --2- 172.21.15.73:0/77904191 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7fb8900383e0 0x7fb89003a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-02T23:40:07.685 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.657+0000 7fb8a6ffd640 1 -- 172.21.15.73:0/77904191 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 2728+0+0 (crc 0 0 0) 0x7fb8a0048040 con 0x7fb8b012d690 2024-08-02T23:40:07.686 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.657+0000 7f2b94ff9640 1 -- 172.21.15.73:0/918785841 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 5) ==== 45563+0+0 (crc 0 0 0) 0x7f2b88014af0 con 0x7f2b980a4770 2024-08-02T23:40:07.686 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.657+0000 7f2b9e26d640 1 -- 172.21.15.73:0/918785841 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f2b9809aba0 con 0x7f2b980a4770 2024-08-02T23:40:07.686 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.657+0000 7fb8a6ffd640 1 --2- 172.21.15.73:0/77904191 >> v2:172.21.15.73:6804/3612805766 conn(0x7fb89003d3e0 0x7fb89003f8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-02T23:40:07.686 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.657+0000 7fb8a6ffd640 1 -- 172.21.15.73:0/77904191 --> v2:172.21.15.73:6804/3612805766 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb89003ff40 con 0x7fb89003d3e0 2024-08-02T23:40:07.686 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.657+0000 7fb8a6ffd640 1 -- 172.21.15.73:0/77904191 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_get_version_reply(handle=1 version=16) ==== 24+0+0 (crc 0 0 0) 0x7fb8a0048350 con 0x7fb8b012d690 2024-08-02T23:40:07.687 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.657+0000 7f0563fff640 1 --2- 172.21.15.73:0/1256299241 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7f0548038390 0x7f054803a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-02T23:40:07.687 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.657+0000 7f2b94ff9640 1 --2- 172.21.15.73:0/918785841 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7f2b840383e0 0x7f2b8403a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-02T23:40:07.687 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.657+0000 7fb8add9b640 1 --2- 172.21.15.73:0/77904191 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7fb8900383e0 0x7fb89003a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-02T23:40:07.687 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.658+0000 7f2b94ff9640 1 -- 172.21.15.73:0/918785841 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 2728+0+0 (crc 0 0 0) 0x7f2b88019040 con 0x7f2b980a4770 2024-08-02T23:40:07.688 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.658+0000 7f2482ffd640 1 --2- 172.21.15.73:0/2552051713 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7f2470038390 0x7f247003a8a0 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-02T23:40:07.688 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.658+0000 7f2b94ff9640 1 --2- 172.21.15.73:0/918785841 >> v2:172.21.15.73:6801/4128208498 conn(0x7f2b8403d3e0 0x7f2b8403f8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-02T23:40:07.688 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.658+0000 7f2b94ff9640 1 -- 172.21.15.73:0/918785841 --> v2:172.21.15.73:6801/4128208498 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2b8403ff40 con 0x7f2b8403d3e0 2024-08-02T23:40:07.688 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.658+0000 7f2b94ff9640 1 -- 172.21.15.73:0/918785841 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_get_version_reply(handle=1 version=16) ==== 24+0+0 (crc 0 0 0) 0x7f2b88013400 con 0x7f2b980a4770 2024-08-02T23:40:07.689 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.658+0000 7f2b977fe640 1 --2- 172.21.15.73:0/918785841 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7f2b840383e0 0x7f2b8403a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-02T23:40:07.689 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.658+0000 7f2483fff640 1 --2- 172.21.15.73:0/2552051713 >> v2:172.21.15.131:6809/1209903557 conn(0x7f2484131b90 0x7f248411e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-08-02T23:40:07.689 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.658+0000 7f05690f6640 1 --2- 172.21.15.73:0/1256299241 >> v2:172.21.15.131:6802/3558187202 conn(0x7f054803d390 0x7f054803f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-08-02T23:40:07.689 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.658+0000 7fb8add9b640 1 --2- 172.21.15.73:0/77904191 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7fb8900383e0 0x7fb89003a8f0 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-02T23:40:07.689 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.658+0000 7fb8aed9d640 1 --2- 172.21.15.73:0/77904191 >> v2:172.21.15.73:6804/3612805766 conn(0x7fb89003d3e0 0x7fb89003f8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-08-02T23:40:07.690 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.658+0000 7f2b9c80a640 1 --2- 172.21.15.73:0/918785841 >> v2:172.21.15.73:6801/4128208498 conn(0x7f2b8403d3e0 0x7f2b8403f8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-08-02T23:40:07.690 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.658+0000 7f2b977fe640 1 --2- 172.21.15.73:0/918785841 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7f2b840383e0 0x7f2b8403a8f0 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-02T23:40:07.690 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.659+0000 7f0563fff640 1 --2- 172.21.15.73:0/1256299241 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7f0548038390 0x7f054803a8a0 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-02T23:40:07.690 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.660+0000 7fb8aed9d640 1 --2- 172.21.15.73:0/77904191 >> v2:172.21.15.73:6804/3612805766 conn(0x7fb89003d3e0 0x7fb89003f8b0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-02T23:40:07.691 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.660+0000 7f2b9c80a640 1 --2- 172.21.15.73:0/918785841 >> v2:172.21.15.73:6801/4128208498 conn(0x7f2b8403d3e0 0x7f2b8403f8b0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-02T23:40:07.691 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.660+0000 7f05690f6640 1 --2- 172.21.15.73:0/1256299241 >> v2:172.21.15.131:6802/3558187202 conn(0x7f054803d390 0x7f054803f860 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-02T23:40:07.691 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.660+0000 7f2483fff640 1 --2- 172.21.15.73:0/2552051713 >> v2:172.21.15.131:6809/1209903557 conn(0x7f2484131b90 0x7f248411e280 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-02T23:40:07.691 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.661+0000 7fb8a6ffd640 1 -- 172.21.15.73:0/77904191 <== osd.1 v2:172.21.15.73:6804/3612805766 1 ==== command_reply(tid 1: 0 ) ==== 8+0+5172 (crc 0 0 0) 0x7fb89003ff40 con 0x7fb89003d3e0 2024-08-02T23:40:07.692 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.661+0000 7f2b94ff9640 1 -- 172.21.15.73:0/918785841 <== osd.0 v2:172.21.15.73:6801/4128208498 1 ==== command_reply(tid 1: 0 ) ==== 8+0+5172 (crc 0 0 0) 0x7f2b8403ff40 con 0x7f2b8403d3e0 2024-08-02T23:40:07.692 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.661+0000 7f246bfff640 1 -- 172.21.15.73:0/2552051713 <== osd.3 v2:172.21.15.131:6809/1209903557 1 ==== command_reply(tid 1: 0 ) ==== 8+0+5172 (crc 0 0 0) 0x7f247c00a820 con 0x7f2484131b90 2024-08-02T23:40:07.692 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.661+0000 7f05617fa640 1 -- 172.21.15.73:0/1256299241 <== osd.2 v2:172.21.15.131:6802/3558187202 1 ==== command_reply(tid 1: 0 ) ==== 8+0+5172 (crc 0 0 0) 0x7f054803fef0 con 0x7f054803d390 2024-08-02T23:40:07.693 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.665+0000 7f2b9e26d640 1 -- 172.21.15.73:0/918785841 --> v2:172.21.15.73:6801/4128208498 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f2b98124460 con 0x7f2b8403d3e0 2024-08-02T23:40:07.693 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.665+0000 7fb8a4ff9640 1 -- 172.21.15.73:0/77904191 --> v2:172.21.15.73:6804/3612805766 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fb8b006eaa0 con 0x7fb89003d3e0 2024-08-02T23:40:07.693 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.666+0000 7f2b94ff9640 1 -- 172.21.15.73:0/918785841 <== osd.0 v2:172.21.15.73:6801/4128208498 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f2b98124460 con 0x7f2b8403d3e0 2024-08-02T23:40:07.693 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.666+0000 7fb8a6ffd640 1 -- 172.21.15.73:0/77904191 <== osd.1 v2:172.21.15.73:6804/3612805766 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fb8a800a820 con 0x7fb89003d3e0 2024-08-02T23:40:07.694 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.666+0000 7f2b7e7fc640 1 -- 172.21.15.73:0/918785841 >> v2:172.21.15.73:6801/4128208498 conn(0x7f2b8403d3e0 msgr2=0x7f2b8403f8b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-02T23:40:07.694 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.666+0000 7f2b7e7fc640 1 --2- 172.21.15.73:0/918785841 >> v2:172.21.15.73:6801/4128208498 conn(0x7f2b8403d3e0 0x7f2b8403f8b0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:40:07.694 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.666+0000 7f2b7e7fc640 1 -- 172.21.15.73:0/918785841 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7f2b840383e0 msgr2=0x7f2b8403a8f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-02T23:40:07.695 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.666+0000 7fb8a4ff9640 1 -- 172.21.15.73:0/77904191 >> v2:172.21.15.73:6804/3612805766 conn(0x7fb89003d3e0 msgr2=0x7fb89003f8b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-02T23:40:07.695 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.666+0000 7f2b7e7fc640 1 --2- 172.21.15.73:0/918785841 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7f2b840383e0 0x7f2b8403a8f0 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:40:07.695 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.666+0000 7fb8a4ff9640 1 --2- 172.21.15.73:0/77904191 >> v2:172.21.15.73:6804/3612805766 conn(0x7fb89003d3e0 0x7fb89003f8b0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:40:07.695 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.666+0000 7f2b7e7fc640 1 -- 172.21.15.73:0/918785841 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2b980a4770 msgr2=0x7f2b98120b60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-02T23:40:07.696 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.666+0000 7f2b7e7fc640 1 --2- 172.21.15.73:0/918785841 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2b980a4770 0x7f2b98120b60 crc :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:40:07.696 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.666+0000 7f2b7e7fc640 1 -- 172.21.15.73:0/918785841 shutdown_connections 2024-08-02T23:40:07.696 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.666+0000 7f2b7e7fc640 1 --2- 172.21.15.73:0/918785841 >> v2:172.21.15.73:6801/4128208498 conn(0x7f2b8403d3e0 0x7f2b8403f8b0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:40:07.696 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.666+0000 7f2b7e7fc640 1 --2- 172.21.15.73:0/918785841 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7f2b840383e0 0x7f2b8403a8f0 crc :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:40:07.696 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.666+0000 7f2b7e7fc640 1 --2- 172.21.15.73:0/918785841 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2b980a4770 0x7f2b98120b60 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:40:07.696 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.666+0000 7f2b7e7fc640 1 -- 172.21.15.73:0/918785841 >> 172.21.15.73:0/918785841 conn(0x7f2b980a3510 msgr2=0x7f2b980a12e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-02T23:40:07.696 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.666+0000 7fb8a4ff9640 1 -- 172.21.15.73:0/77904191 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7fb8900383e0 msgr2=0x7fb89003a8f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-02T23:40:07.696 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.666+0000 7fb8a4ff9640 1 --2- 172.21.15.73:0/77904191 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7fb8900383e0 0x7fb89003a8f0 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:40:07.696 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.667+0000 7fb8a4ff9640 1 -- 172.21.15.73:0/77904191 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb8b012d690 msgr2=0x7fb8b011b490 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-02T23:40:07.696 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.667+0000 7fb8a4ff9640 1 --2- 172.21.15.73:0/77904191 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb8b012d690 0x7fb8b011b490 crc :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:40:07.697 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.667+0000 7f2b7e7fc640 1 -- 172.21.15.73:0/918785841 shutdown_connections 2024-08-02T23:40:07.697 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.667+0000 7f2b7e7fc640 1 -- 172.21.15.73:0/918785841 wait complete. 2024-08-02T23:40:07.698 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.667+0000 7fb8a4ff9640 1 -- 172.21.15.73:0/77904191 shutdown_connections 2024-08-02T23:40:07.698 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.667+0000 7fb8a4ff9640 1 --2- 172.21.15.73:0/77904191 >> v2:172.21.15.73:6804/3612805766 conn(0x7fb89003d3e0 0x7fb89003f8b0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:40:07.698 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.667+0000 7fb8a4ff9640 1 --2- 172.21.15.73:0/77904191 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7fb8900383e0 0x7fb89003a8f0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:40:07.698 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.667+0000 7fb8a4ff9640 1 --2- 172.21.15.73:0/77904191 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb8b012d690 0x7fb8b011b490 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:40:07.698 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.667+0000 7fb8a4ff9640 1 -- 172.21.15.73:0/77904191 >> 172.21.15.73:0/77904191 conn(0x7fb8b00a0590 msgr2=0x7fb8b009db70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-02T23:40:07.698 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.668+0000 7fb8a4ff9640 1 -- 172.21.15.73:0/77904191 shutdown_connections 2024-08-02T23:40:07.699 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.668+0000 7fb8a4ff9640 1 -- 172.21.15.73:0/77904191 wait complete. 2024-08-02T23:40:07.699 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.668+0000 7f2489d99640 1 -- 172.21.15.73:0/2552051713 --> v2:172.21.15.131:6809/1209903557 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f24841bc5f0 con 0x7f2484131b90 2024-08-02T23:40:07.699 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.669+0000 7f246bfff640 1 -- 172.21.15.73:0/2552051713 <== osd.3 v2:172.21.15.131:6809/1209903557 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f24841bc5f0 con 0x7f2484131b90 2024-08-02T23:40:07.700 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.669+0000 7f2489d99640 1 -- 172.21.15.73:0/2552051713 >> v2:172.21.15.131:6809/1209903557 conn(0x7f2484131b90 msgr2=0x7f248411e280 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-02T23:40:07.700 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.669+0000 7f2489d99640 1 --2- 172.21.15.73:0/2552051713 >> v2:172.21.15.131:6809/1209903557 conn(0x7f2484131b90 0x7f248411e280 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:40:07.700 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.669+0000 7f2489d99640 1 -- 172.21.15.73:0/2552051713 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7f2470038390 msgr2=0x7f247003a8a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-02T23:40:07.700 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.669+0000 7f2489d99640 1 --2- 172.21.15.73:0/2552051713 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7f2470038390 0x7f247003a8a0 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:40:07.700 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.669+0000 7f2489d99640 1 -- 172.21.15.73:0/2552051713 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2484131070 msgr2=0x7f248411dd40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-02T23:40:07.700 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.669+0000 7f2489d99640 1 --2- 172.21.15.73:0/2552051713 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2484131070 0x7f248411dd40 crc :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:40:07.700 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.669+0000 7f2489d99640 1 -- 172.21.15.73:0/2552051713 shutdown_connections 2024-08-02T23:40:07.700 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.669+0000 7f2489d99640 1 --2- 172.21.15.73:0/2552051713 >> v2:172.21.15.131:6809/1209903557 conn(0x7f2484131b90 0x7f248411e280 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:40:07.700 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.669+0000 7f2489d99640 1 --2- 172.21.15.73:0/2552051713 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7f2470038390 0x7f247003a8a0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:40:07.701 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.669+0000 7f2489d99640 1 --2- 172.21.15.73:0/2552051713 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2484131070 0x7f248411dd40 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:40:07.701 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.669+0000 7f2489d99640 1 -- 172.21.15.73:0/2552051713 >> 172.21.15.73:0/2552051713 conn(0x7f248406cf90 msgr2=0x7f24840a46e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-02T23:40:07.701 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.669+0000 7f056ab59640 1 -- 172.21.15.73:0/1256299241 --> v2:172.21.15.131:6802/3558187202 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f056411ecf0 con 0x7f054803d390 2024-08-02T23:40:07.701 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.670+0000 7f2489d99640 1 -- 172.21.15.73:0/2552051713 shutdown_connections 2024-08-02T23:40:07.701 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.670+0000 7f2489d99640 1 -- 172.21.15.73:0/2552051713 wait complete. 2024-08-02T23:40:07.702 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.671+0000 7f05617fa640 1 -- 172.21.15.73:0/1256299241 <== osd.2 v2:172.21.15.131:6802/3558187202 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f056411ecf0 con 0x7f054803d390 2024-08-02T23:40:07.703 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.671+0000 7f056ab59640 1 -- 172.21.15.73:0/1256299241 >> v2:172.21.15.131:6802/3558187202 conn(0x7f054803d390 msgr2=0x7f054803f860 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-02T23:40:07.703 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.671+0000 7f056ab59640 1 --2- 172.21.15.73:0/1256299241 >> v2:172.21.15.131:6802/3558187202 conn(0x7f054803d390 0x7f054803f860 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:40:07.703 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.671+0000 7f056ab59640 1 -- 172.21.15.73:0/1256299241 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7f0548038390 msgr2=0x7f054803a8a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-02T23:40:07.703 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.671+0000 7f056ab59640 1 --2- 172.21.15.73:0/1256299241 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7f0548038390 0x7f054803a8a0 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:40:07.703 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.671+0000 7f056ab59640 1 -- 172.21.15.73:0/1256299241 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f0564132a40 msgr2=0x7f056411aad0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-02T23:40:07.703 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.671+0000 7f056ab59640 1 --2- 172.21.15.73:0/1256299241 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f0564132a40 0x7f056411aad0 crc :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:40:07.703 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.671+0000 7f056ab59640 1 -- 172.21.15.73:0/1256299241 shutdown_connections 2024-08-02T23:40:07.703 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.671+0000 7f056ab59640 1 --2- 172.21.15.73:0/1256299241 >> v2:172.21.15.131:6802/3558187202 conn(0x7f054803d390 0x7f054803f860 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:40:07.703 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.671+0000 7f056ab59640 1 --2- 172.21.15.73:0/1256299241 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7f0548038390 0x7f054803a8a0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:40:07.703 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.671+0000 7f056ab59640 1 --2- 172.21.15.73:0/1256299241 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f0564132a40 0x7f056411aad0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:40:07.704 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.671+0000 7f056ab59640 1 -- 172.21.15.73:0/1256299241 >> 172.21.15.73:0/1256299241 conn(0x7f05640a0590 msgr2=0x7f05641ba390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-02T23:40:07.704 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.672+0000 7f056ab59640 1 -- 172.21.15.73:0/1256299241 shutdown_connections 2024-08-02T23:40:07.704 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.672+0000 7f056ab59640 1 -- 172.21.15.73:0/1256299241 wait complete. 2024-08-02T23:40:07.704 INFO:teuthology.orchestra.run.smithi073.stdout:38654705673 2024-08-02T23:40:07.705 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-08-02T23:40:07.705 INFO:teuthology.orchestra.run.smithi073.stdout:38654705673 2024-08-02T23:40:07.706 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-08-02T23:40:07.706 INFO:teuthology.orchestra.run.smithi073.stdout:38654705673 2024-08-02T23:40:07.707 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-08-02T23:40:07.708 INFO:teuthology.orchestra.run.smithi073.stdout:42949672968 2024-08-02T23:40:07.708 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-08-02T23:40:07.860 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.854+0000 7f78fc119640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-02T23:40:07.861 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.854+0000 7fdc41599640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-02T23:40:07.861 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.854+0000 7f8610f99640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-02T23:40:07.861 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.854+0000 7f4b4b159640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-02T23:40:07.862 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.859+0000 7f78fc119640 1 Processor -- start 2024-08-02T23:40:07.862 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.860+0000 7fdc41599640 1 Processor -- start 2024-08-02T23:40:07.862 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.860+0000 7f8610f99640 1 Processor -- start 2024-08-02T23:40:07.862 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.860+0000 7fdc41599640 1 -- start start 2024-08-02T23:40:07.863 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.860+0000 7f8610f99640 1 -- start start 2024-08-02T23:40:07.863 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.860+0000 7f78fc119640 1 -- start start 2024-08-02T23:40:07.863 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.860+0000 7f4b4b159640 1 Processor -- start 2024-08-02T23:40:07.863 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.860+0000 7f4b4b159640 1 -- start start 2024-08-02T23:40:07.864 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.861+0000 7f78fc119640 1 --2- >> v2:172.21.15.73:3300/0 conn(0x7f78f40a4770 0x7f78f40a4b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-02T23:40:07.864 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.861+0000 7f78fc119640 1 -- --> v1:172.21.15.73:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f78f406e030 con 0x7f78f40a50c0 2024-08-02T23:40:07.864 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.861+0000 7fdc41599640 1 --2- >> v2:172.21.15.73:3300/0 conn(0x7fdc3c131b90 0x7fdc3c139930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-02T23:40:07.864 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.861+0000 7f78fc119640 1 -- --> v2:172.21.15.73:3300/0 -- mon_getmap magic: 0 -- 0x7f78f40a0ed0 con 0x7f78f40a4770 2024-08-02T23:40:07.865 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.861+0000 7f8610f99640 1 --2- >> v2:172.21.15.73:3300/0 conn(0x7f860c131070 0x7f860c131480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-02T23:40:07.865 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.861+0000 7f8610f99640 1 -- --> v1:172.21.15.73:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f860c06df40 con 0x7f860c1319c0 2024-08-02T23:40:07.865 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.861+0000 7f8610f99640 1 -- --> v2:172.21.15.73:3300/0 -- mon_getmap magic: 0 -- 0x7f860c0a0e40 con 0x7f860c131070 2024-08-02T23:40:07.865 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.861+0000 7fdc41599640 1 -- --> v1:172.21.15.73:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdc3c06df40 con 0x7fdc3c131070 2024-08-02T23:40:07.865 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.861+0000 7f4b4b159640 1 --2- >> v2:172.21.15.73:3300/0 conn(0x7f4b4409ae70 0x7f4b4409b280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-02T23:40:07.866 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.861+0000 7fdc3affd640 1 --1- >> v1:172.21.15.73:6789/0 conn(0x7fdc3c131070 0x7fdc3c131480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.73:6789/0 says I am v1:172.21.15.73:43420/0 (socket says 172.21.15.73:43420) 2024-08-02T23:40:07.866 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.861+0000 7f4b4b159640 1 -- --> v1:172.21.15.73:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4b4409aa90 con 0x7f4b44132080 2024-08-02T23:40:07.866 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.861+0000 7fdc3affd640 1 -- 172.21.15.73:0/3760948363 learned_addr learned my addr 172.21.15.73:0/3760948363 (peer_addr_for_me v1:172.21.15.73:0/0) 2024-08-02T23:40:07.867 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.861+0000 7f4b4b159640 1 -- --> v2:172.21.15.73:3300/0 -- mon_getmap magic: 0 -- 0x7f4b4406bd50 con 0x7f4b4409ae70 2024-08-02T23:40:07.867 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.861+0000 7f8609d9b640 1 --1- >> v1:172.21.15.73:6789/0 conn(0x7f860c1319c0 0x7f860c0a22f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.73:6789/0 says I am v1:172.21.15.73:43442/0 (socket says 172.21.15.73:43442) 2024-08-02T23:40:07.867 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.861+0000 7fdc41599640 1 -- 172.21.15.73:0/3760948363 --> v2:172.21.15.73:3300/0 -- mon_getmap magic: 0 -- 0x7fdc3c0a0e40 con 0x7fdc3c131b90 2024-08-02T23:40:07.867 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.861+0000 7f8609d9b640 1 -- 172.21.15.73:0/428195361 learned_addr learned my addr 172.21.15.73:0/428195361 (peer_addr_for_me v1:172.21.15.73:0/0) 2024-08-02T23:40:07.868 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.861+0000 7f78f96b4640 1 --1- >> v1:172.21.15.73:6789/0 conn(0x7f78f40a50c0 0x7f78f4131000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.73:6789/0 says I am v1:172.21.15.73:43426/0 (socket says 172.21.15.73:43426) 2024-08-02T23:40:07.868 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.861+0000 7f78f96b4640 1 -- 172.21.15.73:0/3721326940 learned_addr learned my addr 172.21.15.73:0/3721326940 (peer_addr_for_me v1:172.21.15.73:0/0) 2024-08-02T23:40:07.868 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.861+0000 7f78f9eb5640 1 --2- 172.21.15.73:0/3721326940 >> v2:172.21.15.73:3300/0 conn(0x7f78f40a4770 0x7f78f40a4b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-02T23:40:07.869 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.861+0000 7fdc3a7fc640 1 --2- 172.21.15.73:0/3760948363 >> v2:172.21.15.73:3300/0 conn(0x7fdc3c131b90 0x7fdc3c139930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-02T23:40:07.869 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.861+0000 7fdc39ffb640 1 -- 172.21.15.73:0/3760948363 <== mon.0 v1:172.21.15.73:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1688140764 0 0) 0x7fdc3c06df40 con 0x7fdc3c131070 2024-08-02T23:40:07.869 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.861+0000 7fdc39ffb640 1 -- 172.21.15.73:0/3760948363 --> v1:172.21.15.73:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdc240034f0 con 0x7fdc3c131070 2024-08-02T23:40:07.869 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.861+0000 7f860a59c640 1 --2- 172.21.15.73:0/428195361 >> v2:172.21.15.73:3300/0 conn(0x7f860c131070 0x7f860c131480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-02T23:40:07.870 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.862+0000 7f4b48ef5640 1 --2- >> v2:172.21.15.73:3300/0 conn(0x7f4b4409ae70 0x7f4b4409b280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-02T23:40:07.870 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.862+0000 7f4b43fff640 1 --1- >> v1:172.21.15.73:6789/0 conn(0x7f4b44132080 0x7f4b44132490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.73:6789/0 says I am v1:172.21.15.73:43446/0 (socket says 172.21.15.73:43446) 2024-08-02T23:40:07.870 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.862+0000 7f4b43fff640 1 -- 172.21.15.73:0/607976382 learned_addr learned my addr 172.21.15.73:0/607976382 (peer_addr_for_me v1:172.21.15.73:0/0) 2024-08-02T23:40:07.870 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.862+0000 7f78f8eb3640 1 -- 172.21.15.73:0/3721326940 <== mon.0 v1:172.21.15.73:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 190806200 0 0) 0x7f78f406e030 con 0x7f78f40a50c0 2024-08-02T23:40:07.870 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.862+0000 7f78f8eb3640 1 -- 172.21.15.73:0/3721326940 --> v1:172.21.15.73:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f78e00034f0 con 0x7f78f40a50c0 2024-08-02T23:40:07.871 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.862+0000 7fdc3a7fc640 1 -- 172.21.15.73:0/3760948363 >> v1:172.21.15.73:6789/0 conn(0x7fdc3c131070 legacy=0x7fdc3c131480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-02T23:40:07.871 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.862+0000 7fdc3a7fc640 1 -- 172.21.15.73:0/3760948363 --> v2:172.21.15.73:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc3c06dc30 con 0x7fdc3c131b90 2024-08-02T23:40:07.871 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.862+0000 7f860a59c640 1 -- 172.21.15.73:0/428195361 >> v1:172.21.15.73:6789/0 conn(0x7f860c1319c0 legacy=0x7f860c0a22f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-02T23:40:07.871 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.862+0000 7f860a59c640 1 -- 172.21.15.73:0/428195361 --> v2:172.21.15.73:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f860c06dc30 con 0x7f860c131070 2024-08-02T23:40:07.871 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.862+0000 7f4b48ef5640 1 -- 172.21.15.73:0/607976382 >> v1:172.21.15.73:6789/0 conn(0x7f4b44132080 legacy=0x7f4b44132490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-02T23:40:07.871 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.862+0000 7f4b48ef5640 1 -- 172.21.15.73:0/607976382 --> v2:172.21.15.73:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b4409bde0 con 0x7f4b4409ae70 2024-08-02T23:40:07.872 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.862+0000 7f860a59c640 1 --2- 172.21.15.73:0/428195361 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f860c131070 0x7f860c131480 crc :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96e00dc143d064bd server_cookie=0 in_seq=0 out_seq=0 2024-08-02T23:40:07.872 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.862+0000 7fdc3a7fc640 1 --2- 172.21.15.73:0/3760948363 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fdc3c131b90 0x7fdc3c139930 crc :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95e990cf5aa06fcc server_cookie=0 in_seq=0 out_seq=0 2024-08-02T23:40:07.872 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.862+0000 7f78f9eb5640 1 -- 172.21.15.73:0/3721326940 >> v1:172.21.15.73:6789/0 conn(0x7f78f40a50c0 legacy=0x7f78f4131000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-02T23:40:07.872 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.862+0000 7f78f9eb5640 1 -- 172.21.15.73:0/3721326940 --> v2:172.21.15.73:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78f406e260 con 0x7f78f40a4770 2024-08-02T23:40:07.873 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.862+0000 7f4b48ef5640 1 --2- 172.21.15.73:0/607976382 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f4b4409ae70 0x7f4b4409b280 crc :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6903f9cf642726d server_cookie=0 in_seq=0 out_seq=0 2024-08-02T23:40:07.873 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.862+0000 7f860959a640 1 -- 172.21.15.73:0/428195361 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f85fc009ce0 con 0x7f860c131070 2024-08-02T23:40:07.873 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.862+0000 7f860959a640 1 -- 172.21.15.73:0/428195361 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f85fc009e40 con 0x7f860c131070 2024-08-02T23:40:07.873 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.862+0000 7f78f9eb5640 1 --2- 172.21.15.73:0/3721326940 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f78f40a4770 0x7f78f40a4b80 crc :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b76265ba5c8eae3 server_cookie=0 in_seq=0 out_seq=0 2024-08-02T23:40:07.874 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.862+0000 7fdc39ffb640 1 -- 172.21.15.73:0/3760948363 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fdc30009ce0 con 0x7fdc3c131b90 2024-08-02T23:40:07.874 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.862+0000 7fdc39ffb640 1 -- 172.21.15.73:0/3760948363 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7fdc30009e40 con 0x7fdc3c131b90 2024-08-02T23:40:07.874 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.862+0000 7fdc39ffb640 1 -- 172.21.15.73:0/3760948363 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fdc3000e360 con 0x7fdc3c131b90 2024-08-02T23:40:07.874 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.862+0000 7f4b437fe640 1 -- 172.21.15.73:0/607976382 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f4b38009ce0 con 0x7f4b4409ae70 2024-08-02T23:40:07.875 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.862+0000 7f860959a640 1 -- 172.21.15.73:0/428195361 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f85fc009ce0 con 0x7f860c131070 2024-08-02T23:40:07.875 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.863+0000 7f4b437fe640 1 -- 172.21.15.73:0/607976382 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f4b38009e40 con 0x7f4b4409ae70 2024-08-02T23:40:07.875 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.863+0000 7f4b437fe640 1 -- 172.21.15.73:0/607976382 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f4b3800e360 con 0x7f4b4409ae70 2024-08-02T23:40:07.875 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.863+0000 7f78f8eb3640 1 -- 172.21.15.73:0/3721326940 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f78f0009ce0 con 0x7f78f40a4770 2024-08-02T23:40:07.875 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.863+0000 7f78f8eb3640 1 -- 172.21.15.73:0/3721326940 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f78f0009e40 con 0x7f78f40a4770 2024-08-02T23:40:07.876 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.863+0000 7fdc41599640 1 -- 172.21.15.73:0/3760948363 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fdc3c131b90 msgr2=0x7fdc3c139930 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-02T23:40:07.876 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.863+0000 7fdc41599640 1 --2- 172.21.15.73:0/3760948363 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fdc3c131b90 0x7fdc3c139930 crc :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:40:07.876 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.863+0000 7f4b4b159640 1 -- 172.21.15.73:0/607976382 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f4b4409ae70 msgr2=0x7f4b4409b280 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-02T23:40:07.876 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.863+0000 7f4b4b159640 1 --2- 172.21.15.73:0/607976382 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f4b4409ae70 0x7f4b4409b280 crc :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:40:07.877 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.863+0000 7f78f8eb3640 1 -- 172.21.15.73:0/3721326940 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f78f0009ce0 con 0x7f78f40a4770 2024-08-02T23:40:07.877 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.864+0000 7fdc41599640 1 -- 172.21.15.73:0/3760948363 shutdown_connections 2024-08-02T23:40:07.877 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.864+0000 7f4b4b159640 1 -- 172.21.15.73:0/607976382 shutdown_connections 2024-08-02T23:40:07.877 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.864+0000 7fdc41599640 1 --2- 172.21.15.73:0/3760948363 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fdc3c131b90 0x7fdc3c139930 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:40:07.878 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.864+0000 7f4b4b159640 1 --2- 172.21.15.73:0/607976382 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f4b4409ae70 0x7f4b4409b280 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:40:07.878 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.864+0000 7f4b4b159640 1 -- 172.21.15.73:0/607976382 >> 172.21.15.73:0/607976382 conn(0x7f4b440a1860 msgr2=0x7f4b440a1c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-02T23:40:07.878 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.864+0000 7fdc41599640 1 -- 172.21.15.73:0/3760948363 >> 172.21.15.73:0/3760948363 conn(0x7fdc3c06cf90 msgr2=0x7fdc3c06d3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-02T23:40:07.878 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.864+0000 7f8610f99640 1 -- 172.21.15.73:0/428195361 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f860c131070 msgr2=0x7f860c131480 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-02T23:40:07.878 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.864+0000 7f8610f99640 1 --2- 172.21.15.73:0/428195361 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f860c131070 0x7f860c131480 crc :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:40:07.879 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.864+0000 7fdc41599640 1 -- 172.21.15.73:0/3760948363 shutdown_connections 2024-08-02T23:40:07.879 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.864+0000 7f4b4b159640 1 -- 172.21.15.73:0/607976382 shutdown_connections 2024-08-02T23:40:07.879 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.864+0000 7f4b4b159640 1 -- 172.21.15.73:0/607976382 wait complete. 2024-08-02T23:40:07.879 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.864+0000 7f8610f99640 1 -- 172.21.15.73:0/428195361 shutdown_connections 2024-08-02T23:40:07.880 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.864+0000 7f8610f99640 1 --2- 172.21.15.73:0/428195361 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f860c131070 0x7f860c131480 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:40:07.880 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.864+0000 7f8610f99640 1 -- 172.21.15.73:0/428195361 >> 172.21.15.73:0/428195361 conn(0x7f860c06cf90 msgr2=0x7f860c06d3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-02T23:40:07.880 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.864+0000 7fdc41599640 1 -- 172.21.15.73:0/3760948363 wait complete. 2024-08-02T23:40:07.880 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.864+0000 7f78fc119640 1 -- 172.21.15.73:0/3721326940 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f78f40a4770 msgr2=0x7f78f40a4b80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-02T23:40:07.880 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.864+0000 7f78fc119640 1 --2- 172.21.15.73:0/3721326940 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f78f40a4770 0x7f78f40a4b80 crc :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:40:07.880 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.864+0000 7f4b4b159640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-02T23:40:07.881 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.864+0000 7f8610f99640 1 -- 172.21.15.73:0/428195361 shutdown_connections 2024-08-02T23:40:07.881 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.864+0000 7fdc41599640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-02T23:40:07.881 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.864+0000 7f8610f99640 1 -- 172.21.15.73:0/428195361 wait complete. 2024-08-02T23:40:07.882 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.864+0000 7f78fc119640 1 -- 172.21.15.73:0/3721326940 shutdown_connections 2024-08-02T23:40:07.882 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.864+0000 7f78fc119640 1 --2- 172.21.15.73:0/3721326940 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f78f40a4770 0x7f78f40a4b80 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:40:07.882 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.864+0000 7f78fc119640 1 -- 172.21.15.73:0/3721326940 >> 172.21.15.73:0/3721326940 conn(0x7f78f40a3510 msgr2=0x7f78f40a3920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-02T23:40:07.882 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.864+0000 7f78fc119640 1 -- 172.21.15.73:0/3721326940 shutdown_connections 2024-08-02T23:40:07.882 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.864+0000 7f78fc119640 1 -- 172.21.15.73:0/3721326940 wait complete. 2024-08-02T23:40:07.882 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.864+0000 7f8610f99640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-02T23:40:07.883 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.864+0000 7f78fc119640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-02T23:40:07.883 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.865+0000 7f4b4b159640 1 Processor -- start 2024-08-02T23:40:07.883 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.865+0000 7f4b4b159640 1 -- start start 2024-08-02T23:40:07.883 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.865+0000 7fdc41599640 1 Processor -- start 2024-08-02T23:40:07.884 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.865+0000 7f78fc119640 1 Processor -- start 2024-08-02T23:40:07.884 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.865+0000 7fdc41599640 1 -- start start 2024-08-02T23:40:07.884 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.865+0000 7f78fc119640 1 -- start start 2024-08-02T23:40:07.885 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.865+0000 7f8610f99640 1 Processor -- start 2024-08-02T23:40:07.885 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.865+0000 7f8610f99640 1 -- start start 2024-08-02T23:40:07.885 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.866+0000 7f78fc119640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f78f40a4770 0x7f78f411ea90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-02T23:40:07.885 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.866+0000 7f78fc119640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 -- 0x7f78f41228e0 con 0x7f78f40a4770 2024-08-02T23:40:07.885 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.866+0000 7fdc41599640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fdc3c131070 0x7fdc3c11da40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-02T23:40:07.886 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.866+0000 7f4b4b159640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f4b4409ae70 0x7f4b4411f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-02T23:40:07.886 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.866+0000 7fdc41599640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 -- 0x7fdc3c09bf00 con 0x7fdc3c131070 2024-08-02T23:40:07.886 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.866+0000 7f4b4b159640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 -- 0x7f4b441204d0 con 0x7f4b4409ae70 2024-08-02T23:40:07.886 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.866+0000 7fdc3affd640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fdc3c131070 0x7fdc3c11da40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-02T23:40:07.887 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.866+0000 7f78f9eb5640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f78f40a4770 0x7f78f411ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-02T23:40:07.887 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.866+0000 7fdc3affd640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fdc3c131070 0x7fdc3c11da40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:44268/0 (socket says 172.21.15.73:44268) 2024-08-02T23:40:07.887 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.866+0000 7fdc3affd640 1 -- 172.21.15.73:0/1204902602 learned_addr learned my addr 172.21.15.73:0/1204902602 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-08-02T23:40:07.888 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.866+0000 7f4b48ef5640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f4b4409ae70 0x7f4b4411f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-02T23:40:07.888 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.866+0000 7f4b48ef5640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f4b4409ae70 0x7f4b4411f0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:44274/0 (socket says 172.21.15.73:44274) 2024-08-02T23:40:07.888 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.866+0000 7f4b48ef5640 1 -- 172.21.15.73:0/3070055702 learned_addr learned my addr 172.21.15.73:0/3070055702 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-08-02T23:40:07.888 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.866+0000 7f78f9eb5640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f78f40a4770 0x7f78f411ea90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:44252/0 (socket says 172.21.15.73:44252) 2024-08-02T23:40:07.888 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.866+0000 7f78f9eb5640 1 -- 172.21.15.73:0/959741601 learned_addr learned my addr 172.21.15.73:0/959741601 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-08-02T23:40:07.888 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.866+0000 7f8610f99640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f860c131070 0x7f860c11da70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-02T23:40:07.888 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.866+0000 7f8610f99640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 -- 0x7f860c1308c0 con 0x7f860c131070 2024-08-02T23:40:07.889 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.867+0000 7fdc3affd640 1 -- 172.21.15.73:0/1204902602 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc3c130850 con 0x7fdc3c131070 2024-08-02T23:40:07.889 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.867+0000 7f4b48ef5640 1 -- 172.21.15.73:0/3070055702 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b4409c7b0 con 0x7f4b4409ae70 2024-08-02T23:40:07.889 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.867+0000 7fdc3affd640 1 --2- 172.21.15.73:0/1204902602 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fdc3c131070 0x7fdc3c11da40 crc :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-02T23:40:07.890 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.867+0000 7f4b48ef5640 1 --2- 172.21.15.73:0/3070055702 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f4b4409ae70 0x7f4b4411f0f0 crc :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-02T23:40:07.890 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.867+0000 7f860a59c640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f860c131070 0x7f860c11da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-02T23:40:07.890 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.867+0000 7f860a59c640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f860c131070 0x7f860c11da70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:44282/0 (socket says 172.21.15.73:44282) 2024-08-02T23:40:07.890 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.867+0000 7f78f9eb5640 1 -- 172.21.15.73:0/959741601 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78f4131900 con 0x7f78f40a4770 2024-08-02T23:40:07.891 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.867+0000 7fdc237fe640 1 -- 172.21.15.73:0/1204902602 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fdc2c0059d0 con 0x7fdc3c131070 2024-08-02T23:40:07.891 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.867+0000 7fdc237fe640 1 -- 172.21.15.73:0/1204902602 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7fdc2c005b30 con 0x7fdc3c131070 2024-08-02T23:40:07.891 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.867+0000 7fdc41599640 1 -- 172.21.15.73:0/1204902602 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc3c11f660 con 0x7fdc3c131070 2024-08-02T23:40:07.891 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.867+0000 7f4b417fa640 1 -- 172.21.15.73:0/3070055702 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f4b38014620 con 0x7f4b4409ae70 2024-08-02T23:40:07.891 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.867+0000 7fdc237fe640 1 -- 172.21.15.73:0/1204902602 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fdc2c005dd0 con 0x7fdc3c131070 2024-08-02T23:40:07.892 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.867+0000 7f860a59c640 1 -- 172.21.15.73:0/380516133 learned_addr learned my addr 172.21.15.73:0/380516133 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-08-02T23:40:07.892 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.867+0000 7f4b4b159640 1 -- 172.21.15.73:0/3070055702 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b4412fae0 con 0x7f4b4409ae70 2024-08-02T23:40:07.892 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.867+0000 7f4b417fa640 1 -- 172.21.15.73:0/3070055702 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f4b38014780 con 0x7f4b4409ae70 2024-08-02T23:40:07.892 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.867+0000 7f4b417fa640 1 -- 172.21.15.73:0/3070055702 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f4b38014a20 con 0x7f4b4409ae70 2024-08-02T23:40:07.892 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.867+0000 7f4b4b159640 1 -- 172.21.15.73:0/3070055702 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b441201a0 con 0x7f4b4409ae70 2024-08-02T23:40:07.892 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.867+0000 7f78f9eb5640 1 --2- 172.21.15.73:0/959741601 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f78f40a4770 0x7f78f411ea90 crc :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-02T23:40:07.893 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.867+0000 7fdc41599640 1 -- 172.21.15.73:0/1204902602 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc3c1205c0 con 0x7fdc3c131070 2024-08-02T23:40:07.893 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.867+0000 7f860a59c640 1 -- 172.21.15.73:0/380516133 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f860c11f6d0 con 0x7f860c131070 2024-08-02T23:40:07.893 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.868+0000 7f860a59c640 1 --2- 172.21.15.73:0/380516133 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f860c131070 0x7f860c11da70 crc :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-02T23:40:07.893 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.868+0000 7f78ea7fc640 1 -- 172.21.15.73:0/959741601 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f78f0014540 con 0x7f78f40a4770 2024-08-02T23:40:07.894 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.868+0000 7f78ea7fc640 1 -- 172.21.15.73:0/959741601 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f78f00146a0 con 0x7f78f40a4770 2024-08-02T23:40:07.894 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.868+0000 7f78fc119640 1 -- 172.21.15.73:0/959741601 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78f4120670 con 0x7f78f40a4770 2024-08-02T23:40:07.894 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.868+0000 7f85faffd640 1 -- 172.21.15.73:0/380516133 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f85fc009ce0 con 0x7f860c131070 2024-08-02T23:40:07.894 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.868+0000 7f78ea7fc640 1 -- 172.21.15.73:0/959741601 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f78f0014940 con 0x7f78f40a4770 2024-08-02T23:40:07.894 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.868+0000 7f78fc119640 1 -- 172.21.15.73:0/959741601 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78f411fdb0 con 0x7f78f40a4770 2024-08-02T23:40:07.895 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.868+0000 7f85faffd640 1 -- 172.21.15.73:0/380516133 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f85fc014620 con 0x7f860c131070 2024-08-02T23:40:07.895 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.868+0000 7f85faffd640 1 -- 172.21.15.73:0/380516133 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f85fc0148c0 con 0x7f860c131070 2024-08-02T23:40:07.895 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.868+0000 7f4b4b159640 1 -- 172.21.15.73:0/3070055702 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4b44070ab0 con 0x7f4b4409ae70 2024-08-02T23:40:07.895 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.869+0000 7f8610f99640 1 -- 172.21.15.73:0/380516133 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f860c11ffb0 con 0x7f860c131070 2024-08-02T23:40:07.896 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.869+0000 7f4b417fa640 1 -- 172.21.15.73:0/3070055702 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 5) ==== 45563+0+0 (crc 0 0 0) 0x7f4b38014b80 con 0x7f4b4409ae70 2024-08-02T23:40:07.896 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.869+0000 7f8610f99640 1 -- 172.21.15.73:0/380516133 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f860c120320 con 0x7f860c131070 2024-08-02T23:40:07.896 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.868+0000 7fdc237fe640 1 -- 172.21.15.73:0/1204902602 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 5) ==== 45563+0+0 (crc 0 0 0) 0x7fdc2c0070b0 con 0x7fdc3c131070 2024-08-02T23:40:07.896 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.869+0000 7fdc237fe640 1 --2- 172.21.15.73:0/1204902602 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7fdc280383e0 0x7fdc2803a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-02T23:40:07.896 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.868+0000 7fdc41599640 1 -- 172.21.15.73:0/1204902602 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdc3c131480 con 0x7fdc3c131070 2024-08-02T23:40:07.896 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.869+0000 7fdc237fe640 1 -- 172.21.15.73:0/1204902602 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 2728+0+0 (crc 0 0 0) 0x7fdc2c047ba0 con 0x7fdc3c131070 2024-08-02T23:40:07.897 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.869+0000 7f4b417fa640 1 --2- 172.21.15.73:0/3070055702 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7f4b280383e0 0x7f4b2803a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-02T23:40:07.897 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.869+0000 7f4b417fa640 1 -- 172.21.15.73:0/3070055702 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 2728+0+0 (crc 0 0 0) 0x7f4b38019040 con 0x7f4b4409ae70 2024-08-02T23:40:07.897 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.869+0000 7f4b43fff640 1 --2- 172.21.15.73:0/3070055702 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7f4b280383e0 0x7f4b2803a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-02T23:40:07.897 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.869+0000 7fdc3a7fc640 1 --2- 172.21.15.73:0/1204902602 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7fdc280383e0 0x7fdc2803a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-02T23:40:07.898 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.869+0000 7f78ea7fc640 1 -- 172.21.15.73:0/959741601 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 5) ==== 45563+0+0 (crc 0 0 0) 0x7f78f0011380 con 0x7f78f40a4770 2024-08-02T23:40:07.898 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.870+0000 7f8610f99640 1 -- 172.21.15.73:0/380516133 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f860c131480 con 0x7f860c131070 2024-08-02T23:40:07.898 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.870+0000 7fdc3a7fc640 1 --2- 172.21.15.73:0/1204902602 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7fdc280383e0 0x7fdc2803a8f0 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-02T23:40:07.898 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.870+0000 7f78ea7fc640 1 --2- 172.21.15.73:0/959741601 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7f78d80383e0 0x7f78d803a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-02T23:40:07.899 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.870+0000 7f78fc119640 1 -- 172.21.15.73:0/959741601 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f78f409aba0 con 0x7f78f40a4770 2024-08-02T23:40:07.899 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.870+0000 7f4b43fff640 1 --2- 172.21.15.73:0/3070055702 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7f4b280383e0 0x7f4b2803a8f0 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-02T23:40:07.899 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.870+0000 7f78f96b4640 1 --2- 172.21.15.73:0/959741601 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7f78d80383e0 0x7f78d803a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-02T23:40:07.899 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.871+0000 7f78f96b4640 1 --2- 172.21.15.73:0/959741601 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7f78d80383e0 0x7f78d803a8f0 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-02T23:40:07.900 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.874+0000 7f4b417fa640 1 -- 172.21.15.73:0/3070055702 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186747 (crc 0 0 0) 0x7f4b38018a90 con 0x7f4b4409ae70 2024-08-02T23:40:07.900 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.879+0000 7fdc237fe640 1 -- 172.21.15.73:0/1204902602 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186747 (crc 0 0 0) 0x7fdc2c00da20 con 0x7fdc3c131070 2024-08-02T23:40:07.900 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.879+0000 7f78ea7fc640 1 -- 172.21.15.73:0/959741601 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 2728+0+0 (crc 0 0 0) 0x7f78f0019040 con 0x7f78f40a4770 2024-08-02T23:40:07.901 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.880+0000 7f85faffd640 1 -- 172.21.15.73:0/380516133 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 5) ==== 45563+0+0 (crc 0 0 0) 0x7f85fc014a20 con 0x7f860c131070 2024-08-02T23:40:07.901 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.880+0000 7f85faffd640 1 --2- 172.21.15.73:0/380516133 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7f85ec0383e0 0x7f85ec03a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-02T23:40:07.901 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.880+0000 7f85faffd640 1 -- 172.21.15.73:0/380516133 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 2728+0+0 (crc 0 0 0) 0x7f85fc049000 con 0x7f860c131070 2024-08-02T23:40:07.901 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.880+0000 7f8609d9b640 1 --2- 172.21.15.73:0/380516133 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7f85ec0383e0 0x7f85ec03a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-02T23:40:07.901 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.881+0000 7f8609d9b640 1 --2- 172.21.15.73:0/380516133 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7f85ec0383e0 0x7f85ec03a8f0 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-02T23:40:07.901 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.884+0000 7f78ea7fc640 1 -- 172.21.15.73:0/959741601 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186747 (crc 0 0 0) 0x7f78f0018d60 con 0x7f78f40a4770 2024-08-02T23:40:07.902 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:07.889+0000 7f85faffd640 1 -- 172.21.15.73:0/380516133 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186747 (crc 0 0 0) 0x7f85fc01aca0 con 0x7f860c131070 2024-08-02T23:40:08.062 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.060+0000 7f4b4b159640 1 -- 172.21.15.73:0/3070055702 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f4b4409c7b0 con 0x7f4b4409ae70 2024-08-02T23:40:08.062 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.060+0000 7f4b417fa640 1 -- 172.21.15.73:0/3070055702 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (crc 0 0 0) 0x7f4b4409c7b0 con 0x7f4b4409ae70 2024-08-02T23:40:08.063 INFO:teuthology.orchestra.run.smithi073.stdout:38654705673 2024-08-02T23:40:08.065 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.063+0000 7f4b4b159640 1 -- 172.21.15.73:0/3070055702 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7f4b280383e0 msgr2=0x7f4b2803a8f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-02T23:40:08.066 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.063+0000 7f4b4b159640 1 --2- 172.21.15.73:0/3070055702 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7f4b280383e0 0x7f4b2803a8f0 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:40:08.066 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.063+0000 7f4b4b159640 1 -- 172.21.15.73:0/3070055702 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f4b4409ae70 msgr2=0x7f4b4411f0f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-02T23:40:08.066 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.063+0000 7f4b4b159640 1 --2- 172.21.15.73:0/3070055702 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f4b4409ae70 0x7f4b4411f0f0 crc :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:40:08.066 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.064+0000 7f4b4b159640 1 -- 172.21.15.73:0/3070055702 shutdown_connections 2024-08-02T23:40:08.066 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.064+0000 7f4b4b159640 1 --2- 172.21.15.73:0/3070055702 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7f4b280383e0 0x7f4b2803a8f0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:40:08.066 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.064+0000 7f4b4b159640 1 --2- 172.21.15.73:0/3070055702 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f4b4409ae70 0x7f4b4411f0f0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:40:08.066 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.064+0000 7f4b4b159640 1 -- 172.21.15.73:0/3070055702 >> 172.21.15.73:0/3070055702 conn(0x7f4b440a1860 msgr2=0x7f4b4406fca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-02T23:40:08.066 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.064+0000 7f4b4b159640 1 -- 172.21.15.73:0/3070055702 shutdown_connections 2024-08-02T23:40:08.066 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.064+0000 7f4b4b159640 1 -- 172.21.15.73:0/3070055702 wait complete. 2024-08-02T23:40:08.067 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.065+0000 7fdc41599640 1 -- 172.21.15.73:0/1204902602 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7fdc3c073220 con 0x7fdc3c131070 2024-08-02T23:40:08.067 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.065+0000 7fdc237fe640 1 -- 172.21.15.73:0/1204902602 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (crc 0 0 0) 0x7fdc2c00d610 con 0x7fdc3c131070 2024-08-02T23:40:08.067 INFO:teuthology.orchestra.run.smithi073.stdout:38654705673 2024-08-02T23:40:08.071 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.069+0000 7fdc41599640 1 -- 172.21.15.73:0/1204902602 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7fdc280383e0 msgr2=0x7fdc2803a8f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-02T23:40:08.071 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.069+0000 7fdc41599640 1 --2- 172.21.15.73:0/1204902602 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7fdc280383e0 0x7fdc2803a8f0 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:40:08.071 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.069+0000 7fdc41599640 1 -- 172.21.15.73:0/1204902602 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fdc3c131070 msgr2=0x7fdc3c11da40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-02T23:40:08.071 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.069+0000 7fdc41599640 1 --2- 172.21.15.73:0/1204902602 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fdc3c131070 0x7fdc3c11da40 crc :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:40:08.071 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.069+0000 7fdc41599640 1 -- 172.21.15.73:0/1204902602 shutdown_connections 2024-08-02T23:40:08.071 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.069+0000 7fdc41599640 1 --2- 172.21.15.73:0/1204902602 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7fdc280383e0 0x7fdc2803a8f0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:40:08.072 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.069+0000 7fdc41599640 1 --2- 172.21.15.73:0/1204902602 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fdc3c131070 0x7fdc3c11da40 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:40:08.072 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.069+0000 7fdc41599640 1 -- 172.21.15.73:0/1204902602 >> 172.21.15.73:0/1204902602 conn(0x7fdc3c06cf90 msgr2=0x7fdc3c132f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-02T23:40:08.072 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.069+0000 7fdc41599640 1 -- 172.21.15.73:0/1204902602 shutdown_connections 2024-08-02T23:40:08.072 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.069+0000 7fdc41599640 1 -- 172.21.15.73:0/1204902602 wait complete. 2024-08-02T23:40:08.074 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.072+0000 7f78fc119640 1 -- 172.21.15.73:0/959741601 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f78f4073220 con 0x7f78f40a4770 2024-08-02T23:40:08.074 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.072+0000 7f78ea7fc640 1 -- 172.21.15.73:0/959741601 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (crc 0 0 0) 0x7f78f0018b80 con 0x7f78f40a4770 2024-08-02T23:40:08.074 INFO:teuthology.orchestra.run.smithi073.stdout:42949672968 2024-08-02T23:40:08.077 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.074+0000 7f78fc119640 1 -- 172.21.15.73:0/959741601 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7f78d80383e0 msgr2=0x7f78d803a8f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-02T23:40:08.077 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.074+0000 7f78fc119640 1 --2- 172.21.15.73:0/959741601 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7f78d80383e0 0x7f78d803a8f0 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:40:08.077 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.074+0000 7f78fc119640 1 -- 172.21.15.73:0/959741601 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f78f40a4770 msgr2=0x7f78f411ea90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-02T23:40:08.077 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.074+0000 7f78fc119640 1 --2- 172.21.15.73:0/959741601 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f78f40a4770 0x7f78f411ea90 crc :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:40:08.077 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.075+0000 7f78fc119640 1 -- 172.21.15.73:0/959741601 shutdown_connections 2024-08-02T23:40:08.077 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.075+0000 7f78fc119640 1 --2- 172.21.15.73:0/959741601 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7f78d80383e0 0x7f78d803a8f0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:40:08.077 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.075+0000 7f78fc119640 1 --2- 172.21.15.73:0/959741601 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f78f40a4770 0x7f78f411ea90 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:40:08.077 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.075+0000 7f78fc119640 1 -- 172.21.15.73:0/959741601 >> 172.21.15.73:0/959741601 conn(0x7f78f40a3510 msgr2=0x7f78f4133030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-02T23:40:08.077 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.075+0000 7f78fc119640 1 -- 172.21.15.73:0/959741601 shutdown_connections 2024-08-02T23:40:08.077 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.075+0000 7f78fc119640 1 -- 172.21.15.73:0/959741601 wait complete. 2024-08-02T23:40:08.078 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705673 got 38654705673 for osd.2 2024-08-02T23:40:08.079 DEBUG:teuthology.parallel:result is None 2024-08-02T23:40:08.080 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.078+0000 7f8610f99640 1 -- 172.21.15.73:0/380516133 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f860c073220 con 0x7f860c131070 2024-08-02T23:40:08.080 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.079+0000 7f85faffd640 1 -- 172.21.15.73:0/380516133 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (crc 0 0 0) 0x7f85fc007e20 con 0x7f860c131070 2024-08-02T23:40:08.081 INFO:teuthology.orchestra.run.smithi073.stdout:38654705673 2024-08-02T23:40:08.083 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.081+0000 7f8610f99640 1 -- 172.21.15.73:0/380516133 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7f85ec0383e0 msgr2=0x7f85ec03a8f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-02T23:40:08.084 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.081+0000 7f8610f99640 1 --2- 172.21.15.73:0/380516133 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7f85ec0383e0 0x7f85ec03a8f0 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:40:08.084 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.081+0000 7f8610f99640 1 -- 172.21.15.73:0/380516133 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f860c131070 msgr2=0x7f860c11da70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-02T23:40:08.084 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.081+0000 7f8610f99640 1 --2- 172.21.15.73:0/380516133 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f860c131070 0x7f860c11da70 crc :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:40:08.084 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.081+0000 7f8610f99640 1 -- 172.21.15.73:0/380516133 shutdown_connections 2024-08-02T23:40:08.084 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.081+0000 7f8610f99640 1 --2- 172.21.15.73:0/380516133 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7f85ec0383e0 0x7f85ec03a8f0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:40:08.084 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.081+0000 7f8610f99640 1 --2- 172.21.15.73:0/380516133 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f860c131070 0x7f860c11da70 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:40:08.084 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.081+0000 7f8610f99640 1 -- 172.21.15.73:0/380516133 >> 172.21.15.73:0/380516133 conn(0x7f860c06cf90 msgr2=0x7f860c1ba620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-02T23:40:08.084 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.082+0000 7f8610f99640 1 -- 172.21.15.73:0/380516133 shutdown_connections 2024-08-02T23:40:08.084 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.082+0000 7f8610f99640 1 -- 172.21.15.73:0/380516133 wait complete. 2024-08-02T23:40:08.085 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705673 got 38654705673 for osd.1 2024-08-02T23:40:08.085 DEBUG:teuthology.parallel:result is None 2024-08-02T23:40:08.089 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672968 got 42949672968 for osd.3 2024-08-02T23:40:08.090 DEBUG:teuthology.parallel:result is None 2024-08-02T23:40:08.095 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705673 got 38654705673 for osd.0 2024-08-02T23:40:08.095 DEBUG:teuthology.parallel:result is None 2024-08-02T23:40:08.095 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-08-02T23:40:08.095 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-08-02T23:40:08.241 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.236+0000 7feab1c2a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-02T23:40:08.241 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.240+0000 7feab1c2a640 1 Processor -- start 2024-08-02T23:40:08.241 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.240+0000 7feab1c2a640 1 -- start start 2024-08-02T23:40:08.243 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.240+0000 7feab1c2a640 1 --2- >> v2:172.21.15.73:3300/0 conn(0x7feaac1339f0 0x7feaac135e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-02T23:40:08.243 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.240+0000 7feab1c2a640 1 -- --> v1:172.21.15.73:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feaac06e4f0 con 0x7feaac12d690 2024-08-02T23:40:08.243 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.240+0000 7feab1c2a640 1 -- --> v2:172.21.15.73:3300/0 -- mon_getmap magic: 0 -- 0x7feaac12d240 con 0x7feaac1339f0 2024-08-02T23:40:08.243 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.241+0000 7feaaaffd640 1 --2- >> v2:172.21.15.73:3300/0 conn(0x7feaac1339f0 0x7feaac135e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-02T23:40:08.243 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.241+0000 7feaab7fe640 1 --1- >> v1:172.21.15.73:6789/0 conn(0x7feaac12d690 0x7feaac12daa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.73:6789/0 says I am v1:172.21.15.73:43458/0 (socket says 172.21.15.73:43458) 2024-08-02T23:40:08.243 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.241+0000 7feaab7fe640 1 -- 172.21.15.73:0/1882708384 learned_addr learned my addr 172.21.15.73:0/1882708384 (peer_addr_for_me v1:172.21.15.73:0/0) 2024-08-02T23:40:08.243 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.241+0000 7feaaaffd640 1 -- 172.21.15.73:0/1882708384 >> v1:172.21.15.73:6789/0 conn(0x7feaac12d690 legacy=0x7feaac12daa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-02T23:40:08.243 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.241+0000 7feaaaffd640 1 -- 172.21.15.73:0/1882708384 --> v2:172.21.15.73:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feaac09e870 con 0x7feaac1339f0 2024-08-02T23:40:08.244 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.241+0000 7feaaa7fc640 1 -- 172.21.15.73:0/1882708384 <== mon.0 v1:172.21.15.73:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2868881071 0 0) 0x7feaac06e4f0 con 0x7feaac12d690 2024-08-02T23:40:08.244 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.241+0000 7feaaaffd640 1 --2- 172.21.15.73:0/1882708384 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7feaac1339f0 0x7feaac135e10 crc :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60ebf4de42840dd5 server_cookie=0 in_seq=0 out_seq=0 2024-08-02T23:40:08.244 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.241+0000 7feaaa7fc640 1 -- 172.21.15.73:0/1882708384 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fea94009ce0 con 0x7feaac1339f0 2024-08-02T23:40:08.244 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.241+0000 7feaaa7fc640 1 -- 172.21.15.73:0/1882708384 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7fea94009e40 con 0x7feaac1339f0 2024-08-02T23:40:08.244 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.241+0000 7feaaa7fc640 1 -- 172.21.15.73:0/1882708384 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fea9400e360 con 0x7feaac1339f0 2024-08-02T23:40:08.245 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.242+0000 7feab1c2a640 1 -- 172.21.15.73:0/1882708384 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7feaac1339f0 msgr2=0x7feaac135e10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-02T23:40:08.245 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.242+0000 7feab1c2a640 1 --2- 172.21.15.73:0/1882708384 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7feaac1339f0 0x7feaac135e10 crc :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:40:08.245 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.242+0000 7feab1c2a640 1 -- 172.21.15.73:0/1882708384 shutdown_connections 2024-08-02T23:40:08.245 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.242+0000 7feab1c2a640 1 --2- 172.21.15.73:0/1882708384 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7feaac1339f0 0x7feaac135e10 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:40:08.245 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.242+0000 7feab1c2a640 1 -- 172.21.15.73:0/1882708384 >> 172.21.15.73:0/1882708384 conn(0x7feaac0a0590 msgr2=0x7feaac0a09a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-02T23:40:08.245 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.242+0000 7feab1c2a640 1 -- 172.21.15.73:0/1882708384 shutdown_connections 2024-08-02T23:40:08.245 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.242+0000 7feab1c2a640 1 -- 172.21.15.73:0/1882708384 wait complete. 2024-08-02T23:40:08.245 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.242+0000 7feab1c2a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-02T23:40:08.245 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.243+0000 7feab1c2a640 1 Processor -- start 2024-08-02T23:40:08.246 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.243+0000 7feab1c2a640 1 -- start start 2024-08-02T23:40:08.247 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.244+0000 7feab1c2a640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7feaac12d690 0x7feaac129b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-02T23:40:08.247 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.244+0000 7feab1c2a640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 -- 0x7feaac11ecf0 con 0x7feaac12d690 2024-08-02T23:40:08.247 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.244+0000 7feaab7fe640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7feaac12d690 0x7feaac129b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-02T23:40:08.248 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.244+0000 7feaab7fe640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7feaac12d690 0x7feaac129b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:44296/0 (socket says 172.21.15.73:44296) 2024-08-02T23:40:08.248 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.244+0000 7feaab7fe640 1 -- 172.21.15.73:0/1723083089 learned_addr learned my addr 172.21.15.73:0/1723083089 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-08-02T23:40:08.248 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.244+0000 7feaab7fe640 1 -- 172.21.15.73:0/1723083089 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feaac11de00 con 0x7feaac12d690 2024-08-02T23:40:08.248 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.244+0000 7feaab7fe640 1 --2- 172.21.15.73:0/1723083089 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7feaac12d690 0x7feaac129b70 crc :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-02T23:40:08.248 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.244+0000 7feaa3fff640 1 -- 172.21.15.73:0/1723083089 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fea9c004af0 con 0x7feaac12d690 2024-08-02T23:40:08.248 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.244+0000 7feaa3fff640 1 -- 172.21.15.73:0/1723083089 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7fea9c004c50 con 0x7feaac12d690 2024-08-02T23:40:08.248 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.244+0000 7feaa3fff640 1 -- 172.21.15.73:0/1723083089 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fea9c007120 con 0x7feaac12d690 2024-08-02T23:40:08.248 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.244+0000 7feab1c2a640 1 -- 172.21.15.73:0/1723083089 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feaac131c70 con 0x7feaac12d690 2024-08-02T23:40:08.248 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.244+0000 7feab1c2a640 1 -- 172.21.15.73:0/1723083089 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feaac073020 con 0x7feaac12d690 2024-08-02T23:40:08.248 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.245+0000 7feaa3fff640 1 -- 172.21.15.73:0/1723083089 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 5) ==== 45563+0+0 (crc 0 0 0) 0x7fea9c0072d0 con 0x7feaac12d690 2024-08-02T23:40:08.248 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.245+0000 7feab1c2a640 1 -- 172.21.15.73:0/1723083089 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feaac11d670 con 0x7feaac12d690 2024-08-02T23:40:08.248 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.245+0000 7feaa3fff640 1 --2- 172.21.15.73:0/1723083089 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7fea8c038390 0x7fea8c03a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-02T23:40:08.248 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.246+0000 7feaa3fff640 1 -- 172.21.15.73:0/1723083089 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 2728+0+0 (crc 0 0 0) 0x7fea9c047b30 con 0x7feaac12d690 2024-08-02T23:40:08.248 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.246+0000 7feaaaffd640 1 --2- 172.21.15.73:0/1723083089 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7fea8c038390 0x7fea8c03a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-02T23:40:08.249 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.246+0000 7feaaaffd640 1 --2- 172.21.15.73:0/1723083089 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7fea8c038390 0x7fea8c03a8a0 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-02T23:40:08.252 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.250+0000 7feaa3fff640 1 -- 172.21.15.73:0/1723083089 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186747 (crc 0 0 0) 0x7fea9c00a4b0 con 0x7feaac12d690 2024-08-02T23:40:08.422 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.421+0000 7feab1c2a640 1 -- 172.21.15.73:0/1723083089 --> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7feaac09a9f0 con 0x7fea8c038390 2024-08-02T23:40:08.424 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.422+0000 7feaa3fff640 1 -- 172.21.15.73:0/1723083089 <== mgr.4100 v2:172.21.15.131:6800/3351532440 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+31209 (crc 0 0 0) 0x7fea9400e510 con 0x7fea8c038390 2024-08-02T23:40:08.424 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-08-02T23:40:08.425 INFO:teuthology.orchestra.run.smithi073.stderr:dumped all 2024-08-02T23:40:08.426 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.424+0000 7feab1c2a640 1 -- 172.21.15.73:0/1723083089 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7fea8c038390 msgr2=0x7fea8c03a8a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-02T23:40:08.426 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.424+0000 7feab1c2a640 1 --2- 172.21.15.73:0/1723083089 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7fea8c038390 0x7fea8c03a8a0 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:40:08.426 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.424+0000 7feab1c2a640 1 -- 172.21.15.73:0/1723083089 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7feaac12d690 msgr2=0x7feaac129b70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-02T23:40:08.426 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.424+0000 7feab1c2a640 1 --2- 172.21.15.73:0/1723083089 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7feaac12d690 0x7feaac129b70 crc :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:40:08.426 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.424+0000 7feab1c2a640 1 -- 172.21.15.73:0/1723083089 shutdown_connections 2024-08-02T23:40:08.426 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.424+0000 7feab1c2a640 1 --2- 172.21.15.73:0/1723083089 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7fea8c038390 0x7fea8c03a8a0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:40:08.426 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.424+0000 7feab1c2a640 1 --2- 172.21.15.73:0/1723083089 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7feaac12d690 0x7feaac129b70 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:40:08.427 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.424+0000 7feab1c2a640 1 -- 172.21.15.73:0/1723083089 >> 172.21.15.73:0/1723083089 conn(0x7feaac0a0590 msgr2=0x7feaac0a09a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-02T23:40:08.427 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.424+0000 7feab1c2a640 1 -- 172.21.15.73:0/1723083089 shutdown_connections 2024-08-02T23:40:08.427 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.424+0000 7feab1c2a640 1 -- 172.21.15.73:0/1723083089 wait complete. 2024-08-02T23:40:08.436 INFO:teuthology.orchestra.run.smithi073.stdout:{"pg_ready":true,"pg_map":{"version":12,"stamp":"2024-08-02T23:40:07.672348+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":209190912,"kb_used":3696,"kb_used_data":3696,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":209187216,"statfs":{"total":214211493888,"available":214207709184,"internally_reserved":0,"allocated":3784704,"data_stored":3784704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-08-02T23:40:06.536112+0000","last_change":"2024-08-02T23:40:06.540330+0000","last_active":"2024-08-02T23:40:06.536112+0000","last_peered":"2024-08-02T23:40:06.536112+0000","last_clean":"2024-08-02T23:40:06.536112+0000","last_became_active":"2024-08-02T23:40:04.552702+0000","last_became_peered":"2024-08-02T23:40:04.552702+0000","last_unstale":"2024-08-02T23:40:06.536112+0000","last_undegraded":"2024-08-02T23:40:06.536112+0000","last_fullsized":"2024-08-02T23:40:06.536112+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-02T23:40:03.497423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-02T23:40:03.497423+0000","last_clean_scrub_stamp":"2024-08-02T23:40:03.497423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-08-02T23:40:06.541178+0000","last_change":"2024-08-02T23:40:06.542935+0000","last_active":"2024-08-02T23:40:06.541178+0000","last_peered":"2024-08-02T23:40:06.541178+0000","last_clean":"2024-08-02T23:40:06.541178+0000","last_became_active":"2024-08-02T23:40:04.561840+0000","last_became_peered":"2024-08-02T23:40:04.561840+0000","last_unstale":"2024-08-02T23:40:06.541178+0000","last_undegraded":"2024-08-02T23:40:06.541178+0000","last_fullsized":"2024-08-02T23:40:06.541178+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-02T23:40:03.497423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-02T23:40:03.497423+0000","last_clean_scrub_stamp":"2024-08-02T23:40:03.497423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-08-02T23:40:06.538133+0000","last_change":"2024-08-02T23:40:06.540945+0000","last_active":"2024-08-02T23:40:06.538133+0000","last_peered":"2024-08-02T23:40:06.538133+0000","last_clean":"2024-08-02T23:40:06.538133+0000","last_became_active":"2024-08-02T23:40:04.560865+0000","last_became_peered":"2024-08-02T23:40:04.560865+0000","last_unstale":"2024-08-02T23:40:06.538133+0000","last_undegraded":"2024-08-02T23:40:06.538133+0000","last_fullsized":"2024-08-02T23:40:06.538133+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-02T23:40:03.497423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-02T23:40:03.497423+0000","last_clean_scrub_stamp":"2024-08-02T23:40:03.497423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-08-02T23:40:06.536532+0000","last_change":"2024-08-02T23:40:06.540508+0000","last_active":"2024-08-02T23:40:06.536532+0000","last_peered":"2024-08-02T23:40:06.536532+0000","last_clean":"2024-08-02T23:40:06.536532+0000","last_became_active":"2024-08-02T23:40:04.553411+0000","last_became_peered":"2024-08-02T23:40:04.553411+0000","last_unstale":"2024-08-02T23:40:06.536532+0000","last_undegraded":"2024-08-02T23:40:06.536532+0000","last_fullsized":"2024-08-02T23:40:06.536532+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-02T23:40:03.497423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-02T23:40:03.497423+0000","last_clean_scrub_stamp":"2024-08-02T23:40:03.497423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"15'1","reported_seq":17,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-08-02T23:40:06.538737+0000","last_change":"2024-08-02T23:40:06.538737+0000","last_active":"2024-08-02T23:40:06.538737+0000","last_peered":"2024-08-02T23:40:06.538737+0000","last_clean":"2024-08-02T23:40:06.538737+0000","last_became_active":"2024-08-02T23:40:04.553959+0000","last_became_peered":"2024-08-02T23:40:04.553959+0000","last_unstale":"2024-08-02T23:40:06.538737+0000","last_undegraded":"2024-08-02T23:40:06.538737+0000","last_fullsized":"2024-08-02T23:40:06.538737+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-02T23:40:03.497423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-02T23:40:03.497423+0000","last_clean_scrub_stamp":"2024-08-02T23:40:03.497423+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-08-02T23:40:06.536931+0000","last_change":"2024-08-02T23:40:06.540125+0000","last_active":"2024-08-02T23:40:06.536931+0000","last_peered":"2024-08-02T23:40:06.536931+0000","last_clean":"2024-08-02T23:40:06.536931+0000","last_became_active":"2024-08-02T23:40:04.556895+0000","last_became_peered":"2024-08-02T23:40:04.556895+0000","last_unstale":"2024-08-02T23:40:06.536931+0000","last_undegraded":"2024-08-02T23:40:06.536931+0000","last_fullsized":"2024-08-02T23:40:06.536931+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-02T23:40:03.497423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-02T23:40:03.497423+0000","last_clean_scrub_stamp":"2024-08-02T23:40:03.497423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-08-02T23:40:06.536368+0000","last_change":"2024-08-02T23:40:06.538748+0000","last_active":"2024-08-02T23:40:06.536368+0000","last_peered":"2024-08-02T23:40:06.536368+0000","last_clean":"2024-08-02T23:40:06.536368+0000","last_became_active":"2024-08-02T23:40:04.561398+0000","last_became_peered":"2024-08-02T23:40:04.561398+0000","last_unstale":"2024-08-02T23:40:06.536368+0000","last_undegraded":"2024-08-02T23:40:06.536368+0000","last_fullsized":"2024-08-02T23:40:06.536368+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-02T23:40:03.497423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-02T23:40:03.497423+0000","last_clean_scrub_stamp":"2024-08-02T23:40:03.497423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"14'1","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-08-02T23:40:06.540070+0000","last_change":"2024-08-02T23:40:06.541978+0000","last_active":"2024-08-02T23:40:06.540070+0000","last_peered":"2024-08-02T23:40:06.540070+0000","last_clean":"2024-08-02T23:40:06.540070+0000","last_became_active":"2024-08-02T23:40:04.550062+0000","last_became_peered":"2024-08-02T23:40:04.550062+0000","last_unstale":"2024-08-02T23:40:06.540070+0000","last_undegraded":"2024-08-02T23:40:06.540070+0000","last_fullsized":"2024-08-02T23:40:06.540070+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-02T23:40:03.497423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-02T23:40:03.497423+0000","last_clean_scrub_stamp":"2024-08-02T23:40:03.497423+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":150,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-08-02T23:40:06.536600+0000","last_change":"2024-08-02T23:40:00.557074+0000","last_active":"2024-08-02T23:40:06.536600+0000","last_peered":"2024-08-02T23:40:06.536600+0000","last_clean":"2024-08-02T23:40:06.536600+0000","last_became_active":"2024-08-02T23:40:00.546868+0000","last_became_peered":"2024-08-02T23:40:00.546868+0000","last_unstale":"2024-08-02T23:40:06.536600+0000","last_undegraded":"2024-08-02T23:40:06.536600+0000","last_fullsized":"2024-08-02T23:40:06.536600+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-02T23:39:59.483999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-02T23:39:59.483999+0000","last_clean_scrub_stamp":"2024-08-02T23:39:59.483999+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":214211493888,"available":214207709184,"internally_reserved":0,"allocated":3784704,"data_stored":3784704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":16,"acting":16,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53552873472,"available":53551665152,"internally_reserved":0,"allocated":1208320,"data_stored":1208320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":3,"up_from":10,"seq":42949672968,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":1180,"kb_used_data":1180,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52296548,"statfs":{"total":53552873472,"available":53551665152,"internally_reserved":0,"allocated":1208320,"data_stored":1208320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705673,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":540,"kb_used_data":540,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52297188,"statfs":{"total":53552873472,"available":53552320512,"internally_reserved":0,"allocated":552960,"data_stored":552960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705673,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":536,"kb_used_data":536,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52297192,"statfs":{"total":53552873472,"available":53552324608,"internally_reserved":0,"allocated":548864,"data_stored":548864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705673,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":1440,"kb_used_data":1440,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52296288,"statfs":{"total":53552873472,"available":53551398912,"internally_reserved":0,"allocated":1474560,"data_stored":1474560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":3,"total":53552873472,"available":53551665152,"internally_reserved":0,"allocated":1208320,"data_stored":1208320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53552873472,"available":53551398912,"internally_reserved":0,"allocated":1474560,"data_stored":1474560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53552873472,"available":53552324608,"internally_reserved":0,"allocated":548864,"data_stored":548864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53552873472,"available":53552320512,"internally_reserved":0,"allocated":552960,"data_stored":552960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":53552873472,"available":53551665152,"internally_reserved":0,"allocated":1208320,"data_stored":1208320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-02T23:40:08.437 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-08-02T23:40:08.542 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.536+0000 7f555c759640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-02T23:40:08.542 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.539+0000 7f555c759640 1 Processor -- start 2024-08-02T23:40:08.542 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.540+0000 7f555c759640 1 -- start start 2024-08-02T23:40:08.544 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.540+0000 7f555c759640 1 --2- >> v2:172.21.15.73:3300/0 conn(0x7f5554134350 0x7f5554136770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-02T23:40:08.544 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.540+0000 7f555c759640 1 -- --> v1:172.21.15.73:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f555406fef0 con 0x7f555406e880 2024-08-02T23:40:08.544 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.540+0000 7f555c759640 1 -- --> v2:172.21.15.73:3300/0 -- mon_getmap magic: 0 -- 0x7f555409e5d0 con 0x7f5554134350 2024-08-02T23:40:08.544 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.540+0000 7f5559cf4640 1 --2- >> v2:172.21.15.73:3300/0 conn(0x7f5554134350 0x7f5554136770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-02T23:40:08.544 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.540+0000 7f555a4f5640 1 --1- >> v1:172.21.15.73:6789/0 conn(0x7f555406e880 0x7f555406ec90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.73:6789/0 says I am v1:172.21.15.73:43464/0 (socket says 172.21.15.73:43464) 2024-08-02T23:40:08.544 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.541+0000 7f555a4f5640 1 -- 172.21.15.73:0/701666721 learned_addr learned my addr 172.21.15.73:0/701666721 (peer_addr_for_me v1:172.21.15.73:0/0) 2024-08-02T23:40:08.544 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.541+0000 7f5559cf4640 1 -- 172.21.15.73:0/701666721 >> v1:172.21.15.73:6789/0 conn(0x7f555406e880 legacy=0x7f555406ec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-02T23:40:08.544 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.541+0000 7f5559cf4640 1 -- 172.21.15.73:0/701666721 --> v2:172.21.15.73:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f555406e430 con 0x7f5554134350 2024-08-02T23:40:08.544 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.541+0000 7f5559cf4640 1 --2- 172.21.15.73:0/701666721 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5554134350 0x7f5554136770 crc :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58d4b17f8450ef12 server_cookie=0 in_seq=0 out_seq=0 2024-08-02T23:40:08.544 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.541+0000 7f55594f3640 1 -- 172.21.15.73:0/701666721 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f5550009ce0 con 0x7f5554134350 2024-08-02T23:40:08.545 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.541+0000 7f55594f3640 1 -- 172.21.15.73:0/701666721 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f5550009e40 con 0x7f5554134350 2024-08-02T23:40:08.545 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.541+0000 7f55594f3640 1 -- 172.21.15.73:0/701666721 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f555000e360 con 0x7f5554134350 2024-08-02T23:40:08.545 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.542+0000 7f555c759640 1 -- 172.21.15.73:0/701666721 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5554134350 msgr2=0x7f5554136770 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-02T23:40:08.545 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.542+0000 7f555c759640 1 --2- 172.21.15.73:0/701666721 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5554134350 0x7f5554136770 crc :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:40:08.545 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.542+0000 7f555c759640 1 -- 172.21.15.73:0/701666721 shutdown_connections 2024-08-02T23:40:08.545 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.542+0000 7f555c759640 1 --2- 172.21.15.73:0/701666721 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5554134350 0x7f5554136770 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:40:08.545 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.542+0000 7f555c759640 1 -- 172.21.15.73:0/701666721 >> 172.21.15.73:0/701666721 conn(0x7f55540a0590 msgr2=0x7f55540a09a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-02T23:40:08.545 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.542+0000 7f555c759640 1 -- 172.21.15.73:0/701666721 shutdown_connections 2024-08-02T23:40:08.545 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.542+0000 7f555c759640 1 -- 172.21.15.73:0/701666721 wait complete. 2024-08-02T23:40:08.545 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.542+0000 7f555c759640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-02T23:40:08.547 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.543+0000 7f555c759640 1 Processor -- start 2024-08-02T23:40:08.547 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.543+0000 7f555c759640 1 -- start start 2024-08-02T23:40:08.547 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.544+0000 7f555c759640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f555406e880 0x7f55541209d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-02T23:40:08.547 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.544+0000 7f555c759640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 -- 0x7f55541222b0 con 0x7f555406e880 2024-08-02T23:40:08.548 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.544+0000 7f555a4f5640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f555406e880 0x7f55541209d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-02T23:40:08.548 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.544+0000 7f555a4f5640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f555406e880 0x7f55541209d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:44314/0 (socket says 172.21.15.73:44314) 2024-08-02T23:40:08.548 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.544+0000 7f555a4f5640 1 -- 172.21.15.73:0/3281186481 learned_addr learned my addr 172.21.15.73:0/3281186481 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-08-02T23:40:08.548 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.544+0000 7f555a4f5640 1 -- 172.21.15.73:0/3281186481 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55541239c0 con 0x7f555406e880 2024-08-02T23:40:08.548 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.544+0000 7f555a4f5640 1 --2- 172.21.15.73:0/3281186481 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f555406e880 0x7f55541209d0 crc :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-02T23:40:08.548 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.544+0000 7f5546ffd640 1 -- 172.21.15.73:0/3281186481 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f55480059d0 con 0x7f555406e880 2024-08-02T23:40:08.548 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.544+0000 7f5546ffd640 1 -- 172.21.15.73:0/3281186481 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f5548005b30 con 0x7f555406e880 2024-08-02T23:40:08.548 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.544+0000 7f555c759640 1 -- 172.21.15.73:0/3281186481 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5554131d20 con 0x7f555406e880 2024-08-02T23:40:08.548 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.544+0000 7f5546ffd640 1 -- 172.21.15.73:0/3281186481 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f5548005dd0 con 0x7f555406e880 2024-08-02T23:40:08.548 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.544+0000 7f555c759640 1 -- 172.21.15.73:0/3281186481 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f555409d440 con 0x7f555406e880 2024-08-02T23:40:08.548 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.545+0000 7f5546ffd640 1 -- 172.21.15.73:0/3281186481 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 5) ==== 45563+0+0 (crc 0 0 0) 0x7f55480070b0 con 0x7f555406e880 2024-08-02T23:40:08.548 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.545+0000 7f555c759640 1 -- 172.21.15.73:0/3281186481 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f55540a2320 con 0x7f555406e880 2024-08-02T23:40:08.548 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.545+0000 7f5546ffd640 1 --2- 172.21.15.73:0/3281186481 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7f5538038390 0x7f553803a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-02T23:40:08.549 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.545+0000 7f5546ffd640 1 -- 172.21.15.73:0/3281186481 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 2728+0+0 (crc 0 0 0) 0x7f5548047b50 con 0x7f555406e880 2024-08-02T23:40:08.549 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.546+0000 7f5559cf4640 1 --2- 172.21.15.73:0/3281186481 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7f5538038390 0x7f553803a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-02T23:40:08.549 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.546+0000 7f5559cf4640 1 --2- 172.21.15.73:0/3281186481 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7f5538038390 0x7f553803a8a0 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-02T23:40:08.552 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.550+0000 7f5546ffd640 1 -- 172.21.15.73:0/3281186481 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186747 (crc 0 0 0) 0x7f554800d330 con 0x7f555406e880 2024-08-02T23:40:08.724 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.723+0000 7f555c759640 1 -- 172.21.15.73:0/3281186481 --> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f555406f3a0 con 0x7f5538038390 2024-08-02T23:40:08.725 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.724+0000 7f5546ffd640 1 -- 172.21.15.73:0/3281186481 <== mgr.4100 v2:172.21.15.131:6800/3351532440 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+31209 (crc 0 0 0) 0x7f5550013400 con 0x7f5538038390 2024-08-02T23:40:08.726 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-08-02T23:40:08.727 INFO:teuthology.orchestra.run.smithi073.stderr:dumped all 2024-08-02T23:40:08.728 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.726+0000 7f555c759640 1 -- 172.21.15.73:0/3281186481 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7f5538038390 msgr2=0x7f553803a8a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-02T23:40:08.728 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.726+0000 7f555c759640 1 --2- 172.21.15.73:0/3281186481 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7f5538038390 0x7f553803a8a0 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:40:08.728 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.726+0000 7f555c759640 1 -- 172.21.15.73:0/3281186481 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f555406e880 msgr2=0x7f55541209d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-02T23:40:08.728 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.726+0000 7f555c759640 1 --2- 172.21.15.73:0/3281186481 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f555406e880 0x7f55541209d0 crc :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:40:08.728 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.726+0000 7f555c759640 1 -- 172.21.15.73:0/3281186481 shutdown_connections 2024-08-02T23:40:08.728 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.726+0000 7f555c759640 1 --2- 172.21.15.73:0/3281186481 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7f5538038390 0x7f553803a8a0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:40:08.729 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.726+0000 7f555c759640 1 --2- 172.21.15.73:0/3281186481 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f555406e880 0x7f55541209d0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:40:08.729 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.726+0000 7f555c759640 1 -- 172.21.15.73:0/3281186481 >> 172.21.15.73:0/3281186481 conn(0x7f55540a0590 msgr2=0x7f55540720e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-02T23:40:08.729 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.726+0000 7f555c759640 1 -- 172.21.15.73:0/3281186481 shutdown_connections 2024-08-02T23:40:08.729 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.726+0000 7f555c759640 1 -- 172.21.15.73:0/3281186481 wait complete. 2024-08-02T23:40:08.738 INFO:teuthology.orchestra.run.smithi073.stdout:{"pg_ready":true,"pg_map":{"version":12,"stamp":"2024-08-02T23:40:07.672348+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":209190912,"kb_used":3696,"kb_used_data":3696,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":209187216,"statfs":{"total":214211493888,"available":214207709184,"internally_reserved":0,"allocated":3784704,"data_stored":3784704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-08-02T23:40:06.536112+0000","last_change":"2024-08-02T23:40:06.540330+0000","last_active":"2024-08-02T23:40:06.536112+0000","last_peered":"2024-08-02T23:40:06.536112+0000","last_clean":"2024-08-02T23:40:06.536112+0000","last_became_active":"2024-08-02T23:40:04.552702+0000","last_became_peered":"2024-08-02T23:40:04.552702+0000","last_unstale":"2024-08-02T23:40:06.536112+0000","last_undegraded":"2024-08-02T23:40:06.536112+0000","last_fullsized":"2024-08-02T23:40:06.536112+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-02T23:40:03.497423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-02T23:40:03.497423+0000","last_clean_scrub_stamp":"2024-08-02T23:40:03.497423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-08-02T23:40:06.541178+0000","last_change":"2024-08-02T23:40:06.542935+0000","last_active":"2024-08-02T23:40:06.541178+0000","last_peered":"2024-08-02T23:40:06.541178+0000","last_clean":"2024-08-02T23:40:06.541178+0000","last_became_active":"2024-08-02T23:40:04.561840+0000","last_became_peered":"2024-08-02T23:40:04.561840+0000","last_unstale":"2024-08-02T23:40:06.541178+0000","last_undegraded":"2024-08-02T23:40:06.541178+0000","last_fullsized":"2024-08-02T23:40:06.541178+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-02T23:40:03.497423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-02T23:40:03.497423+0000","last_clean_scrub_stamp":"2024-08-02T23:40:03.497423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-08-02T23:40:06.538133+0000","last_change":"2024-08-02T23:40:06.540945+0000","last_active":"2024-08-02T23:40:06.538133+0000","last_peered":"2024-08-02T23:40:06.538133+0000","last_clean":"2024-08-02T23:40:06.538133+0000","last_became_active":"2024-08-02T23:40:04.560865+0000","last_became_peered":"2024-08-02T23:40:04.560865+0000","last_unstale":"2024-08-02T23:40:06.538133+0000","last_undegraded":"2024-08-02T23:40:06.538133+0000","last_fullsized":"2024-08-02T23:40:06.538133+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-02T23:40:03.497423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-02T23:40:03.497423+0000","last_clean_scrub_stamp":"2024-08-02T23:40:03.497423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-08-02T23:40:06.536532+0000","last_change":"2024-08-02T23:40:06.540508+0000","last_active":"2024-08-02T23:40:06.536532+0000","last_peered":"2024-08-02T23:40:06.536532+0000","last_clean":"2024-08-02T23:40:06.536532+0000","last_became_active":"2024-08-02T23:40:04.553411+0000","last_became_peered":"2024-08-02T23:40:04.553411+0000","last_unstale":"2024-08-02T23:40:06.536532+0000","last_undegraded":"2024-08-02T23:40:06.536532+0000","last_fullsized":"2024-08-02T23:40:06.536532+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-02T23:40:03.497423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-02T23:40:03.497423+0000","last_clean_scrub_stamp":"2024-08-02T23:40:03.497423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"15'1","reported_seq":17,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-08-02T23:40:06.538737+0000","last_change":"2024-08-02T23:40:06.538737+0000","last_active":"2024-08-02T23:40:06.538737+0000","last_peered":"2024-08-02T23:40:06.538737+0000","last_clean":"2024-08-02T23:40:06.538737+0000","last_became_active":"2024-08-02T23:40:04.553959+0000","last_became_peered":"2024-08-02T23:40:04.553959+0000","last_unstale":"2024-08-02T23:40:06.538737+0000","last_undegraded":"2024-08-02T23:40:06.538737+0000","last_fullsized":"2024-08-02T23:40:06.538737+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-02T23:40:03.497423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-02T23:40:03.497423+0000","last_clean_scrub_stamp":"2024-08-02T23:40:03.497423+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-08-02T23:40:06.536931+0000","last_change":"2024-08-02T23:40:06.540125+0000","last_active":"2024-08-02T23:40:06.536931+0000","last_peered":"2024-08-02T23:40:06.536931+0000","last_clean":"2024-08-02T23:40:06.536931+0000","last_became_active":"2024-08-02T23:40:04.556895+0000","last_became_peered":"2024-08-02T23:40:04.556895+0000","last_unstale":"2024-08-02T23:40:06.536931+0000","last_undegraded":"2024-08-02T23:40:06.536931+0000","last_fullsized":"2024-08-02T23:40:06.536931+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-02T23:40:03.497423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-02T23:40:03.497423+0000","last_clean_scrub_stamp":"2024-08-02T23:40:03.497423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-08-02T23:40:06.536368+0000","last_change":"2024-08-02T23:40:06.538748+0000","last_active":"2024-08-02T23:40:06.536368+0000","last_peered":"2024-08-02T23:40:06.536368+0000","last_clean":"2024-08-02T23:40:06.536368+0000","last_became_active":"2024-08-02T23:40:04.561398+0000","last_became_peered":"2024-08-02T23:40:04.561398+0000","last_unstale":"2024-08-02T23:40:06.536368+0000","last_undegraded":"2024-08-02T23:40:06.536368+0000","last_fullsized":"2024-08-02T23:40:06.536368+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-02T23:40:03.497423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-02T23:40:03.497423+0000","last_clean_scrub_stamp":"2024-08-02T23:40:03.497423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"14'1","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-08-02T23:40:06.540070+0000","last_change":"2024-08-02T23:40:06.541978+0000","last_active":"2024-08-02T23:40:06.540070+0000","last_peered":"2024-08-02T23:40:06.540070+0000","last_clean":"2024-08-02T23:40:06.540070+0000","last_became_active":"2024-08-02T23:40:04.550062+0000","last_became_peered":"2024-08-02T23:40:04.550062+0000","last_unstale":"2024-08-02T23:40:06.540070+0000","last_undegraded":"2024-08-02T23:40:06.540070+0000","last_fullsized":"2024-08-02T23:40:06.540070+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-02T23:40:03.497423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-02T23:40:03.497423+0000","last_clean_scrub_stamp":"2024-08-02T23:40:03.497423+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":150,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-08-02T23:40:06.536600+0000","last_change":"2024-08-02T23:40:00.557074+0000","last_active":"2024-08-02T23:40:06.536600+0000","last_peered":"2024-08-02T23:40:06.536600+0000","last_clean":"2024-08-02T23:40:06.536600+0000","last_became_active":"2024-08-02T23:40:00.546868+0000","last_became_peered":"2024-08-02T23:40:00.546868+0000","last_unstale":"2024-08-02T23:40:06.536600+0000","last_undegraded":"2024-08-02T23:40:06.536600+0000","last_fullsized":"2024-08-02T23:40:06.536600+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-02T23:39:59.483999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-02T23:39:59.483999+0000","last_clean_scrub_stamp":"2024-08-02T23:39:59.483999+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":214211493888,"available":214207709184,"internally_reserved":0,"allocated":3784704,"data_stored":3784704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":16,"acting":16,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53552873472,"available":53551665152,"internally_reserved":0,"allocated":1208320,"data_stored":1208320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":3,"up_from":10,"seq":42949672968,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":1180,"kb_used_data":1180,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52296548,"statfs":{"total":53552873472,"available":53551665152,"internally_reserved":0,"allocated":1208320,"data_stored":1208320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705673,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":540,"kb_used_data":540,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52297188,"statfs":{"total":53552873472,"available":53552320512,"internally_reserved":0,"allocated":552960,"data_stored":552960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705673,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":536,"kb_used_data":536,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52297192,"statfs":{"total":53552873472,"available":53552324608,"internally_reserved":0,"allocated":548864,"data_stored":548864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705673,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":1440,"kb_used_data":1440,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52296288,"statfs":{"total":53552873472,"available":53551398912,"internally_reserved":0,"allocated":1474560,"data_stored":1474560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":3,"total":53552873472,"available":53551665152,"internally_reserved":0,"allocated":1208320,"data_stored":1208320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53552873472,"available":53551398912,"internally_reserved":0,"allocated":1474560,"data_stored":1474560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53552873472,"available":53552324608,"internally_reserved":0,"allocated":548864,"data_stored":548864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53552873472,"available":53552320512,"internally_reserved":0,"allocated":552960,"data_stored":552960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":53552873472,"available":53551665152,"internally_reserved":0,"allocated":1208320,"data_stored":1208320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-02T23:40:08.739 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-08-02T23:40:08.739 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-08-02T23:40:08.739 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-08-02T23:40:08.740 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-08-02T23:40:08.843 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.839+0000 7f1176a2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-02T23:40:08.844 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.842+0000 7f1176a2d640 1 Processor -- start 2024-08-02T23:40:08.844 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.842+0000 7f1176a2d640 1 -- start start 2024-08-02T23:40:08.845 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.843+0000 7f1176a2d640 1 --2- >> v2:172.21.15.73:3300/0 conn(0x7f11701308e0 0x7f1170130cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-02T23:40:08.845 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.843+0000 7f1176a2d640 1 -- --> v1:172.21.15.73:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f117009be00 con 0x7f1170131230 2024-08-02T23:40:08.845 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.843+0000 7f1176a2d640 1 -- --> v2:172.21.15.73:3300/0 -- mon_getmap magic: 0 -- 0x7f117009c030 con 0x7f11701308e0 2024-08-02T23:40:08.845 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.843+0000 7f116ffff640 1 --2- >> v2:172.21.15.73:3300/0 conn(0x7f11701308e0 0x7f1170130cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-02T23:40:08.845 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.843+0000 7f116f7fe640 1 --1- >> v1:172.21.15.73:6789/0 conn(0x7f1170131230 0x7f117009a750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.73:6789/0 says I am v1:172.21.15.73:43474/0 (socket says 172.21.15.73:43474) 2024-08-02T23:40:08.846 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.843+0000 7f116f7fe640 1 -- 172.21.15.73:0/2557982956 learned_addr learned my addr 172.21.15.73:0/2557982956 (peer_addr_for_me v1:172.21.15.73:0/0) 2024-08-02T23:40:08.846 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.843+0000 7f116ffff640 1 -- 172.21.15.73:0/2557982956 >> v1:172.21.15.73:6789/0 conn(0x7f1170131230 legacy=0x7f117009a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-02T23:40:08.846 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.843+0000 7f116ffff640 1 -- 172.21.15.73:0/2557982956 --> v2:172.21.15.73:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f117006ebf0 con 0x7f11701308e0 2024-08-02T23:40:08.846 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.843+0000 7f116ffff640 1 --2- 172.21.15.73:0/2557982956 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f11701308e0 0x7f1170130cf0 crc :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9675a8f7b37c2591 server_cookie=0 in_seq=0 out_seq=0 2024-08-02T23:40:08.846 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.843+0000 7f116effd640 1 -- 172.21.15.73:0/2557982956 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f116000ac70 con 0x7f11701308e0 2024-08-02T23:40:08.846 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.844+0000 7f116effd640 1 -- 172.21.15.73:0/2557982956 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f116000add0 con 0x7f11701308e0 2024-08-02T23:40:08.846 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.844+0000 7f116effd640 1 -- 172.21.15.73:0/2557982956 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f116000f470 con 0x7f11701308e0 2024-08-02T23:40:08.847 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.844+0000 7f1176a2d640 1 -- 172.21.15.73:0/2557982956 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f11701308e0 msgr2=0x7f1170130cf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-02T23:40:08.847 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.844+0000 7f1176a2d640 1 --2- 172.21.15.73:0/2557982956 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f11701308e0 0x7f1170130cf0 crc :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:40:08.847 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.844+0000 7f1176a2d640 1 -- 172.21.15.73:0/2557982956 shutdown_connections 2024-08-02T23:40:08.847 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.844+0000 7f1176a2d640 1 --2- 172.21.15.73:0/2557982956 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f11701308e0 0x7f1170130cf0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:40:08.847 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.844+0000 7f1176a2d640 1 -- 172.21.15.73:0/2557982956 >> 172.21.15.73:0/2557982956 conn(0x7f117009ee30 msgr2=0x7f117009f240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-02T23:40:08.847 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.844+0000 7f1176a2d640 1 -- 172.21.15.73:0/2557982956 shutdown_connections 2024-08-02T23:40:08.848 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.844+0000 7f1176a2d640 1 -- 172.21.15.73:0/2557982956 wait complete. 2024-08-02T23:40:08.848 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.844+0000 7f1176a2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-02T23:40:08.848 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.845+0000 7f1176a2d640 1 Processor -- start 2024-08-02T23:40:08.848 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.846+0000 7f1176a2d640 1 -- start start 2024-08-02T23:40:08.850 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.846+0000 7f1176a2d640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f11701308e0 0x7f117008bf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-02T23:40:08.850 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.846+0000 7f1176a2d640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 -- 0x7f1170098f80 con 0x7f11701308e0 2024-08-02T23:40:08.850 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.846+0000 7f116ffff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f11701308e0 0x7f117008bf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-02T23:40:08.850 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.846+0000 7f116ffff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f11701308e0 0x7f117008bf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:44338/0 (socket says 172.21.15.73:44338) 2024-08-02T23:40:08.850 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.846+0000 7f116ffff640 1 -- 172.21.15.73:0/127365097 learned_addr learned my addr 172.21.15.73:0/127365097 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-08-02T23:40:08.850 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.846+0000 7f116ffff640 1 -- 172.21.15.73:0/127365097 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f117009a520 con 0x7f11701308e0 2024-08-02T23:40:08.850 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.847+0000 7f116ffff640 1 --2- 172.21.15.73:0/127365097 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f11701308e0 0x7f117008bf80 crc :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-02T23:40:08.850 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.847+0000 7f116cff9640 1 -- 172.21.15.73:0/127365097 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f1160011420 con 0x7f11701308e0 2024-08-02T23:40:08.850 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.847+0000 7f116cff9640 1 -- 172.21.15.73:0/127365097 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f1160011580 con 0x7f11701308e0 2024-08-02T23:40:08.850 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.847+0000 7f116cff9640 1 -- 172.21.15.73:0/127365097 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f1160011820 con 0x7f11701308e0 2024-08-02T23:40:08.850 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.847+0000 7f1176a2d640 1 -- 172.21.15.73:0/127365097 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f117008da40 con 0x7f11701308e0 2024-08-02T23:40:08.850 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.847+0000 7f1176a2d640 1 -- 172.21.15.73:0/127365097 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1170098c70 con 0x7f11701308e0 2024-08-02T23:40:08.850 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.848+0000 7f116cff9640 1 -- 172.21.15.73:0/127365097 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 5) ==== 45563+0+0 (crc 0 0 0) 0x7f11600119d0 con 0x7f11701308e0 2024-08-02T23:40:08.851 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.848+0000 7f116cff9640 1 --2- 172.21.15.73:0/127365097 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7f115c038390 0x7f115c03a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-02T23:40:08.851 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.848+0000 7f116cff9640 1 -- 172.21.15.73:0/127365097 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 2728+0+0 (crc 0 0 0) 0x7f1160018040 con 0x7f11701308e0 2024-08-02T23:40:08.851 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.848+0000 7f1176a2d640 1 -- 172.21.15.73:0/127365097 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f117008e980 con 0x7f11701308e0 2024-08-02T23:40:08.851 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.848+0000 7f116f7fe640 1 --2- 172.21.15.73:0/127365097 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7f115c038390 0x7f115c03a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-02T23:40:08.851 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.849+0000 7f116f7fe640 1 --2- 172.21.15.73:0/127365097 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7f115c038390 0x7f115c03a8a0 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-02T23:40:08.854 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:08.853+0000 7f116cff9640 1 -- 172.21.15.73:0/127365097 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186747 (crc 0 0 0) 0x7f116000cb70 con 0x7f11701308e0 2024-08-02T23:40:09.067 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:09.065+0000 7f1176a2d640 1 -- 172.21.15.73:0/127365097 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f117008d370 con 0x7f11701308e0 2024-08-02T23:40:09.067 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:09.065+0000 7f116cff9640 1 -- 172.21.15.73:0/127365097 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (crc 0 0 0) 0x7f116000c760 con 0x7f11701308e0 2024-08-02T23:40:09.067 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-08-02T23:40:09.067 INFO:teuthology.orchestra.run.smithi073.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-08-02T23:40:09.069 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:09.067+0000 7f1176a2d640 1 -- 172.21.15.73:0/127365097 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7f115c038390 msgr2=0x7f115c03a8a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-02T23:40:09.069 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:09.067+0000 7f1176a2d640 1 --2- 172.21.15.73:0/127365097 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7f115c038390 0x7f115c03a8a0 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:40:09.069 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:09.067+0000 7f1176a2d640 1 -- 172.21.15.73:0/127365097 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f11701308e0 msgr2=0x7f117008bf80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-02T23:40:09.069 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:09.067+0000 7f1176a2d640 1 --2- 172.21.15.73:0/127365097 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f11701308e0 0x7f117008bf80 crc :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:40:09.069 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:09.067+0000 7f1176a2d640 1 -- 172.21.15.73:0/127365097 shutdown_connections 2024-08-02T23:40:09.069 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:09.067+0000 7f1176a2d640 1 --2- 172.21.15.73:0/127365097 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7f115c038390 0x7f115c03a8a0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:40:09.069 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:09.067+0000 7f1176a2d640 1 --2- 172.21.15.73:0/127365097 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f11701308e0 0x7f117008bf80 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:40:09.070 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:09.067+0000 7f1176a2d640 1 -- 172.21.15.73:0/127365097 >> 172.21.15.73:0/127365097 conn(0x7f117009ee30 msgr2=0x7f117009d750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-02T23:40:09.070 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:09.067+0000 7f1176a2d640 1 -- 172.21.15.73:0/127365097 shutdown_connections 2024-08-02T23:40:09.070 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:40:09.067+0000 7f1176a2d640 1 -- 172.21.15.73:0/127365097 wait complete. 2024-08-02T23:40:09.079 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-08-02T23:40:09.080 INFO:teuthology.run_tasks:Running task workunit... 2024-08-02T23:40:09.095 INFO:tasks.workunit:Pulling workunits from ref 26c3fb8e197dcf7a49a54d1f4c8a7362ee35a8ea 2024-08-02T23:40:09.095 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-08-02T23:40:09.095 DEBUG:teuthology.orchestra.run.smithi073:> stat -- /home/ubuntu/cephtest/mnt.0 2024-08-02T23:40:09.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-02T23:40:09.111 INFO:teuthology.orchestra.run.smithi073.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-08-02T23:40:09.111 DEBUG:teuthology.orchestra.run.smithi073:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-08-02T23:40:09.168 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-08-02T23:40:09.168 DEBUG:teuthology.orchestra.run.smithi073:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-08-02T23:40:09.222 INFO:tasks.workunit:timeout=3h 2024-08-02T23:40:09.223 INFO:tasks.workunit:cleanup=True 2024-08-02T23:40:09.223 DEBUG:teuthology.orchestra.run.smithi073:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/ceph/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 26c3fb8e197dcf7a49a54d1f4c8a7362ee35a8ea 2024-08-02T23:40:09.278 INFO:tasks.workunit.client.0.smithi073.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-08-02T23:40:56.171 INFO:tasks.workunit.client.0.smithi073.stderr:Note: switching to '26c3fb8e197dcf7a49a54d1f4c8a7362ee35a8ea'. 2024-08-02T23:40:56.171 INFO:tasks.workunit.client.0.smithi073.stderr: 2024-08-02T23:40:56.171 INFO:tasks.workunit.client.0.smithi073.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-08-02T23:40:56.171 INFO:tasks.workunit.client.0.smithi073.stderr:changes and commit them, and you can discard any commits you make in this 2024-08-02T23:40:56.172 INFO:tasks.workunit.client.0.smithi073.stderr:state without impacting any branches by switching back to a branch. 2024-08-02T23:40:56.172 INFO:tasks.workunit.client.0.smithi073.stderr: 2024-08-02T23:40:56.172 INFO:tasks.workunit.client.0.smithi073.stderr:If you want to create a new branch to retain commits you create, you may 2024-08-02T23:40:56.172 INFO:tasks.workunit.client.0.smithi073.stderr:do so (now or later) by using -c with the switch command. Example: 2024-08-02T23:40:56.172 INFO:tasks.workunit.client.0.smithi073.stderr: 2024-08-02T23:40:56.172 INFO:tasks.workunit.client.0.smithi073.stderr: git switch -c 2024-08-02T23:40:56.172 INFO:tasks.workunit.client.0.smithi073.stderr: 2024-08-02T23:40:56.172 INFO:tasks.workunit.client.0.smithi073.stderr:Or undo this operation with: 2024-08-02T23:40:56.172 INFO:tasks.workunit.client.0.smithi073.stderr: 2024-08-02T23:40:56.172 INFO:tasks.workunit.client.0.smithi073.stderr: git switch - 2024-08-02T23:40:56.172 INFO:tasks.workunit.client.0.smithi073.stderr: 2024-08-02T23:40:56.173 INFO:tasks.workunit.client.0.smithi073.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-08-02T23:40:56.173 INFO:tasks.workunit.client.0.smithi073.stderr: 2024-08-02T23:40:56.173 INFO:tasks.workunit.client.0.smithi073.stderr:HEAD is now at 26c3fb8e197 Merge pull request #58932 from cbodley/wip-67267-squid 2024-08-02T23:40:56.176 DEBUG:teuthology.orchestra.run.smithi073:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-08-02T23:40:56.244 INFO:tasks.workunit.client.0.smithi073.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-08-02T23:40:56.246 INFO:tasks.workunit.client.0.smithi073.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-08-02T23:40:56.246 INFO:tasks.workunit.client.0.smithi073.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-08-02T23:40:56.908 INFO:tasks.workunit.client.0.smithi073.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-08-02T23:40:56.965 INFO:tasks.workunit.client.0.smithi073.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-08-02T23:40:57.004 INFO:tasks.workunit.client.0.smithi073.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-08-02T23:40:57.006 INFO:tasks.workunit.client.0.smithi073.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-08-02T23:40:57.006 INFO:tasks.workunit.client.0.smithi073.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-08-02T23:40:57.044 INFO:tasks.workunit.client.0.smithi073.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-08-02T23:40:57.047 DEBUG:teuthology.orchestra.run.smithi073:> set -ex 2024-08-02T23:40:57.047 DEBUG:teuthology.orchestra.run.smithi073:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-08-02T23:40:57.103 INFO:tasks.workunit:Running workunits matching rados/test.sh on client.0... 2024-08-02T23:40:57.104 INFO:tasks.workunit:Running workunit rados/test.sh... 2024-08-02T23:40:57.105 DEBUG:teuthology.orchestra.run.smithi073:workunit test rados/test.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=26c3fb8e197dcf7a49a54d1f4c8a7362ee35a8ea TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 CRIMSON_COMPAT=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh --crimson 2024-08-02T23:40:57.164 INFO:tasks.workunit.client.0.smithi073.stderr:+ parallel=1 2024-08-02T23:40:57.164 INFO:tasks.workunit.client.0.smithi073.stderr:+ '[' --crimson = --serial ']' 2024-08-02T23:40:57.164 INFO:tasks.workunit.client.0.smithi073.stderr:+ crimson=0 2024-08-02T23:40:57.164 INFO:tasks.workunit.client.0.smithi073.stderr:+ '[' --crimson = --crimson ']' 2024-08-02T23:40:57.164 INFO:tasks.workunit.client.0.smithi073.stderr:+ parallel=0 2024-08-02T23:40:57.164 INFO:tasks.workunit.client.0.smithi073.stderr:+ crimson=1 2024-08-02T23:40:57.164 INFO:tasks.workunit.client.0.smithi073.stderr:+ color= 2024-08-02T23:40:57.164 INFO:tasks.workunit.client.0.smithi073.stderr:+ '[' -t 1 ']' 2024-08-02T23:40:57.165 INFO:tasks.workunit.client.0.smithi073.stderr:+ trap cleanup EXIT ERR HUP INT QUIT 2024-08-02T23:40:57.165 INFO:tasks.workunit.client.0.smithi073.stderr:+ GTEST_OUTPUT_DIR=/home/ubuntu/cephtest/archive/unit_test_xml_report 2024-08-02T23:40:57.165 INFO:tasks.workunit.client.0.smithi073.stderr:+ mkdir -p /home/ubuntu/cephtest/archive/unit_test_xml_report 2024-08-02T23:40:57.166 INFO:tasks.workunit.client.0.smithi073.stderr:+ declare -A pids 2024-08-02T23:40:57.166 INFO:tasks.workunit.client.0.smithi073.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-08-02T23:40:57.166 INFO:tasks.workunit.client.0.smithi073.stderr:+ '[' 0 -eq 1 ']' 2024-08-02T23:40:57.166 INFO:tasks.workunit.client.0.smithi073.stderr:+ ceph_test_rados_api_aio 2024-08-02T23:40:57.180 INFO:tasks.workunit.client.0.smithi073.stdout:Running main() from gmock_main.cc 2024-08-02T23:40:57.180 INFO:tasks.workunit.client.0.smithi073.stdout:[==========] Running 42 tests from 2 test suites. 2024-08-02T23:40:57.180 INFO:tasks.workunit.client.0.smithi073.stdout:[----------] Global test environment set-up. 2024-08-02T23:40:57.180 INFO:tasks.workunit.client.0.smithi073.stdout:[----------] 26 tests from LibRadosAio 2024-08-02T23:40:57.180 INFO:tasks.workunit.client.0.smithi073.stdout:[ RUN ] LibRadosAio.TooBig 2024-08-02T23:40:59.411 INFO:tasks.workunit.client.0.smithi073.stdout:[ OK ] LibRadosAio.TooBig (2231 ms) 2024-08-02T23:40:59.411 INFO:tasks.workunit.client.0.smithi073.stdout:[ RUN ] LibRadosAio.SimpleWrite 2024-08-02T23:41:02.431 INFO:tasks.workunit.client.0.smithi073.stdout:[ OK ] LibRadosAio.SimpleWrite (3020 ms) 2024-08-02T23:41:02.431 INFO:tasks.workunit.client.0.smithi073.stdout:[ RUN ] LibRadosAio.WaitForSafe 2024-08-02T23:41:05.446 INFO:tasks.workunit.client.0.smithi073.stdout:[ OK ] LibRadosAio.WaitForSafe (3015 ms) 2024-08-02T23:41:05.446 INFO:tasks.workunit.client.0.smithi073.stdout:[ RUN ] LibRadosAio.RoundTrip 2024-08-02T23:41:08.456 INFO:tasks.workunit.client.0.smithi073.stdout:[ OK ] LibRadosAio.RoundTrip (3010 ms) 2024-08-02T23:41:08.456 INFO:tasks.workunit.client.0.smithi073.stdout:[ RUN ] LibRadosAio.RoundTrip2 2024-08-02T23:41:11.467 INFO:tasks.workunit.client.0.smithi073.stdout:[ OK ] LibRadosAio.RoundTrip2 (3011 ms) 2024-08-02T23:41:11.467 INFO:tasks.workunit.client.0.smithi073.stdout:[ RUN ] LibRadosAio.RoundTrip3 2024-08-02T23:41:14.476 INFO:tasks.workunit.client.0.smithi073.stdout:[ OK ] LibRadosAio.RoundTrip3 (3010 ms) 2024-08-02T23:41:14.476 INFO:tasks.workunit.client.0.smithi073.stdout:[ RUN ] LibRadosAio.RoundTripAppend 2024-08-02T23:41:17.494 INFO:tasks.workunit.client.0.smithi073.stdout:[ OK ] LibRadosAio.RoundTripAppend (3017 ms) 2024-08-02T23:41:17.494 INFO:tasks.workunit.client.0.smithi073.stdout:[ RUN ] LibRadosAio.RemoveTest 2024-08-02T23:41:20.506 INFO:tasks.workunit.client.0.smithi073.stdout:[ OK ] LibRadosAio.RemoveTest (3012 ms) 2024-08-02T23:41:20.506 INFO:tasks.workunit.client.0.smithi073.stdout:[ RUN ] LibRadosAio.XattrsRoundTrip 2024-08-02T23:41:23.533 INFO:tasks.workunit.client.0.smithi073.stdout:[ OK ] LibRadosAio.XattrsRoundTrip (3025 ms) 2024-08-02T23:41:23.533 INFO:tasks.workunit.client.0.smithi073.stdout:[ RUN ] LibRadosAio.RmXattr 2024-08-02T23:41:26.538 INFO:tasks.workunit.client.0.smithi073.stdout:[ OK ] LibRadosAio.RmXattr (3007 ms) 2024-08-02T23:41:26.538 INFO:tasks.workunit.client.0.smithi073.stdout:[ RUN ] LibRadosAio.XattrIter 2024-08-02T23:41:29.549 INFO:tasks.workunit.client.0.smithi073.stdout:[ OK ] LibRadosAio.XattrIter (3011 ms) 2024-08-02T23:41:29.549 INFO:tasks.workunit.client.0.smithi073.stdout:[ RUN ] LibRadosAio.IsComplete 2024-08-02T23:41:32.567 INFO:tasks.workunit.client.0.smithi073.stdout:[ OK ] LibRadosAio.IsComplete (3018 ms) 2024-08-02T23:41:32.568 INFO:tasks.workunit.client.0.smithi073.stdout:[ RUN ] LibRadosAio.IsSafe 2024-08-02T23:41:35.576 INFO:tasks.workunit.client.0.smithi073.stdout:[ OK ] LibRadosAio.IsSafe (3009 ms) 2024-08-02T23:41:35.576 INFO:tasks.workunit.client.0.smithi073.stdout:[ RUN ] LibRadosAio.ReturnValue 2024-08-02T23:41:38.594 INFO:tasks.workunit.client.0.smithi073.stdout:[ OK ] LibRadosAio.ReturnValue (3018 ms) 2024-08-02T23:41:38.594 INFO:tasks.workunit.client.0.smithi073.stdout:[ RUN ] LibRadosAio.Flush 2024-08-02T23:41:41.610 INFO:tasks.workunit.client.0.smithi073.stdout:[ OK ] LibRadosAio.Flush (3016 ms) 2024-08-02T23:41:41.610 INFO:tasks.workunit.client.0.smithi073.stdout:[ RUN ] LibRadosAio.FlushAsync 2024-08-02T23:41:44.630 INFO:tasks.workunit.client.0.smithi073.stdout:[ OK ] LibRadosAio.FlushAsync (3018 ms) 2024-08-02T23:41:44.631 INFO:tasks.workunit.client.0.smithi073.stdout:[ RUN ] LibRadosAio.RoundTripWriteFull 2024-08-02T23:41:47.394 INFO:tasks.workunit.client.0.smithi073.stdout:[ OK ] LibRadosAio.RoundTripWriteFull (2766 ms) 2024-08-02T23:41:47.395 INFO:tasks.workunit.client.0.smithi073.stdout:[ RUN ] LibRadosAio.RoundTripWriteSame 2024-08-02T23:41:50.449 INFO:tasks.workunit.client.0.smithi073.stdout:[ OK ] LibRadosAio.RoundTripWriteSame (3055 ms) 2024-08-02T23:41:50.449 INFO:tasks.workunit.client.0.smithi073.stdout:[ RUN ] LibRadosAio.SimpleStat 2024-08-02T23:41:53.459 INFO:tasks.workunit.client.0.smithi073.stdout:[ OK ] LibRadosAio.SimpleStat (3010 ms) 2024-08-02T23:41:53.459 INFO:tasks.workunit.client.0.smithi073.stdout:[ RUN ] LibRadosAio.OperateMtime 2024-08-02T23:41:56.480 INFO:tasks.workunit.client.0.smithi073.stdout:[ OK ] LibRadosAio.OperateMtime (3021 ms) 2024-08-02T23:41:56.480 INFO:tasks.workunit.client.0.smithi073.stdout:[ RUN ] LibRadosAio.Operate2Mtime 2024-08-02T23:41:59.497 INFO:tasks.workunit.client.0.smithi073.stdout:[ OK ] LibRadosAio.Operate2Mtime (3017 ms) 2024-08-02T23:41:59.497 INFO:tasks.workunit.client.0.smithi073.stdout:[ RUN ] LibRadosAio.SimpleStatNS 2024-08-02T23:42:02.398 INFO:tasks.workunit.client.0.smithi073.stdout:[ OK ] LibRadosAio.SimpleStatNS (2901 ms) 2024-08-02T23:42:02.398 INFO:tasks.workunit.client.0.smithi073.stdout:[ RUN ] LibRadosAio.StatRemove 2024-08-02T23:42:05.413 INFO:tasks.workunit.client.0.smithi073.stdout:[ OK ] LibRadosAio.StatRemove (3015 ms) 2024-08-02T23:42:05.413 INFO:tasks.workunit.client.0.smithi073.stdout:[ RUN ] LibRadosAio.ExecuteClass 2024-08-02T23:42:08.418 INFO:tasks.workunit.client.0.smithi073.stdout:[ OK ] LibRadosAio.ExecuteClass (3005 ms) 2024-08-02T23:42:08.418 INFO:tasks.workunit.client.0.smithi073.stdout:[ RUN ] LibRadosAio.MultiWrite 2024-08-02T23:42:11.435 INFO:tasks.workunit.client.0.smithi073.stdout:[ OK ] LibRadosAio.MultiWrite (3017 ms) 2024-08-02T23:42:11.435 INFO:tasks.workunit.client.0.smithi073.stdout:[ RUN ] LibRadosAio.AioUnlock 2024-08-02T23:42:14.413 INFO:tasks.workunit.client.0.smithi073.stdout:[ OK ] LibRadosAio.AioUnlock (2977 ms) 2024-08-02T23:42:14.413 INFO:tasks.workunit.client.0.smithi073.stdout:[----------] 26 tests from LibRadosAio (77232 ms total) 2024-08-02T23:42:14.413 INFO:tasks.workunit.client.0.smithi073.stdout: 2024-08-02T23:42:14.413 INFO:tasks.workunit.client.0.smithi073.stdout:[----------] 16 tests from LibRadosAioEC 2024-08-02T23:42:14.413 INFO:tasks.workunit.client.0.smithi073.stdout:[ RUN ] LibRadosAioEC.SimpleWrite 2024-08-02T23:42:14.413 INFO:tasks.workunit.client.0.smithi073.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.1.0-1260-g26c3fb8e/rpm/el9/BUILD/ceph-19.1.0-1260-g26c3fb8e/src/test/librados/aio.cc:1100: Skipped 2024-08-02T23:42:14.413 INFO:tasks.workunit.client.0.smithi073.stdout:Not supported by crimson yet. Skipped 2024-08-02T23:42:14.413 INFO:tasks.workunit.client.0.smithi073.stdout:[ SKIPPED ] LibRadosAioEC.SimpleWrite (0 ms) 2024-08-02T23:42:14.414 INFO:tasks.workunit.client.0.smithi073.stdout:[ RUN ] LibRadosAioEC.WaitForComplete 2024-08-02T23:42:14.414 INFO:tasks.workunit.client.0.smithi073.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.1.0-1260-g26c3fb8e/rpm/el9/BUILD/ceph-19.1.0-1260-g26c3fb8e/src/test/librados/aio.cc:1132: Skipped 2024-08-02T23:42:14.414 INFO:tasks.workunit.client.0.smithi073.stdout:Not supported by crimson yet. Skipped 2024-08-02T23:42:14.414 INFO:tasks.workunit.client.0.smithi073.stdout:[ SKIPPED ] LibRadosAioEC.WaitForComplete (0 ms) 2024-08-02T23:42:14.414 INFO:tasks.workunit.client.0.smithi073.stdout:[ RUN ] LibRadosAioEC.RoundTrip 2024-08-02T23:42:14.414 INFO:tasks.workunit.client.0.smithi073.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.1.0-1260-g26c3fb8e/rpm/el9/BUILD/ceph-19.1.0-1260-g26c3fb8e/src/test/librados/aio.cc:1149: Skipped 2024-08-02T23:42:14.414 INFO:tasks.workunit.client.0.smithi073.stdout:Not supported by crimson yet. Skipped 2024-08-02T23:42:14.414 INFO:tasks.workunit.client.0.smithi073.stdout:[ SKIPPED ] LibRadosAioEC.RoundTrip (0 ms) 2024-08-02T23:42:14.414 INFO:tasks.workunit.client.0.smithi073.stdout:[ RUN ] LibRadosAioEC.RoundTrip2 2024-08-02T23:42:14.414 INFO:tasks.workunit.client.0.smithi073.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.1.0-1260-g26c3fb8e/rpm/el9/BUILD/ceph-19.1.0-1260-g26c3fb8e/src/test/librados/aio.cc:1182: Skipped 2024-08-02T23:42:14.414 INFO:tasks.workunit.client.0.smithi073.stdout:Not supported by crimson yet. Skipped 2024-08-02T23:42:14.415 INFO:tasks.workunit.client.0.smithi073.stdout:[ SKIPPED ] LibRadosAioEC.RoundTrip2 (0 ms) 2024-08-02T23:42:14.415 INFO:tasks.workunit.client.0.smithi073.stdout:[ RUN ] LibRadosAioEC.RoundTripAppend 2024-08-02T23:42:14.415 INFO:tasks.workunit.client.0.smithi073.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.1.0-1260-g26c3fb8e/rpm/el9/BUILD/ceph-19.1.0-1260-g26c3fb8e/src/test/librados/aio.cc:1215: Skipped 2024-08-02T23:42:14.415 INFO:tasks.workunit.client.0.smithi073.stdout:Not supported by crimson yet. Skipped 2024-08-02T23:42:14.415 INFO:tasks.workunit.client.0.smithi073.stdout:[ SKIPPED ] LibRadosAioEC.RoundTripAppend (0 ms) 2024-08-02T23:42:14.415 INFO:tasks.workunit.client.0.smithi073.stdout:[ RUN ] LibRadosAioEC.IsComplete 2024-08-02T23:42:14.415 INFO:tasks.workunit.client.0.smithi073.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.1.0-1260-g26c3fb8e/rpm/el9/BUILD/ceph-19.1.0-1260-g26c3fb8e/src/test/librados/aio.cc:1286: Skipped 2024-08-02T23:42:14.415 INFO:tasks.workunit.client.0.smithi073.stdout:Not supported by crimson yet. Skipped 2024-08-02T23:42:14.415 INFO:tasks.workunit.client.0.smithi073.stdout:[ SKIPPED ] LibRadosAioEC.IsComplete (0 ms) 2024-08-02T23:42:14.415 INFO:tasks.workunit.client.0.smithi073.stdout:[ RUN ] LibRadosAioEC.IsSafe 2024-08-02T23:42:14.415 INFO:tasks.workunit.client.0.smithi073.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.1.0-1260-g26c3fb8e/rpm/el9/BUILD/ceph-19.1.0-1260-g26c3fb8e/src/test/librados/aio.cc:1326: Skipped 2024-08-02T23:42:14.415 INFO:tasks.workunit.client.0.smithi073.stdout:Not supported by crimson yet. Skipped 2024-08-02T23:42:14.415 INFO:tasks.workunit.client.0.smithi073.stdout:[ SKIPPED ] LibRadosAioEC.IsSafe (0 ms) 2024-08-02T23:42:14.416 INFO:tasks.workunit.client.0.smithi073.stdout:[ RUN ] LibRadosAioEC.ReturnValue 2024-08-02T23:42:14.416 INFO:tasks.workunit.client.0.smithi073.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.1.0-1260-g26c3fb8e/rpm/el9/BUILD/ceph-19.1.0-1260-g26c3fb8e/src/test/librados/aio.cc:1366: Skipped 2024-08-02T23:42:14.416 INFO:tasks.workunit.client.0.smithi073.stdout:Not supported by crimson yet. Skipped 2024-08-02T23:42:14.416 INFO:tasks.workunit.client.0.smithi073.stdout:[ SKIPPED ] LibRadosAioEC.ReturnValue (0 ms) 2024-08-02T23:42:14.416 INFO:tasks.workunit.client.0.smithi073.stdout:[ RUN ] LibRadosAioEC.Flush 2024-08-02T23:42:14.416 INFO:tasks.workunit.client.0.smithi073.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.1.0-1260-g26c3fb8e/rpm/el9/BUILD/ceph-19.1.0-1260-g26c3fb8e/src/test/librados/aio.cc:1385: Skipped 2024-08-02T23:42:14.416 INFO:tasks.workunit.client.0.smithi073.stdout:Not supported by crimson yet. Skipped 2024-08-02T23:42:14.416 INFO:tasks.workunit.client.0.smithi073.stdout:[ SKIPPED ] LibRadosAioEC.Flush (0 ms) 2024-08-02T23:42:14.416 INFO:tasks.workunit.client.0.smithi073.stdout:[ RUN ] LibRadosAioEC.FlushAsync 2024-08-02T23:42:14.416 INFO:tasks.workunit.client.0.smithi073.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.1.0-1260-g26c3fb8e/rpm/el9/BUILD/ceph-19.1.0-1260-g26c3fb8e/src/test/librados/aio.cc:1415: Skipped 2024-08-02T23:42:14.416 INFO:tasks.workunit.client.0.smithi073.stdout:Not supported by crimson yet. Skipped 2024-08-02T23:42:14.416 INFO:tasks.workunit.client.0.smithi073.stdout:[ SKIPPED ] LibRadosAioEC.FlushAsync (0 ms) 2024-08-02T23:42:14.417 INFO:tasks.workunit.client.0.smithi073.stdout:[ RUN ] LibRadosAioEC.RoundTripWriteFull 2024-08-02T23:42:14.417 INFO:tasks.workunit.client.0.smithi073.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.1.0-1260-g26c3fb8e/rpm/el9/BUILD/ceph-19.1.0-1260-g26c3fb8e/src/test/librados/aio.cc:1454: Skipped 2024-08-02T23:42:14.417 INFO:tasks.workunit.client.0.smithi073.stdout:Not supported by crimson yet. Skipped 2024-08-02T23:42:14.417 INFO:tasks.workunit.client.0.smithi073.stdout:[ SKIPPED ] LibRadosAioEC.RoundTripWriteFull (0 ms) 2024-08-02T23:42:14.417 INFO:tasks.workunit.client.0.smithi073.stdout:[ RUN ] LibRadosAioEC.SimpleStat 2024-08-02T23:42:14.417 INFO:tasks.workunit.client.0.smithi073.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.1.0-1260-g26c3fb8e/rpm/el9/BUILD/ceph-19.1.0-1260-g26c3fb8e/src/test/librados/aio.cc:1498: Skipped 2024-08-02T23:42:14.417 INFO:tasks.workunit.client.0.smithi073.stdout:Not supported by crimson yet. Skipped 2024-08-02T23:42:14.417 INFO:tasks.workunit.client.0.smithi073.stdout:[ SKIPPED ] LibRadosAioEC.SimpleStat (0 ms) 2024-08-02T23:42:14.417 INFO:tasks.workunit.client.0.smithi073.stdout:[ RUN ] LibRadosAioEC.SimpleStatNS 2024-08-02T23:42:14.417 INFO:tasks.workunit.client.0.smithi073.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.1.0-1260-g26c3fb8e/rpm/el9/BUILD/ceph-19.1.0-1260-g26c3fb8e/src/test/librados/aio.cc:1532: Skipped 2024-08-02T23:42:14.417 INFO:tasks.workunit.client.0.smithi073.stdout:Not supported by crimson yet. Skipped 2024-08-02T23:42:14.418 INFO:tasks.workunit.client.0.smithi073.stdout:[ SKIPPED ] LibRadosAioEC.SimpleStatNS (0 ms) 2024-08-02T23:42:14.418 INFO:tasks.workunit.client.0.smithi073.stdout:[ RUN ] LibRadosAioEC.StatRemove 2024-08-02T23:42:14.418 INFO:tasks.workunit.client.0.smithi073.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.1.0-1260-g26c3fb8e/rpm/el9/BUILD/ceph-19.1.0-1260-g26c3fb8e/src/test/librados/aio.cc:1593: Skipped 2024-08-02T23:42:14.418 INFO:tasks.workunit.client.0.smithi073.stdout:Not supported by crimson yet. Skipped 2024-08-02T23:42:14.418 INFO:tasks.workunit.client.0.smithi073.stdout:[ SKIPPED ] LibRadosAioEC.StatRemove (0 ms) 2024-08-02T23:42:14.418 INFO:tasks.workunit.client.0.smithi073.stdout:[ RUN ] LibRadosAioEC.ExecuteClass 2024-08-02T23:42:14.418 INFO:tasks.workunit.client.0.smithi073.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.1.0-1260-g26c3fb8e/rpm/el9/BUILD/ceph-19.1.0-1260-g26c3fb8e/src/test/librados/aio.cc:1649: Skipped 2024-08-02T23:42:14.418 INFO:tasks.workunit.client.0.smithi073.stdout:Not supported by crimson yet. Skipped 2024-08-02T23:42:14.418 INFO:tasks.workunit.client.0.smithi073.stdout:[ SKIPPED ] LibRadosAioEC.ExecuteClass (0 ms) 2024-08-02T23:42:14.418 INFO:tasks.workunit.client.0.smithi073.stdout:[ RUN ] LibRadosAioEC.MultiWrite 2024-08-02T23:42:14.418 INFO:tasks.workunit.client.0.smithi073.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.1.0-1260-g26c3fb8e/rpm/el9/BUILD/ceph-19.1.0-1260-g26c3fb8e/src/test/librados/aio.cc:1681: Skipped 2024-08-02T23:42:14.418 INFO:tasks.workunit.client.0.smithi073.stdout:Not supported by crimson yet. Skipped 2024-08-02T23:42:14.418 INFO:tasks.workunit.client.0.smithi073.stdout:[ SKIPPED ] LibRadosAioEC.MultiWrite (0 ms) 2024-08-02T23:42:14.419 INFO:tasks.workunit.client.0.smithi073.stdout:[----------] 16 tests from LibRadosAioEC (0 ms total) 2024-08-02T23:42:14.419 INFO:tasks.workunit.client.0.smithi073.stdout: 2024-08-02T23:42:14.419 INFO:tasks.workunit.client.0.smithi073.stdout:[----------] Global test environment tear-down 2024-08-02T23:42:14.419 INFO:tasks.workunit.client.0.smithi073.stdout:[==========] 42 tests from 2 test suites ran. (77232 ms total) 2024-08-02T23:42:14.419 INFO:tasks.workunit.client.0.smithi073.stdout:[ PASSED ] 26 tests. 2024-08-02T23:42:14.419 INFO:tasks.workunit.client.0.smithi073.stdout:[ SKIPPED ] 16 tests, listed below: 2024-08-02T23:42:14.419 INFO:tasks.workunit.client.0.smithi073.stdout:[ SKIPPED ] LibRadosAioEC.SimpleWrite 2024-08-02T23:42:14.419 INFO:tasks.workunit.client.0.smithi073.stdout:[ SKIPPED ] LibRadosAioEC.WaitForComplete 2024-08-02T23:42:14.419 INFO:tasks.workunit.client.0.smithi073.stdout:[ SKIPPED ] LibRadosAioEC.RoundTrip 2024-08-02T23:42:14.419 INFO:tasks.workunit.client.0.smithi073.stdout:[ SKIPPED ] LibRadosAioEC.RoundTrip2 2024-08-02T23:42:14.419 INFO:tasks.workunit.client.0.smithi073.stdout:[ SKIPPED ] LibRadosAioEC.RoundTripAppend 2024-08-02T23:42:14.419 INFO:tasks.workunit.client.0.smithi073.stdout:[ SKIPPED ] LibRadosAioEC.IsComplete 2024-08-02T23:42:14.419 INFO:tasks.workunit.client.0.smithi073.stdout:[ SKIPPED ] LibRadosAioEC.IsSafe 2024-08-02T23:42:14.419 INFO:tasks.workunit.client.0.smithi073.stdout:[ SKIPPED ] LibRadosAioEC.ReturnValue 2024-08-02T23:42:14.420 INFO:tasks.workunit.client.0.smithi073.stdout:[ SKIPPED ] LibRadosAioEC.Flush 2024-08-02T23:42:14.420 INFO:tasks.workunit.client.0.smithi073.stdout:[ SKIPPED ] LibRadosAioEC.FlushAsync 2024-08-02T23:42:14.420 INFO:tasks.workunit.client.0.smithi073.stdout:[ SKIPPED ] LibRadosAioEC.RoundTripWriteFull 2024-08-02T23:42:14.420 INFO:tasks.workunit.client.0.smithi073.stdout:[ SKIPPED ] LibRadosAioEC.SimpleStat 2024-08-02T23:42:14.420 INFO:tasks.workunit.client.0.smithi073.stdout:[ SKIPPED ] LibRadosAioEC.SimpleStatNS 2024-08-02T23:42:14.420 INFO:tasks.workunit.client.0.smithi073.stdout:[ SKIPPED ] LibRadosAioEC.StatRemove 2024-08-02T23:42:14.420 INFO:tasks.workunit.client.0.smithi073.stdout:[ SKIPPED ] LibRadosAioEC.ExecuteClass 2024-08-02T23:42:14.420 INFO:tasks.workunit.client.0.smithi073.stdout:[ SKIPPED ] LibRadosAioEC.MultiWrite 2024-08-02T23:42:14.421 INFO:tasks.workunit.client.0.smithi073.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-08-02T23:42:14.421 INFO:tasks.workunit.client.0.smithi073.stderr:+ '[' 0 -eq 1 ']' 2024-08-02T23:42:14.421 INFO:tasks.workunit.client.0.smithi073.stderr:+ ceph_test_rados_api_aio_pp 2024-08-02T23:42:14.433 INFO:tasks.workunit.client.0.smithi073.stdout:Running main() from gmock_main.cc 2024-08-02T23:42:14.433 INFO:tasks.workunit.client.0.smithi073.stdout:[==========] Running 57 tests from 4 test suites. 2024-08-02T23:42:14.433 INFO:tasks.workunit.client.0.smithi073.stdout:[----------] Global test environment set-up. 2024-08-02T23:42:14.433 INFO:tasks.workunit.client.0.smithi073.stdout:[----------] 32 tests from LibRadosAio 2024-08-02T23:42:14.433 INFO:tasks.workunit.client.0.smithi073.stdout:[ RUN ] LibRadosAio.TooBigPP 2024-08-02T23:42:17.415 INFO:tasks.workunit.client.0.smithi073.stdout:[ OK ] LibRadosAio.TooBigPP (2982 ms) 2024-08-02T23:42:17.416 INFO:tasks.workunit.client.0.smithi073.stdout:[ RUN ] LibRadosAio.PoolQuotaPP 2024-08-02T23:42:39.411 INFO:tasks.workunit.client.0.smithi073.stdout:[ OK ] LibRadosAio.PoolQuotaPP (21996 ms) 2024-08-02T23:42:39.412 INFO:tasks.workunit.client.0.smithi073.stdout:[ RUN ] LibRadosAio.SimpleWritePP 2024-08-02T23:42:45.444 INFO:tasks.workunit.client.0.smithi073.stdout:[ OK ] LibRadosAio.SimpleWritePP (6033 ms) 2024-08-02T23:42:45.444 INFO:tasks.workunit.client.0.smithi073.stdout:[ RUN ] LibRadosAio.WaitForSafePP 2024-08-02T23:42:48.450 INFO:tasks.workunit.client.0.smithi073.stdout:[ OK ] LibRadosAio.WaitForSafePP (3006 ms) 2024-08-02T23:42:48.450 INFO:tasks.workunit.client.0.smithi073.stdout:[ RUN ] LibRadosAio.RoundTripPP 2024-08-02T23:42:51.469 INFO:tasks.workunit.client.0.smithi073.stdout:[ OK ] LibRadosAio.RoundTripPP (3019 ms) 2024-08-02T23:42:51.469 INFO:tasks.workunit.client.0.smithi073.stdout:[ RUN ] LibRadosAio.RoundTripPP2 2024-08-02T23:42:54.480 INFO:tasks.workunit.client.0.smithi073.stdout:[ OK ] LibRadosAio.RoundTripPP2 (3011 ms) 2024-08-02T23:42:54.480 INFO:tasks.workunit.client.0.smithi073.stdout:[ RUN ] LibRadosAio.RoundTripPP3 2024-08-02T23:42:57.411 INFO:tasks.workunit.client.0.smithi073.stdout:[ OK ] LibRadosAio.RoundTripPP3 (2931 ms) 2024-08-02T23:42:57.411 INFO:tasks.workunit.client.0.smithi073.stdout:[ RUN ] LibRadosAio.RoundTripSparseReadPP 2024-08-02T23:43:00.429 INFO:tasks.workunit.client.0.smithi073.stdout:[ OK ] LibRadosAio.RoundTripSparseReadPP (3018 ms) 2024-08-02T23:43:00.429 INFO:tasks.workunit.client.0.smithi073.stdout:[ RUN ] LibRadosAio.IsCompletePP 2024-08-02T23:43:03.445 INFO:tasks.workunit.client.0.smithi073.stdout:[ OK ] LibRadosAio.IsCompletePP (3016 ms) 2024-08-02T23:43:03.445 INFO:tasks.workunit.client.0.smithi073.stdout:[ RUN ] LibRadosAio.IsSafePP 2024-08-02T23:43:06.463 INFO:tasks.workunit.client.0.smithi073.stdout:[ OK ] LibRadosAio.IsSafePP (3018 ms) 2024-08-02T23:43:06.463 INFO:tasks.workunit.client.0.smithi073.stdout:[ RUN ] LibRadosAio.ReturnValuePP 2024-08-02T23:43:09.419 INFO:tasks.workunit.client.0.smithi073.stdout:[ OK ] LibRadosAio.ReturnValuePP (2956 ms) 2024-08-02T23:43:09.419 INFO:tasks.workunit.client.0.smithi073.stdout:[ RUN ] LibRadosAio.FlushPP 2024-08-02T23:43:12.430 INFO:tasks.workunit.client.0.smithi073.stdout:[ OK ] LibRadosAio.FlushPP (3011 ms) 2024-08-02T23:43:12.430 INFO:tasks.workunit.client.0.smithi073.stdout:[ RUN ] LibRadosAio.FlushAsyncPP 2024-08-02T23:43:15.446 INFO:tasks.workunit.client.0.smithi073.stdout:[ OK ] LibRadosAio.FlushAsyncPP (3016 ms) 2024-08-02T23:43:15.446 INFO:tasks.workunit.client.0.smithi073.stdout:[ RUN ] LibRadosAio.RoundTripWriteFullPP 2024-08-02T23:43:18.456 INFO:tasks.workunit.client.0.smithi073.stdout:[ OK ] LibRadosAio.RoundTripWriteFullPP (3009 ms) 2024-08-02T23:43:18.456 INFO:tasks.workunit.client.0.smithi073.stdout:[ RUN ] LibRadosAio.RoundTripWriteFullPP2 2024-08-02T23:43:21.470 INFO:tasks.workunit.client.0.smithi073.stdout:[ OK ] LibRadosAio.RoundTripWriteFullPP2 (3015 ms) 2024-08-02T23:43:21.470 INFO:tasks.workunit.client.0.smithi073.stdout:[ RUN ] LibRadosAio.RoundTripWriteSamePP 2024-08-02T23:43:24.486 INFO:tasks.workunit.client.0.smithi073.stdout:[ OK ] LibRadosAio.RoundTripWriteSamePP (3016 ms) 2024-08-02T23:43:24.487 INFO:tasks.workunit.client.0.smithi073.stdout:[ RUN ] LibRadosAio.RoundTripWriteSamePP2 2024-08-02T23:43:27.414 INFO:tasks.workunit.client.0.smithi073.stdout:[ OK ] LibRadosAio.RoundTripWriteSamePP2 (2927 ms) 2024-08-02T23:43:27.415 INFO:tasks.workunit.client.0.smithi073.stdout:[ RUN ] LibRadosAio.SimpleStatPPNS 2024-08-02T23:43:30.423 INFO:tasks.workunit.client.0.smithi073.stdout:[ OK ] LibRadosAio.SimpleStatPPNS (3009 ms) 2024-08-02T23:43:30.424 INFO:tasks.workunit.client.0.smithi073.stdout:[ RUN ] LibRadosAio.SimpleStatPP 2024-08-02T23:43:33.434 INFO:tasks.workunit.client.0.smithi073.stdout:[ OK ] LibRadosAio.SimpleStatPP (3011 ms) 2024-08-02T23:43:33.434 INFO:tasks.workunit.client.0.smithi073.stdout:[ RUN ] LibRadosAio.OperateMtime 2024-08-02T23:43:36.445 INFO:tasks.workunit.client.0.smithi073.stdout:[ OK ] LibRadosAio.OperateMtime (3011 ms) 2024-08-02T23:43:36.445 INFO:tasks.workunit.client.0.smithi073.stdout:[ RUN ] LibRadosAio.OperateMtime2 2024-08-02T23:43:39.436 INFO:tasks.workunit.client.0.smithi073.stdout:[ OK ] LibRadosAio.OperateMtime2 (2991 ms) 2024-08-02T23:43:39.436 INFO:tasks.workunit.client.0.smithi073.stdout:[ RUN ] LibRadosAio.StatRemovePP 2024-08-02T23:43:42.445 INFO:tasks.workunit.client.0.smithi073.stdout:[ OK ] LibRadosAio.StatRemovePP (3009 ms) 2024-08-02T23:43:42.445 INFO:tasks.workunit.client.0.smithi073.stdout:[ RUN ] LibRadosAio.ExecuteClassPP 2024-08-02T23:43:45.459 INFO:tasks.workunit.client.0.smithi073.stdout:[ OK ] LibRadosAio.ExecuteClassPP (3014 ms) 2024-08-02T23:43:45.459 INFO:tasks.workunit.client.0.smithi073.stdout:[ RUN ] LibRadosAio.OmapPP 2024-08-02T23:43:48.464 INFO:tasks.workunit.client.0.smithi073.stdout:[ OK ] LibRadosAio.OmapPP (3005 ms) 2024-08-02T23:43:48.464 INFO:tasks.workunit.client.0.smithi073.stdout:[ RUN ] LibRadosAio.MultiWritePP 2024-08-02T23:43:51.475 INFO:tasks.workunit.client.0.smithi073.stdout:[ OK ] LibRadosAio.MultiWritePP (3011 ms) 2024-08-02T23:43:51.476 INFO:tasks.workunit.client.0.smithi073.stdout:[ RUN ] LibRadosAio.AioUnlockPP 2024-08-02T23:43:54.487 INFO:tasks.workunit.client.0.smithi073.stdout:[ OK ] LibRadosAio.AioUnlockPP (3012 ms) 2024-08-02T23:43:54.487 INFO:tasks.workunit.client.0.smithi073.stdout:[ RUN ] LibRadosAio.RoundTripAppendPP 2024-08-02T23:43:57.420 INFO:tasks.workunit.client.0.smithi073.stdout:[ OK ] LibRadosAio.RoundTripAppendPP (2933 ms) 2024-08-02T23:43:57.420 INFO:tasks.workunit.client.0.smithi073.stdout:[ RUN ] LibRadosAio.RacingRemovePP 2024-08-02T23:44:00.432 INFO:tasks.workunit.client.0.smithi073.stdout:[ OK ] LibRadosAio.RacingRemovePP (3012 ms) 2024-08-02T23:44:00.432 INFO:tasks.workunit.client.0.smithi073.stdout:[ RUN ] LibRadosAio.RoundTripCmpExtPP 2024-08-02T23:44:03.446 INFO:tasks.workunit.client.0.smithi073.stdout:[ OK ] LibRadosAio.RoundTripCmpExtPP (3013 ms) 2024-08-02T23:44:03.446 INFO:tasks.workunit.client.0.smithi073.stdout:[ RUN ] LibRadosAio.RoundTripCmpExtPP2 2024-08-02T23:44:06.456 INFO:tasks.workunit.client.0.smithi073.stdout:[ OK ] LibRadosAio.RoundTripCmpExtPP2 (3011 ms) 2024-08-02T23:44:06.457 INFO:tasks.workunit.client.0.smithi073.stdout:[ RUN ] LibRadosAio.PoolEIOFlag 2024-08-02T23:44:08.921 INFO:tasks.workunit.client.0.smithi073.stdout:setting pool EIO 2024-08-02T23:44:18.570 INFO:tasks.workunit.client.0.smithi073.stdout:waiting for 235 2024-08-02T23:44:19.570 INFO:tasks.workunit.client.0.smithi073.stdout:waiting for 113 2024-08-02T23:44:20.570 INFO:tasks.workunit.client.0.smithi073.stdout:max_success 1000, min_failed 0 2024-08-02T23:44:20.570 INFO:tasks.workunit.client.0.smithi073.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.1.0-1260-g26c3fb8e/rpm/el9/BUILD/ceph-19.1.0-1260-g26c3fb8e/src/test/librados/aio_cxx.cc:2421: Failure 2024-08-02T23:44:20.570 INFO:tasks.workunit.client.0.smithi073.stdout:Value of: max_success + 1 == min_failed 2024-08-02T23:44:20.570 INFO:tasks.workunit.client.0.smithi073.stdout: Actual: false 2024-08-02T23:44:20.570 INFO:tasks.workunit.client.0.smithi073.stdout:Expected: true 2024-08-02T23:44:20.625 INFO:tasks.workunit.client.0.smithi073.stdout:[ FAILED ] LibRadosAio.PoolEIOFlag (14169 ms) 2024-08-02T23:44:20.625 INFO:tasks.workunit.client.0.smithi073.stdout:[ RUN ] LibRadosAio.MultiReads 2024-08-02T23:44:20.625 INFO:tasks.workunit.client.0.smithi073.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.1.0-1260-g26c3fb8e/rpm/el9/BUILD/ceph-19.1.0-1260-g26c3fb8e/src/test/librados/aio_cxx.cc:2427: Skipped 2024-08-02T23:44:20.625 INFO:tasks.workunit.client.0.smithi073.stdout:Not supported by crimson yet. Skipped 2024-08-02T23:44:20.625 INFO:tasks.workunit.client.0.smithi073.stdout:[ SKIPPED ] LibRadosAio.MultiReads (0 ms) 2024-08-02T23:44:20.626 INFO:tasks.workunit.client.0.smithi073.stdout:[----------] 32 tests from LibRadosAio (126192 ms total) 2024-08-02T23:44:20.626 INFO:tasks.workunit.client.0.smithi073.stdout: 2024-08-02T23:44:20.626 INFO:tasks.workunit.client.0.smithi073.stdout:[----------] 4 tests from LibRadosAioPP 2024-08-02T23:44:20.626 INFO:tasks.workunit.client.0.smithi073.stdout:[ RUN ] LibRadosAioPP.ReadIntoBufferlist 2024-08-02T23:44:23.637 INFO:tasks.workunit.client.0.smithi073.stdout:[ OK ] LibRadosAioPP.ReadIntoBufferlist (3012 ms) 2024-08-02T23:44:23.637 INFO:tasks.workunit.client.0.smithi073.stdout:[ RUN ] LibRadosAioPP.XattrsRoundTripPP 2024-08-02T23:44:32.444 INFO:tasks.workunit.client.0.smithi073.stdout:[ OK ] LibRadosAioPP.XattrsRoundTripPP (8807 ms) 2024-08-02T23:44:32.445 INFO:tasks.workunit.client.0.smithi073.stdout:[ RUN ] LibRadosAioPP.RmXattrPP 2024-08-02T23:44:47.429 INFO:tasks.workunit.client.0.smithi073.stdout:[ OK ] LibRadosAioPP.RmXattrPP (14985 ms) 2024-08-02T23:44:47.430 INFO:tasks.workunit.client.0.smithi073.stdout:[ RUN ] LibRadosAioPP.RemoveTestPP 2024-08-02T23:44:50.487 INFO:tasks.workunit.client.0.smithi073.stdout:[ OK ] LibRadosAioPP.RemoveTestPP (3058 ms) 2024-08-02T23:44:50.487 INFO:tasks.workunit.client.0.smithi073.stdout:[----------] 4 tests from LibRadosAioPP (29862 ms total) 2024-08-02T23:44:50.487 INFO:tasks.workunit.client.0.smithi073.stdout: 2024-08-02T23:44:50.488 INFO:tasks.workunit.client.0.smithi073.stdout:[----------] 1 test from LibRadosIoPP 2024-08-02T23:44:50.488 INFO:tasks.workunit.client.0.smithi073.stdout:[ RUN ] LibRadosIoPP.XattrListPP 2024-08-02T23:44:53.500 INFO:tasks.workunit.client.0.smithi073.stdout:[ OK ] LibRadosIoPP.XattrListPP (3012 ms) 2024-08-02T23:44:53.501 INFO:tasks.workunit.client.0.smithi073.stdout:[----------] 1 test from LibRadosIoPP (3012 ms total) 2024-08-02T23:44:53.501 INFO:tasks.workunit.client.0.smithi073.stdout: 2024-08-02T23:44:53.501 INFO:tasks.workunit.client.0.smithi073.stdout:[----------] 20 tests from LibRadosAioEC 2024-08-02T23:44:53.501 INFO:tasks.workunit.client.0.smithi073.stdout:[ RUN ] LibRadosAioEC.SimpleWritePP 2024-08-02T23:44:53.501 INFO:tasks.workunit.client.0.smithi073.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.1.0-1260-g26c3fb8e/rpm/el9/BUILD/ceph-19.1.0-1260-g26c3fb8e/src/test/librados/aio_cxx.cc:1384: Skipped 2024-08-02T23:44:53.501 INFO:tasks.workunit.client.0.smithi073.stdout:Not supported by crimson yet. Skipped 2024-08-02T23:44:53.501 INFO:tasks.workunit.client.0.smithi073.stdout:[ SKIPPED ] LibRadosAioEC.SimpleWritePP (0 ms) 2024-08-02T23:44:53.501 INFO:tasks.workunit.client.0.smithi073.stdout:[ RUN ] LibRadosAioEC.WaitForSafePP 2024-08-02T23:44:53.501 INFO:tasks.workunit.client.0.smithi073.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.1.0-1260-g26c3fb8e/rpm/el9/BUILD/ceph-19.1.0-1260-g26c3fb8e/src/test/librados/aio_cxx.cc:1419: Skipped 2024-08-02T23:44:53.501 INFO:tasks.workunit.client.0.smithi073.stdout:Not supported by crimson yet. Skipped 2024-08-02T23:44:53.501 INFO:tasks.workunit.client.0.smithi073.stdout:[ SKIPPED ] LibRadosAioEC.WaitForSafePP (0 ms) 2024-08-02T23:44:53.501 INFO:tasks.workunit.client.0.smithi073.stdout:[ RUN ] LibRadosAioEC.RoundTripPP 2024-08-02T23:44:53.501 INFO:tasks.workunit.client.0.smithi073.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.1.0-1260-g26c3fb8e/rpm/el9/BUILD/ceph-19.1.0-1260-g26c3fb8e/src/test/librados/aio_cxx.cc:1436: Skipped 2024-08-02T23:44:53.502 INFO:tasks.workunit.client.0.smithi073.stdout:Not supported by crimson yet. Skipped 2024-08-02T23:44:53.502 INFO:tasks.workunit.client.0.smithi073.stdout:[ SKIPPED ] LibRadosAioEC.RoundTripPP (0 ms) 2024-08-02T23:44:53.502 INFO:tasks.workunit.client.0.smithi073.stdout:[ RUN ] LibRadosAioEC.RoundTripPP2 2024-08-02T23:44:53.502 INFO:tasks.workunit.client.0.smithi073.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.1.0-1260-g26c3fb8e/rpm/el9/BUILD/ceph-19.1.0-1260-g26c3fb8e/src/test/librados/aio_cxx.cc:1467: Skipped 2024-08-02T23:44:53.502 INFO:tasks.workunit.client.0.smithi073.stdout:Not supported by crimson yet. Skipped 2024-08-02T23:44:53.502 INFO:tasks.workunit.client.0.smithi073.stdout:[ SKIPPED ] LibRadosAioEC.RoundTripPP2 (0 ms) 2024-08-02T23:44:53.502 INFO:tasks.workunit.client.0.smithi073.stdout:[ RUN ] LibRadosAioEC.RoundTripPP3 2024-08-02T23:44:53.502 INFO:tasks.workunit.client.0.smithi073.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.1.0-1260-g26c3fb8e/rpm/el9/BUILD/ceph-19.1.0-1260-g26c3fb8e/src/test/librados/aio_cxx.cc:1500: Skipped 2024-08-02T23:44:53.502 INFO:tasks.workunit.client.0.smithi073.stdout:Not supported by crimson yet. Skipped 2024-08-02T23:44:53.502 INFO:tasks.workunit.client.0.smithi073.stdout:[ SKIPPED ] LibRadosAioEC.RoundTripPP3 (0 ms) 2024-08-02T23:44:53.502 INFO:tasks.workunit.client.0.smithi073.stdout:[ RUN ] LibRadosAioEC.RoundTripSparseReadPP 2024-08-02T23:44:53.502 INFO:tasks.workunit.client.0.smithi073.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.1.0-1260-g26c3fb8e/rpm/el9/BUILD/ceph-19.1.0-1260-g26c3fb8e/src/test/librados/aio_cxx.cc:1605: Skipped 2024-08-02T23:44:53.503 INFO:tasks.workunit.client.0.smithi073.stdout:Not supported by crimson yet. Skipped 2024-08-02T23:44:53.503 INFO:tasks.workunit.client.0.smithi073.stdout:[ SKIPPED ] LibRadosAioEC.RoundTripSparseReadPP (0 ms) 2024-08-02T23:44:53.503 INFO:tasks.workunit.client.0.smithi073.stdout:[ RUN ] LibRadosAioEC.RoundTripAppendPP 2024-08-02T23:44:53.503 INFO:tasks.workunit.client.0.smithi073.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.1.0-1260-g26c3fb8e/rpm/el9/BUILD/ceph-19.1.0-1260-g26c3fb8e/src/test/librados/aio_cxx.cc:1637: Skipped 2024-08-02T23:44:53.503 INFO:tasks.workunit.client.0.smithi073.stdout:Not supported by crimson yet. Skipped 2024-08-02T23:44:53.503 INFO:tasks.workunit.client.0.smithi073.stdout:[ SKIPPED ] LibRadosAioEC.RoundTripAppendPP (0 ms) 2024-08-02T23:44:53.503 INFO:tasks.workunit.client.0.smithi073.stdout:[ RUN ] LibRadosAioEC.IsCompletePP 2024-08-02T23:44:53.503 INFO:tasks.workunit.client.0.smithi073.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.1.0-1260-g26c3fb8e/rpm/el9/BUILD/ceph-19.1.0-1260-g26c3fb8e/src/test/librados/aio_cxx.cc:1705: Skipped 2024-08-02T23:44:53.504 INFO:tasks.workunit.client.0.smithi073.stdout:Not supported by crimson yet. Skipped 2024-08-02T23:44:53.504 INFO:tasks.workunit.client.0.smithi073.stdout:[ SKIPPED ] LibRadosAioEC.IsCompletePP (0 ms) 2024-08-02T23:44:53.504 INFO:tasks.workunit.client.0.smithi073.stdout:[ RUN ] LibRadosAioEC.IsSafePP 2024-08-02T23:44:53.504 INFO:tasks.workunit.client.0.smithi073.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.1.0-1260-g26c3fb8e/rpm/el9/BUILD/ceph-19.1.0-1260-g26c3fb8e/src/test/librados/aio_cxx.cc:1742: Skipped 2024-08-02T23:44:53.504 INFO:tasks.workunit.client.0.smithi073.stdout:Not supported by crimson yet. Skipped 2024-08-02T23:44:53.504 INFO:tasks.workunit.client.0.smithi073.stdout:[ SKIPPED ] LibRadosAioEC.IsSafePP (0 ms) 2024-08-02T23:44:53.504 INFO:tasks.workunit.client.0.smithi073.stdout:[ RUN ] LibRadosAioEC.ReturnValuePP 2024-08-02T23:44:53.504 INFO:tasks.workunit.client.0.smithi073.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.1.0-1260-g26c3fb8e/rpm/el9/BUILD/ceph-19.1.0-1260-g26c3fb8e/src/test/librados/aio_cxx.cc:1780: Skipped 2024-08-02T23:44:53.504 INFO:tasks.workunit.client.0.smithi073.stdout:Not supported by crimson yet. Skipped 2024-08-02T23:44:53.504 INFO:tasks.workunit.client.0.smithi073.stdout:[ SKIPPED ] LibRadosAioEC.ReturnValuePP (0 ms) 2024-08-02T23:44:53.504 INFO:tasks.workunit.client.0.smithi073.stdout:[ RUN ] LibRadosAioEC.FlushPP 2024-08-02T23:44:53.504 INFO:tasks.workunit.client.0.smithi073.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.1.0-1260-g26c3fb8e/rpm/el9/BUILD/ceph-19.1.0-1260-g26c3fb8e/src/test/librados/aio_cxx.cc:1796: Skipped 2024-08-02T23:44:53.504 INFO:tasks.workunit.client.0.smithi073.stdout:Not supported by crimson yet. Skipped 2024-08-02T23:44:53.504 INFO:tasks.workunit.client.0.smithi073.stdout:[ SKIPPED ] LibRadosAioEC.FlushPP (0 ms) 2024-08-02T23:44:53.505 INFO:tasks.workunit.client.0.smithi073.stdout:[ RUN ] LibRadosAioEC.FlushAsyncPP 2024-08-02T23:44:53.505 INFO:tasks.workunit.client.0.smithi073.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.1.0-1260-g26c3fb8e/rpm/el9/BUILD/ceph-19.1.0-1260-g26c3fb8e/src/test/librados/aio_cxx.cc:1824: Skipped 2024-08-02T23:44:53.505 INFO:tasks.workunit.client.0.smithi073.stdout:Not supported by crimson yet. Skipped 2024-08-02T23:44:53.505 INFO:tasks.workunit.client.0.smithi073.stdout:[ SKIPPED ] LibRadosAioEC.FlushAsyncPP (0 ms) 2024-08-02T23:44:53.505 INFO:tasks.workunit.client.0.smithi073.stdout:[ RUN ] LibRadosAioEC.RoundTripWriteFullPP 2024-08-02T23:44:53.505 INFO:tasks.workunit.client.0.smithi073.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.1.0-1260-g26c3fb8e/rpm/el9/BUILD/ceph-19.1.0-1260-g26c3fb8e/src/test/librados/aio_cxx.cc:1859: Skipped 2024-08-02T23:44:53.505 INFO:tasks.workunit.client.0.smithi073.stdout:Not supported by crimson yet. Skipped 2024-08-02T23:44:53.505 INFO:tasks.workunit.client.0.smithi073.stdout:[ SKIPPED ] LibRadosAioEC.RoundTripWriteFullPP (0 ms) 2024-08-02T23:44:53.505 INFO:tasks.workunit.client.0.smithi073.stdout:[ RUN ] LibRadosAioEC.RoundTripWriteFullPP2 2024-08-02T23:44:53.506 INFO:tasks.workunit.client.0.smithi073.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.1.0-1260-g26c3fb8e/rpm/el9/BUILD/ceph-19.1.0-1260-g26c3fb8e/src/test/librados/aio_cxx.cc:1904: Skipped 2024-08-02T23:44:53.506 INFO:tasks.workunit.client.0.smithi073.stdout:Not supported by crimson yet. Skipped 2024-08-02T23:44:53.506 INFO:tasks.workunit.client.0.smithi073.stdout:[ SKIPPED ] LibRadosAioEC.RoundTripWriteFullPP2 (0 ms) 2024-08-02T23:44:53.506 INFO:tasks.workunit.client.0.smithi073.stdout:[ RUN ] LibRadosAioEC.SimpleStatPP 2024-08-02T23:44:53.506 INFO:tasks.workunit.client.0.smithi073.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.1.0-1260-g26c3fb8e/rpm/el9/BUILD/ceph-19.1.0-1260-g26c3fb8e/src/test/librados/aio_cxx.cc:1946: Skipped 2024-08-02T23:44:53.506 INFO:tasks.workunit.client.0.smithi073.stdout:Not supported by crimson yet. Skipped 2024-08-02T23:44:53.506 INFO:tasks.workunit.client.0.smithi073.stdout:[ SKIPPED ] LibRadosAioEC.SimpleStatPP (0 ms) 2024-08-02T23:44:53.506 INFO:tasks.workunit.client.0.smithi073.stdout:[ RUN ] LibRadosAioEC.SimpleStatPPNS 2024-08-02T23:44:53.506 INFO:tasks.workunit.client.0.smithi073.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.1.0-1260-g26c3fb8e/rpm/el9/BUILD/ceph-19.1.0-1260-g26c3fb8e/src/test/librados/aio_cxx.cc:1977: Skipped 2024-08-02T23:44:53.506 INFO:tasks.workunit.client.0.smithi073.stdout:Not supported by crimson yet. Skipped 2024-08-02T23:44:53.506 INFO:tasks.workunit.client.0.smithi073.stdout:[ SKIPPED ] LibRadosAioEC.SimpleStatPPNS (0 ms) 2024-08-02T23:44:53.506 INFO:tasks.workunit.client.0.smithi073.stdout:[ RUN ] LibRadosAioEC.StatRemovePP 2024-08-02T23:44:53.506 INFO:tasks.workunit.client.0.smithi073.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.1.0-1260-g26c3fb8e/rpm/el9/BUILD/ceph-19.1.0-1260-g26c3fb8e/src/test/librados/aio_cxx.cc:2008: Skipped 2024-08-02T23:44:53.507 INFO:tasks.workunit.client.0.smithi073.stdout:Not supported by crimson yet. Skipped 2024-08-02T23:44:53.507 INFO:tasks.workunit.client.0.smithi073.stdout:[ SKIPPED ] LibRadosAioEC.StatRemovePP (0 ms) 2024-08-02T23:44:53.507 INFO:tasks.workunit.client.0.smithi073.stdout:[ RUN ] LibRadosAioEC.ExecuteClassPP 2024-08-02T23:44:53.507 INFO:tasks.workunit.client.0.smithi073.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.1.0-1260-g26c3fb8e/rpm/el9/BUILD/ceph-19.1.0-1260-g26c3fb8e/src/test/librados/aio_cxx.cc:2059: Skipped 2024-08-02T23:44:53.507 INFO:tasks.workunit.client.0.smithi073.stdout:Not supported by crimson yet. Skipped 2024-08-02T23:44:53.507 INFO:tasks.workunit.client.0.smithi073.stdout:[ SKIPPED ] LibRadosAioEC.ExecuteClassPP (0 ms) 2024-08-02T23:44:53.507 INFO:tasks.workunit.client.0.smithi073.stdout:[ RUN ] LibRadosAioEC.OmapPP 2024-08-02T23:44:53.507 INFO:tasks.workunit.client.0.smithi073.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.1.0-1260-g26c3fb8e/rpm/el9/BUILD/ceph-19.1.0-1260-g26c3fb8e/src/test/librados/aio_cxx.cc:2089: Skipped 2024-08-02T23:44:53.507 INFO:tasks.workunit.client.0.smithi073.stdout:Not supported by crimson yet. Skipped 2024-08-02T23:44:53.507 INFO:tasks.workunit.client.0.smithi073.stdout:[ SKIPPED ] LibRadosAioEC.OmapPP (0 ms) 2024-08-02T23:44:53.507 INFO:tasks.workunit.client.0.smithi073.stdout:[ RUN ] LibRadosAioEC.MultiWritePP 2024-08-02T23:44:53.507 INFO:tasks.workunit.client.0.smithi073.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.1.0-1260-g26c3fb8e/rpm/el9/BUILD/ceph-19.1.0-1260-g26c3fb8e/src/test/librados/aio_cxx.cc:2124: Skipped 2024-08-02T23:44:53.507 INFO:tasks.workunit.client.0.smithi073.stdout:Not supported by crimson yet. Skipped 2024-08-02T23:44:53.507 INFO:tasks.workunit.client.0.smithi073.stdout:[ SKIPPED ] LibRadosAioEC.MultiWritePP (0 ms) 2024-08-02T23:44:53.508 INFO:tasks.workunit.client.0.smithi073.stdout:[----------] 20 tests from LibRadosAioEC (0 ms total) 2024-08-02T23:44:53.508 INFO:tasks.workunit.client.0.smithi073.stdout: 2024-08-02T23:44:53.508 INFO:tasks.workunit.client.0.smithi073.stdout:[----------] Global test environment tear-down 2024-08-02T23:44:53.508 INFO:tasks.workunit.client.0.smithi073.stdout:[==========] 57 tests from 4 test suites ran. (159066 ms total) 2024-08-02T23:44:53.508 INFO:tasks.workunit.client.0.smithi073.stdout:[ PASSED ] 35 tests. 2024-08-02T23:44:53.508 INFO:tasks.workunit.client.0.smithi073.stdout:[ SKIPPED ] 21 tests, listed below: 2024-08-02T23:44:53.508 INFO:tasks.workunit.client.0.smithi073.stdout:[ SKIPPED ] LibRadosAio.MultiReads 2024-08-02T23:44:53.508 INFO:tasks.workunit.client.0.smithi073.stdout:[ SKIPPED ] LibRadosAioEC.SimpleWritePP 2024-08-02T23:44:53.508 INFO:tasks.workunit.client.0.smithi073.stdout:[ SKIPPED ] LibRadosAioEC.WaitForSafePP 2024-08-02T23:44:53.508 INFO:tasks.workunit.client.0.smithi073.stdout:[ SKIPPED ] LibRadosAioEC.RoundTripPP 2024-08-02T23:44:53.508 INFO:tasks.workunit.client.0.smithi073.stdout:[ SKIPPED ] LibRadosAioEC.RoundTripPP2 2024-08-02T23:44:53.508 INFO:tasks.workunit.client.0.smithi073.stdout:[ SKIPPED ] LibRadosAioEC.RoundTripPP3 2024-08-02T23:44:53.508 INFO:tasks.workunit.client.0.smithi073.stdout:[ SKIPPED ] LibRadosAioEC.RoundTripSparseReadPP 2024-08-02T23:44:53.508 INFO:tasks.workunit.client.0.smithi073.stdout:[ SKIPPED ] LibRadosAioEC.RoundTripAppendPP 2024-08-02T23:44:53.509 INFO:tasks.workunit.client.0.smithi073.stdout:[ SKIPPED ] LibRadosAioEC.IsCompletePP 2024-08-02T23:44:53.509 INFO:tasks.workunit.client.0.smithi073.stdout:[ SKIPPED ] LibRadosAioEC.IsSafePP 2024-08-02T23:44:53.509 INFO:tasks.workunit.client.0.smithi073.stdout:[ SKIPPED ] LibRadosAioEC.ReturnValuePP 2024-08-02T23:44:53.509 INFO:tasks.workunit.client.0.smithi073.stdout:[ SKIPPED ] LibRadosAioEC.FlushPP 2024-08-02T23:44:53.509 INFO:tasks.workunit.client.0.smithi073.stdout:[ SKIPPED ] LibRadosAioEC.FlushAsyncPP 2024-08-02T23:44:53.509 INFO:tasks.workunit.client.0.smithi073.stdout:[ SKIPPED ] LibRadosAioEC.RoundTripWriteFullPP 2024-08-02T23:44:53.509 INFO:tasks.workunit.client.0.smithi073.stdout:[ SKIPPED ] LibRadosAioEC.RoundTripWriteFullPP2 2024-08-02T23:44:53.509 INFO:tasks.workunit.client.0.smithi073.stdout:[ SKIPPED ] LibRadosAioEC.SimpleStatPP 2024-08-02T23:44:53.509 INFO:tasks.workunit.client.0.smithi073.stdout:[ SKIPPED ] LibRadosAioEC.SimpleStatPPNS 2024-08-02T23:44:53.509 INFO:tasks.workunit.client.0.smithi073.stdout:[ SKIPPED ] LibRadosAioEC.StatRemovePP 2024-08-02T23:44:53.509 INFO:tasks.workunit.client.0.smithi073.stdout:[ SKIPPED ] LibRadosAioEC.ExecuteClassPP 2024-08-02T23:44:53.509 INFO:tasks.workunit.client.0.smithi073.stdout:[ SKIPPED ] LibRadosAioEC.OmapPP 2024-08-02T23:44:53.509 INFO:tasks.workunit.client.0.smithi073.stdout:[ SKIPPED ] LibRadosAioEC.MultiWritePP 2024-08-02T23:44:53.509 INFO:tasks.workunit.client.0.smithi073.stdout:[ FAILED ] 1 test, listed below: 2024-08-02T23:44:53.509 INFO:tasks.workunit.client.0.smithi073.stdout:[ FAILED ] LibRadosAio.PoolEIOFlag 2024-08-02T23:44:53.510 INFO:tasks.workunit.client.0.smithi073.stdout: 2024-08-02T23:44:53.510 INFO:tasks.workunit.client.0.smithi073.stdout: 1 FAILED TEST 2024-08-02T23:44:53.510 INFO:tasks.workunit.client.0.smithi073.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.1.0-1260-g26c3fb8e/rpm/el9/BUILD/ceph-19.1.0-1260-g26c3fb8e/src/common/mutex_debug.h: In function 'ceph::mutex_debug_detail::mutex_debug_impl< >::~mutex_debug_impl() [with bool Recursive = false]' thread 7fa41c27f780 time 2024-08-02T23:44:53.499125+0000 2024-08-02T23:44:53.510 INFO:tasks.workunit.client.0.smithi073.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.1.0-1260-g26c3fb8e/rpm/el9/BUILD/ceph-19.1.0-1260-g26c3fb8e/src/common/mutex_debug.h: 110: FAILED ceph_assert(r == 0) 2024-08-02T23:44:53.510 INFO:tasks.workunit.client.0.smithi073.stderr: ceph version 19.1.0-1260-g26c3fb8e (26c3fb8e197dcf7a49a54d1f4c8a7362ee35a8ea) squid (rc) 2024-08-02T23:44:53.510 INFO:tasks.workunit.client.0.smithi073.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x120) [0x7fa41c626495] 2024-08-02T23:44:53.510 INFO:tasks.workunit.client.0.smithi073.stderr: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2266a2) [0x7fa41c6266a2] 2024-08-02T23:44:53.510 INFO:tasks.workunit.client.0.smithi073.stderr: 3: ceph_test_rados_api_aio_pp(+0x75d97) [0x557f90937d97] 2024-08-02T23:44:53.511 INFO:tasks.workunit.client.0.smithi073.stderr: 4: /lib64/libc.so.6(+0x40ded) [0x7fa41bc40ded] 2024-08-02T23:44:53.511 INFO:tasks.workunit.client.0.smithi073.stderr: 5: on_exit() 2024-08-02T23:44:53.511 INFO:tasks.workunit.client.0.smithi073.stderr: 6: /lib64/libc.so.6(+0x29597) [0x7fa41bc29597] 2024-08-02T23:44:53.511 INFO:tasks.workunit.client.0.smithi073.stderr: 7: __libc_start_main() 2024-08-02T23:44:53.511 INFO:tasks.workunit.client.0.smithi073.stderr: 8: _start() 2024-08-02T23:44:54.074 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh: line 24: 31410 Aborted (core dumped) ceph_test_rados_$f 2024-08-02T23:44:54.074 INFO:tasks.workunit.client.0.smithi073.stderr:++ cleanup 2024-08-02T23:44:54.074 INFO:tasks.workunit.client.0.smithi073.stderr:++ pkill -P 30939 2024-08-02T23:44:54.105 INFO:tasks.workunit.client.0.smithi073.stderr:++ true 2024-08-02T23:44:54.105 INFO:tasks.workunit.client.0.smithi073.stderr:+ cleanup 2024-08-02T23:44:54.105 INFO:tasks.workunit.client.0.smithi073.stderr:+ pkill -P 30939 2024-08-02T23:44:54.109 DEBUG:teuthology.orchestra.run:got remote process result: 134 2024-08-02T23:44:54.110 INFO:tasks.workunit.client.0.smithi073.stderr:+ true 2024-08-02T23:44:54.110 INFO:tasks.workunit:Stopping ['rados/test.sh --crimson', 'rados/test_pool_quota.sh'] on client.0... 2024-08-02T23:44:54.110 DEBUG:teuthology.orchestra.run.smithi073:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-08-02T23:44:54.431 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_26c3fb8e197dcf7a49a54d1f4c8a7362ee35a8ea/qa/tasks/workunit.py", line 126, in task with parallel() as p: File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_26c3fb8e197dcf7a49a54d1f4c8a7362ee35a8ea/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test rados/test.sh) on smithi073 with status 134: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=26c3fb8e197dcf7a49a54d1f4c8a7362ee35a8ea TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 CRIMSON_COMPAT=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh --crimson' 2024-08-02T23:44:54.618 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=d16edf7c858f4f85a8823bcf33b92306 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_26c3fb8e197dcf7a49a54d1f4c8a7362ee35a8ea/qa/tasks/workunit.py", line 126, in task with parallel() as p: File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_26c3fb8e197dcf7a49a54d1f4c8a7362ee35a8ea/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test rados/test.sh) on smithi073 with status 134: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=26c3fb8e197dcf7a49a54d1f4c8a7362ee35a8ea TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 CRIMSON_COMPAT=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh --crimson' 2024-08-02T23:44:54.620 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-08-02T23:44:54.629 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-08-02T23:44:54.630 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-08-02T23:44:54.738 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:54.733+0000 7fc621599640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-02T23:44:54.739 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:54.737+0000 7fc621599640 1 Processor -- start 2024-08-02T23:44:54.739 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:54.737+0000 7fc621599640 1 -- start start 2024-08-02T23:44:54.740 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:54.738+0000 7fc621599640 1 --2- >> v2:172.21.15.73:3300/0 conn(0x7fc61c1325d0 0x7fc61c1329e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-02T23:44:54.740 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:54.738+0000 7fc621599640 1 -- --> v1:172.21.15.73:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc61c132180 con 0x7fc61c072500 2024-08-02T23:44:54.741 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:54.738+0000 7fc621599640 1 -- --> v2:172.21.15.73:3300/0 -- mon_getmap magic: 0 -- 0x7fc61c09a9f0 con 0x7fc61c1325d0 2024-08-02T23:44:54.741 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:54.738+0000 7fc61affd640 1 --2- >> v2:172.21.15.73:3300/0 conn(0x7fc61c1325d0 0x7fc61c1329e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-02T23:44:54.741 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:54.738+0000 7fc61a7fc640 1 --1- >> v1:172.21.15.73:6789/0 conn(0x7fc61c072500 0x7fc61c072910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.73:6789/0 says I am v1:172.21.15.73:43144/0 (socket says 172.21.15.73:43144) 2024-08-02T23:44:54.741 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:54.738+0000 7fc61a7fc640 1 -- 172.21.15.73:0/1456652151 learned_addr learned my addr 172.21.15.73:0/1456652151 (peer_addr_for_me v1:172.21.15.73:0/0) 2024-08-02T23:44:54.741 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:54.738+0000 7fc61affd640 1 -- 172.21.15.73:0/1456652151 >> v1:172.21.15.73:6789/0 conn(0x7fc61c072500 legacy=0x7fc61c072910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-02T23:44:54.741 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:54.738+0000 7fc61affd640 1 -- 172.21.15.73:0/1456652151 --> v2:172.21.15.73:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc61c09ad00 con 0x7fc61c1325d0 2024-08-02T23:44:54.741 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:54.738+0000 7fc61affd640 1 --2- 172.21.15.73:0/1456652151 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc61c1325d0 0x7fc61c1329e0 crc :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=450fdbc4940c61bd server_cookie=0 in_seq=0 out_seq=0 2024-08-02T23:44:54.741 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:54.739+0000 7fc619ffb640 1 -- 172.21.15.73:0/1456652151 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fc60c009ce0 con 0x7fc61c1325d0 2024-08-02T23:44:54.741 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:54.739+0000 7fc619ffb640 1 -- 172.21.15.73:0/1456652151 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7fc60c009e40 con 0x7fc61c1325d0 2024-08-02T23:44:54.742 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:54.739+0000 7fc619ffb640 1 -- 172.21.15.73:0/1456652151 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fc60c00e360 con 0x7fc61c1325d0 2024-08-02T23:44:54.744 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:54.739+0000 7fc621599640 1 -- 172.21.15.73:0/1456652151 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc61c1325d0 msgr2=0x7fc61c1329e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-02T23:44:54.744 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:54.739+0000 7fc621599640 1 --2- 172.21.15.73:0/1456652151 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc61c1325d0 0x7fc61c1329e0 crc :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:44:54.744 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:54.739+0000 7fc621599640 1 -- 172.21.15.73:0/1456652151 shutdown_connections 2024-08-02T23:44:54.744 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:54.739+0000 7fc621599640 1 --2- 172.21.15.73:0/1456652151 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc61c1325d0 0x7fc61c1329e0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:44:54.744 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:54.739+0000 7fc621599640 1 -- 172.21.15.73:0/1456652151 >> 172.21.15.73:0/1456652151 conn(0x7fc61c0a02d0 msgr2=0x7fc61c0a06e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-02T23:44:54.744 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:54.739+0000 7fc621599640 1 -- 172.21.15.73:0/1456652151 shutdown_connections 2024-08-02T23:44:54.744 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:54.739+0000 7fc621599640 1 -- 172.21.15.73:0/1456652151 wait complete. 2024-08-02T23:44:54.745 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:54.740+0000 7fc621599640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-02T23:44:54.745 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:54.741+0000 7fc621599640 1 Processor -- start 2024-08-02T23:44:54.745 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:54.741+0000 7fc621599640 1 -- start start 2024-08-02T23:44:54.745 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:54.741+0000 7fc621599640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc61c072500 0x7fc61c0702f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-02T23:44:54.745 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:54.741+0000 7fc621599640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 -- 0x7fc61c0a2dc0 con 0x7fc61c072500 2024-08-02T23:44:54.745 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:54.741+0000 7fc61affd640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc61c072500 0x7fc61c0702f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-02T23:44:54.745 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:54.741+0000 7fc61affd640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc61c072500 0x7fc61c0702f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:58994/0 (socket says 172.21.15.73:58994) 2024-08-02T23:44:54.745 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:54.741+0000 7fc61affd640 1 -- 172.21.15.73:0/1633080269 learned_addr learned my addr 172.21.15.73:0/1633080269 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-08-02T23:44:54.746 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:54.742+0000 7fc61affd640 1 -- 172.21.15.73:0/1633080269 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc61c09b640 con 0x7fc61c072500 2024-08-02T23:44:54.746 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:54.742+0000 7fc61affd640 1 --2- 172.21.15.73:0/1633080269 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc61c072500 0x7fc61c0702f0 crc :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-02T23:44:54.746 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:54.742+0000 7fc6037fe640 1 -- 172.21.15.73:0/1633080269 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fc60c014620 con 0x7fc61c072500 2024-08-02T23:44:54.746 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:54.742+0000 7fc6037fe640 1 -- 172.21.15.73:0/1633080269 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7fc60c014780 con 0x7fc61c072500 2024-08-02T23:44:54.746 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:54.742+0000 7fc621599640 1 -- 172.21.15.73:0/1633080269 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc61c123750 con 0x7fc61c072500 2024-08-02T23:44:54.746 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:54.742+0000 7fc6037fe640 1 -- 172.21.15.73:0/1633080269 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fc60c014a20 con 0x7fc61c072500 2024-08-02T23:44:54.746 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:54.742+0000 7fc621599640 1 -- 172.21.15.73:0/1633080269 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc61c124480 con 0x7fc61c072500 2024-08-02T23:44:54.747 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:54.743+0000 7fc621599640 1 -- 172.21.15.73:0/1633080269 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc61c124670 con 0x7fc61c072500 2024-08-02T23:44:54.747 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:54.743+0000 7fc6037fe640 1 -- 172.21.15.73:0/1633080269 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 5) ==== 45563+0+0 (crc 0 0 0) 0x7fc60c014bd0 con 0x7fc61c072500 2024-08-02T23:44:54.747 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:54.743+0000 7fc6037fe640 1 --2- 172.21.15.73:0/1633080269 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7fc608038390 0x7fc60803a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-02T23:44:54.747 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:54.743+0000 7fc6037fe640 1 -- 172.21.15.73:0/1633080269 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(230..230 src has 1..230) ==== 2700+0+0 (crc 0 0 0) 0x7fc60c019040 con 0x7fc61c072500 2024-08-02T23:44:54.748 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:54.743+0000 7fc61a7fc640 1 --2- 172.21.15.73:0/1633080269 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7fc608038390 0x7fc60803a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-02T23:44:54.748 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:54.744+0000 7fc61a7fc640 1 --2- 172.21.15.73:0/1633080269 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7fc608038390 0x7fc60803a8a0 crc :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-02T23:44:54.750 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:54.748+0000 7fc6037fe640 1 -- 172.21.15.73:0/1633080269 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186747 (crc 0 0 0) 0x7fc60c018c00 con 0x7fc61c072500 2024-08-02T23:44:54.928 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:54.926+0000 7fc621599640 1 -- 172.21.15.73:0/1633080269 --> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fc61c0a2560 con 0x7fc608038390 2024-08-02T23:44:54.929 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:54.927+0000 7fc6037fe640 1 -- 172.21.15.73:0/1633080269 <== mgr.4100 v2:172.21.15.131:6800/3351532440 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+31247 (crc 0 0 0) 0x7fc610001250 con 0x7fc608038390 2024-08-02T23:44:54.930 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-08-02T23:44:54.930 INFO:teuthology.orchestra.run.smithi073.stderr:dumped all 2024-08-02T23:44:54.931 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:54.929+0000 7fc621599640 1 -- 172.21.15.73:0/1633080269 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7fc608038390 msgr2=0x7fc60803a8a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-02T23:44:54.931 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:54.929+0000 7fc621599640 1 --2- 172.21.15.73:0/1633080269 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7fc608038390 0x7fc60803a8a0 crc :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:44:54.932 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:54.929+0000 7fc621599640 1 -- 172.21.15.73:0/1633080269 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc61c072500 msgr2=0x7fc61c0702f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-02T23:44:54.932 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:54.929+0000 7fc621599640 1 --2- 172.21.15.73:0/1633080269 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc61c072500 0x7fc61c0702f0 crc :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:44:54.932 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:54.929+0000 7fc621599640 1 -- 172.21.15.73:0/1633080269 shutdown_connections 2024-08-02T23:44:54.932 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:54.929+0000 7fc621599640 1 --2- 172.21.15.73:0/1633080269 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7fc608038390 0x7fc60803a8a0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:44:54.932 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:54.929+0000 7fc621599640 1 --2- 172.21.15.73:0/1633080269 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc61c072500 0x7fc61c0702f0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:44:54.932 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:54.929+0000 7fc621599640 1 -- 172.21.15.73:0/1633080269 >> 172.21.15.73:0/1633080269 conn(0x7fc61c0a02d0 msgr2=0x7fc61c1309b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-02T23:44:54.932 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:54.929+0000 7fc621599640 1 -- 172.21.15.73:0/1633080269 shutdown_connections 2024-08-02T23:44:54.933 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:54.929+0000 7fc621599640 1 -- 172.21.15.73:0/1633080269 wait complete. 2024-08-02T23:44:54.942 INFO:teuthology.orchestra.run.smithi073.stdout:{"pg_ready":true,"pg_map":{"version":369,"stamp":"2024-08-02T23:44:53.494319+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":209190912,"kb_used":22608,"kb_used_data":22608,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":209168304,"statfs":{"total":214211493888,"available":214188343296,"internally_reserved":0,"allocated":23150592,"data_stored":23150592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.060378"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":582,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-08-02T23:44:28.441037+0000","last_change":"2024-08-02T23:40:06.540330+0000","last_active":"2024-08-02T23:44:28.441037+0000","last_peered":"2024-08-02T23:44:28.441037+0000","last_clean":"2024-08-02T23:44:28.441037+0000","last_became_active":"2024-08-02T23:40:04.552702+0000","last_became_peered":"2024-08-02T23:40:04.552702+0000","last_unstale":"2024-08-02T23:44:28.441037+0000","last_undegraded":"2024-08-02T23:44:28.441037+0000","last_fullsized":"2024-08-02T23:44:28.441037+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-02T23:40:03.497423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-02T23:40:03.497423+0000","last_clean_scrub_stamp":"2024-08-02T23:40:03.497423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":582,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-08-02T23:44:28.449743+0000","last_change":"2024-08-02T23:40:06.542935+0000","last_active":"2024-08-02T23:44:28.449743+0000","last_peered":"2024-08-02T23:44:28.449743+0000","last_clean":"2024-08-02T23:44:28.449743+0000","last_became_active":"2024-08-02T23:40:04.561840+0000","last_became_peered":"2024-08-02T23:40:04.561840+0000","last_unstale":"2024-08-02T23:44:28.449743+0000","last_undegraded":"2024-08-02T23:44:28.449743+0000","last_fullsized":"2024-08-02T23:44:28.449743+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-02T23:40:03.497423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-02T23:40:03.497423+0000","last_clean_scrub_stamp":"2024-08-02T23:40:03.497423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":588,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-08-02T23:44:30.466675+0000","last_change":"2024-08-02T23:40:06.540945+0000","last_active":"2024-08-02T23:44:30.466675+0000","last_peered":"2024-08-02T23:44:30.466675+0000","last_clean":"2024-08-02T23:44:30.466675+0000","last_became_active":"2024-08-02T23:40:04.560865+0000","last_became_peered":"2024-08-02T23:40:04.560865+0000","last_unstale":"2024-08-02T23:44:30.466675+0000","last_undegraded":"2024-08-02T23:44:30.466675+0000","last_fullsized":"2024-08-02T23:44:30.466675+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-02T23:40:03.497423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-02T23:40:03.497423+0000","last_clean_scrub_stamp":"2024-08-02T23:40:03.497423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":582,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-08-02T23:44:28.444871+0000","last_change":"2024-08-02T23:40:06.540508+0000","last_active":"2024-08-02T23:44:28.444871+0000","last_peered":"2024-08-02T23:44:28.444871+0000","last_clean":"2024-08-02T23:44:28.444871+0000","last_became_active":"2024-08-02T23:40:04.553411+0000","last_became_peered":"2024-08-02T23:40:04.553411+0000","last_unstale":"2024-08-02T23:44:28.444871+0000","last_undegraded":"2024-08-02T23:44:28.444871+0000","last_fullsized":"2024-08-02T23:44:28.444871+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-02T23:40:03.497423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-02T23:40:03.497423+0000","last_clean_scrub_stamp":"2024-08-02T23:40:03.497423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"16'2","reported_seq":584,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-08-02T23:44:28.446631+0000","last_change":"2024-08-02T23:40:06.538737+0000","last_active":"2024-08-02T23:44:28.446631+0000","last_peered":"2024-08-02T23:44:28.446631+0000","last_clean":"2024-08-02T23:44:28.446631+0000","last_became_active":"2024-08-02T23:40:04.553959+0000","last_became_peered":"2024-08-02T23:40:04.553959+0000","last_unstale":"2024-08-02T23:44:28.446631+0000","last_undegraded":"2024-08-02T23:44:28.446631+0000","last_fullsized":"2024-08-02T23:44:28.446631+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-02T23:40:03.497423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-02T23:40:03.497423+0000","last_clean_scrub_stamp":"2024-08-02T23:40:03.497423+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":582,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-08-02T23:44:28.442643+0000","last_change":"2024-08-02T23:40:06.540125+0000","last_active":"2024-08-02T23:44:28.442643+0000","last_peered":"2024-08-02T23:44:28.442643+0000","last_clean":"2024-08-02T23:44:28.442643+0000","last_became_active":"2024-08-02T23:40:04.556895+0000","last_became_peered":"2024-08-02T23:40:04.556895+0000","last_unstale":"2024-08-02T23:44:28.442643+0000","last_undegraded":"2024-08-02T23:44:28.442643+0000","last_fullsized":"2024-08-02T23:44:28.442643+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-02T23:40:03.497423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-02T23:40:03.497423+0000","last_clean_scrub_stamp":"2024-08-02T23:40:03.497423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":582,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-08-02T23:44:28.444605+0000","last_change":"2024-08-02T23:40:06.538748+0000","last_active":"2024-08-02T23:44:28.444605+0000","last_peered":"2024-08-02T23:44:28.444605+0000","last_clean":"2024-08-02T23:44:28.444605+0000","last_became_active":"2024-08-02T23:40:04.561398+0000","last_became_peered":"2024-08-02T23:40:04.561398+0000","last_unstale":"2024-08-02T23:44:28.444605+0000","last_undegraded":"2024-08-02T23:44:28.444605+0000","last_fullsized":"2024-08-02T23:44:28.444605+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-02T23:40:03.497423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-02T23:40:03.497423+0000","last_clean_scrub_stamp":"2024-08-02T23:40:03.497423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"14'1","reported_seq":583,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-08-02T23:44:28.447433+0000","last_change":"2024-08-02T23:40:06.541978+0000","last_active":"2024-08-02T23:44:28.447433+0000","last_peered":"2024-08-02T23:44:28.447433+0000","last_clean":"2024-08-02T23:44:28.447433+0000","last_became_active":"2024-08-02T23:40:04.550062+0000","last_became_peered":"2024-08-02T23:40:04.550062+0000","last_unstale":"2024-08-02T23:44:28.447433+0000","last_undegraded":"2024-08-02T23:44:28.447433+0000","last_fullsized":"2024-08-02T23:44:28.447433+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-02T23:40:03.497423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-02T23:40:03.497423+0000","last_clean_scrub_stamp":"2024-08-02T23:40:03.497423+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":723,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-08-02T23:44:30.451164+0000","last_change":"2024-08-02T23:40:00.557074+0000","last_active":"2024-08-02T23:44:30.451164+0000","last_peered":"2024-08-02T23:44:30.451164+0000","last_clean":"2024-08-02T23:44:30.451164+0000","last_became_active":"2024-08-02T23:40:00.546868+0000","last_became_peered":"2024-08-02T23:40:00.546868+0000","last_unstale":"2024-08-02T23:44:30.451164+0000","last_undegraded":"2024-08-02T23:44:30.451164+0000","last_fullsized":"2024-08-02T23:44:30.451164+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-02T23:39:59.483999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-02T23:39:59.483999+0000","last_clean_scrub_stamp":"2024-08-02T23:39:59.483999+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":214211493888,"available":214188527616,"internally_reserved":0,"allocated":22966272,"data_stored":22966272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53552873472,"available":53546135552,"internally_reserved":0,"allocated":6737920,"data_stored":6737920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":3,"up_from":10,"seq":42949673230,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":6760,"kb_used_data":6760,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52290968,"statfs":{"total":53552873472,"available":53545951232,"internally_reserved":0,"allocated":6922240,"data_stored":6922240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705934,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":4932,"kb_used_data":4932,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52292796,"statfs":{"total":53552873472,"available":53547823104,"internally_reserved":0,"allocated":5050368,"data_stored":5050368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705934,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":4608,"kb_used_data":4608,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52293120,"statfs":{"total":53552873472,"available":53548154880,"internally_reserved":0,"allocated":4718592,"data_stored":4718592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705934,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":6308,"kb_used_data":6308,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52291420,"statfs":{"total":53552873472,"available":53546414080,"internally_reserved":0,"allocated":6459392,"data_stored":6459392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":3,"total":53552873472,"available":53546135552,"internally_reserved":0,"allocated":6737920,"data_stored":6737920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53552873472,"available":53546414080,"internally_reserved":0,"allocated":6459392,"data_stored":6459392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53552873472,"available":53548154880,"internally_reserved":0,"allocated":4718592,"data_stored":4718592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53552873472,"available":53547823104,"internally_reserved":0,"allocated":5050368,"data_stored":5050368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":53552873472,"available":53546135552,"internally_reserved":0,"allocated":6737920,"data_stored":6737920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-02T23:44:54.943 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-08-02T23:44:55.047 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:55.042+0000 7fbee4c6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-02T23:44:55.048 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:55.046+0000 7fbee4c6d640 1 Processor -- start 2024-08-02T23:44:55.048 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:55.046+0000 7fbee4c6d640 1 -- start start 2024-08-02T23:44:55.050 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:55.047+0000 7fbee4c6d640 1 --2- >> v2:172.21.15.73:3300/0 conn(0x7fbee00a4770 0x7fbee00a4b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-02T23:44:55.050 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:55.047+0000 7fbee4c6d640 1 -- --> v1:172.21.15.73:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbee006e030 con 0x7fbee00a50c0 2024-08-02T23:44:55.050 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:55.047+0000 7fbee4c6d640 1 -- --> v2:172.21.15.73:3300/0 -- mon_getmap magic: 0 -- 0x7fbee00a0ed0 con 0x7fbee00a4770 2024-08-02T23:44:55.050 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:55.047+0000 7fbede59c640 1 --2- >> v2:172.21.15.73:3300/0 conn(0x7fbee00a4770 0x7fbee00a4b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-02T23:44:55.050 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:55.047+0000 7fbeddd9b640 1 --1- >> v1:172.21.15.73:6789/0 conn(0x7fbee00a50c0 0x7fbee0131000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.73:6789/0 says I am v1:172.21.15.73:43156/0 (socket says 172.21.15.73:43156) 2024-08-02T23:44:55.051 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:55.047+0000 7fbeddd9b640 1 -- 172.21.15.73:0/1305220163 learned_addr learned my addr 172.21.15.73:0/1305220163 (peer_addr_for_me v1:172.21.15.73:0/0) 2024-08-02T23:44:55.051 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:55.048+0000 7fbede59c640 1 -- 172.21.15.73:0/1305220163 >> v1:172.21.15.73:6789/0 conn(0x7fbee00a50c0 legacy=0x7fbee0131000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-02T23:44:55.051 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:55.048+0000 7fbede59c640 1 -- 172.21.15.73:0/1305220163 --> v2:172.21.15.73:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbee006e260 con 0x7fbee00a4770 2024-08-02T23:44:55.051 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:55.048+0000 7fbede59c640 1 --2- 172.21.15.73:0/1305220163 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fbee00a4770 0x7fbee00a4b80 crc :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=241755755f7a3b23 server_cookie=0 in_seq=0 out_seq=0 2024-08-02T23:44:55.051 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:55.048+0000 7fbedd59a640 1 -- 172.21.15.73:0/1305220163 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fbec8009ce0 con 0x7fbee00a4770 2024-08-02T23:44:55.051 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:55.048+0000 7fbedd59a640 1 -- 172.21.15.73:0/1305220163 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7fbec8009e40 con 0x7fbee00a4770 2024-08-02T23:44:55.051 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:55.048+0000 7fbedd59a640 1 -- 172.21.15.73:0/1305220163 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fbec800e360 con 0x7fbee00a4770 2024-08-02T23:44:55.054 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:55.049+0000 7fbee4c6d640 1 -- 172.21.15.73:0/1305220163 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fbee00a4770 msgr2=0x7fbee00a4b80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-02T23:44:55.054 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:55.049+0000 7fbee4c6d640 1 --2- 172.21.15.73:0/1305220163 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fbee00a4770 0x7fbee00a4b80 crc :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:44:55.055 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:55.049+0000 7fbee4c6d640 1 -- 172.21.15.73:0/1305220163 shutdown_connections 2024-08-02T23:44:55.055 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:55.049+0000 7fbee4c6d640 1 --2- 172.21.15.73:0/1305220163 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fbee00a4770 0x7fbee00a4b80 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:44:55.055 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:55.049+0000 7fbee4c6d640 1 -- 172.21.15.73:0/1305220163 >> 172.21.15.73:0/1305220163 conn(0x7fbee00a3510 msgr2=0x7fbee00a3920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-02T23:44:55.055 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:55.049+0000 7fbee4c6d640 1 -- 172.21.15.73:0/1305220163 shutdown_connections 2024-08-02T23:44:55.055 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:55.049+0000 7fbee4c6d640 1 -- 172.21.15.73:0/1305220163 wait complete. 2024-08-02T23:44:55.055 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:55.049+0000 7fbee4c6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-02T23:44:55.055 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:55.050+0000 7fbee4c6d640 1 Processor -- start 2024-08-02T23:44:55.056 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:55.050+0000 7fbee4c6d640 1 -- start start 2024-08-02T23:44:55.056 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:55.051+0000 7fbee4c6d640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fbee00a4770 0x7fbee011db50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-02T23:44:55.056 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:55.051+0000 7fbee4c6d640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 -- 0x7fbee0133660 con 0x7fbee00a4770 2024-08-02T23:44:55.056 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:55.051+0000 7fbede59c640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fbee00a4770 0x7fbee011db50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-02T23:44:55.056 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:55.051+0000 7fbede59c640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fbee00a4770 0x7fbee011db50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:59006/0 (socket says 172.21.15.73:59006) 2024-08-02T23:44:55.056 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:55.051+0000 7fbede59c640 1 -- 172.21.15.73:0/2717504965 learned_addr learned my addr 172.21.15.73:0/2717504965 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-08-02T23:44:55.056 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:55.051+0000 7fbede59c640 1 -- 172.21.15.73:0/2717504965 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbee0120630 con 0x7fbee00a4770 2024-08-02T23:44:55.056 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:55.051+0000 7fbede59c640 1 --2- 172.21.15.73:0/2717504965 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fbee00a4770 0x7fbee011db50 crc :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-02T23:44:55.056 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:55.051+0000 7fbed6ffd640 1 -- 172.21.15.73:0/2717504965 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fbec8014620 con 0x7fbee00a4770 2024-08-02T23:44:55.057 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:55.051+0000 7fbed6ffd640 1 -- 172.21.15.73:0/2717504965 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7fbec8014780 con 0x7fbee00a4770 2024-08-02T23:44:55.057 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:55.051+0000 7fbee4c6d640 1 -- 172.21.15.73:0/2717504965 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbee0121050 con 0x7fbee00a4770 2024-08-02T23:44:55.057 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:55.051+0000 7fbed6ffd640 1 -- 172.21.15.73:0/2717504965 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fbec8014a20 con 0x7fbee00a4770 2024-08-02T23:44:55.057 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:55.051+0000 7fbee4c6d640 1 -- 172.21.15.73:0/2717504965 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbee011ffb0 con 0x7fbee00a4770 2024-08-02T23:44:55.057 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:55.052+0000 7fbed6ffd640 1 -- 172.21.15.73:0/2717504965 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 5) ==== 45563+0+0 (crc 0 0 0) 0x7fbec8014b80 con 0x7fbee00a4770 2024-08-02T23:44:55.057 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:55.052+0000 7fbee4c6d640 1 -- 172.21.15.73:0/2717504965 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbee009aba0 con 0x7fbee00a4770 2024-08-02T23:44:55.057 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:55.052+0000 7fbed6ffd640 1 --2- 172.21.15.73:0/2717504965 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7fbec0038390 0x7fbec003a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-02T23:44:55.057 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:55.052+0000 7fbed6ffd640 1 -- 172.21.15.73:0/2717504965 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(230..230 src has 1..230) ==== 2700+0+0 (crc 0 0 0) 0x7fbec8019040 con 0x7fbee00a4770 2024-08-02T23:44:55.057 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:55.053+0000 7fbeddd9b640 1 --2- 172.21.15.73:0/2717504965 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7fbec0038390 0x7fbec003a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-02T23:44:55.058 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:55.053+0000 7fbeddd9b640 1 --2- 172.21.15.73:0/2717504965 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7fbec0038390 0x7fbec003a8a0 crc :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-02T23:44:55.059 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:55.057+0000 7fbed6ffd640 1 -- 172.21.15.73:0/2717504965 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186747 (crc 0 0 0) 0x7fbec8018c00 con 0x7fbee00a4770 2024-08-02T23:44:55.236 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:55.234+0000 7fbee4c6d640 1 -- 172.21.15.73:0/2717504965 --> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fbee0073220 con 0x7fbec0038390 2024-08-02T23:44:55.237 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:55.236+0000 7fbed6ffd640 1 -- 172.21.15.73:0/2717504965 <== mgr.4100 v2:172.21.15.131:6800/3351532440 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+31247 (crc 0 0 0) 0x7fbee0073220 con 0x7fbec0038390 2024-08-02T23:44:55.238 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-08-02T23:44:55.238 INFO:teuthology.orchestra.run.smithi073.stderr:dumped all 2024-08-02T23:44:55.239 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:55.237+0000 7fbee4c6d640 1 -- 172.21.15.73:0/2717504965 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7fbec0038390 msgr2=0x7fbec003a8a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-02T23:44:55.240 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:55.237+0000 7fbee4c6d640 1 --2- 172.21.15.73:0/2717504965 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7fbec0038390 0x7fbec003a8a0 crc :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:44:55.240 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:55.237+0000 7fbee4c6d640 1 -- 172.21.15.73:0/2717504965 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fbee00a4770 msgr2=0x7fbee011db50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-02T23:44:55.240 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:55.237+0000 7fbee4c6d640 1 --2- 172.21.15.73:0/2717504965 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fbee00a4770 0x7fbee011db50 crc :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:44:55.240 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:55.237+0000 7fbee4c6d640 1 -- 172.21.15.73:0/2717504965 shutdown_connections 2024-08-02T23:44:55.240 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:55.237+0000 7fbee4c6d640 1 --2- 172.21.15.73:0/2717504965 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7fbec0038390 0x7fbec003a8a0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:44:55.240 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:55.237+0000 7fbee4c6d640 1 --2- 172.21.15.73:0/2717504965 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fbee00a4770 0x7fbee011db50 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:44:55.240 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:55.237+0000 7fbee4c6d640 1 -- 172.21.15.73:0/2717504965 >> 172.21.15.73:0/2717504965 conn(0x7fbee00a3510 msgr2=0x7fbee00a12e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-02T23:44:55.240 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:55.237+0000 7fbee4c6d640 1 -- 172.21.15.73:0/2717504965 shutdown_connections 2024-08-02T23:44:55.241 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:55.237+0000 7fbee4c6d640 1 -- 172.21.15.73:0/2717504965 wait complete. 2024-08-02T23:44:55.250 INFO:teuthology.orchestra.run.smithi073.stdout:{"pg_ready":true,"pg_map":{"version":369,"stamp":"2024-08-02T23:44:53.494319+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":209190912,"kb_used":22608,"kb_used_data":22608,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":209168304,"statfs":{"total":214211493888,"available":214188343296,"internally_reserved":0,"allocated":23150592,"data_stored":23150592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.060378"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":582,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-08-02T23:44:28.441037+0000","last_change":"2024-08-02T23:40:06.540330+0000","last_active":"2024-08-02T23:44:28.441037+0000","last_peered":"2024-08-02T23:44:28.441037+0000","last_clean":"2024-08-02T23:44:28.441037+0000","last_became_active":"2024-08-02T23:40:04.552702+0000","last_became_peered":"2024-08-02T23:40:04.552702+0000","last_unstale":"2024-08-02T23:44:28.441037+0000","last_undegraded":"2024-08-02T23:44:28.441037+0000","last_fullsized":"2024-08-02T23:44:28.441037+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-02T23:40:03.497423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-02T23:40:03.497423+0000","last_clean_scrub_stamp":"2024-08-02T23:40:03.497423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":582,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-08-02T23:44:28.449743+0000","last_change":"2024-08-02T23:40:06.542935+0000","last_active":"2024-08-02T23:44:28.449743+0000","last_peered":"2024-08-02T23:44:28.449743+0000","last_clean":"2024-08-02T23:44:28.449743+0000","last_became_active":"2024-08-02T23:40:04.561840+0000","last_became_peered":"2024-08-02T23:40:04.561840+0000","last_unstale":"2024-08-02T23:44:28.449743+0000","last_undegraded":"2024-08-02T23:44:28.449743+0000","last_fullsized":"2024-08-02T23:44:28.449743+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-02T23:40:03.497423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-02T23:40:03.497423+0000","last_clean_scrub_stamp":"2024-08-02T23:40:03.497423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":588,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-08-02T23:44:30.466675+0000","last_change":"2024-08-02T23:40:06.540945+0000","last_active":"2024-08-02T23:44:30.466675+0000","last_peered":"2024-08-02T23:44:30.466675+0000","last_clean":"2024-08-02T23:44:30.466675+0000","last_became_active":"2024-08-02T23:40:04.560865+0000","last_became_peered":"2024-08-02T23:40:04.560865+0000","last_unstale":"2024-08-02T23:44:30.466675+0000","last_undegraded":"2024-08-02T23:44:30.466675+0000","last_fullsized":"2024-08-02T23:44:30.466675+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-02T23:40:03.497423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-02T23:40:03.497423+0000","last_clean_scrub_stamp":"2024-08-02T23:40:03.497423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":582,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-08-02T23:44:28.444871+0000","last_change":"2024-08-02T23:40:06.540508+0000","last_active":"2024-08-02T23:44:28.444871+0000","last_peered":"2024-08-02T23:44:28.444871+0000","last_clean":"2024-08-02T23:44:28.444871+0000","last_became_active":"2024-08-02T23:40:04.553411+0000","last_became_peered":"2024-08-02T23:40:04.553411+0000","last_unstale":"2024-08-02T23:44:28.444871+0000","last_undegraded":"2024-08-02T23:44:28.444871+0000","last_fullsized":"2024-08-02T23:44:28.444871+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-02T23:40:03.497423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-02T23:40:03.497423+0000","last_clean_scrub_stamp":"2024-08-02T23:40:03.497423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"16'2","reported_seq":584,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-08-02T23:44:28.446631+0000","last_change":"2024-08-02T23:40:06.538737+0000","last_active":"2024-08-02T23:44:28.446631+0000","last_peered":"2024-08-02T23:44:28.446631+0000","last_clean":"2024-08-02T23:44:28.446631+0000","last_became_active":"2024-08-02T23:40:04.553959+0000","last_became_peered":"2024-08-02T23:40:04.553959+0000","last_unstale":"2024-08-02T23:44:28.446631+0000","last_undegraded":"2024-08-02T23:44:28.446631+0000","last_fullsized":"2024-08-02T23:44:28.446631+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-02T23:40:03.497423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-02T23:40:03.497423+0000","last_clean_scrub_stamp":"2024-08-02T23:40:03.497423+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":582,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-08-02T23:44:28.442643+0000","last_change":"2024-08-02T23:40:06.540125+0000","last_active":"2024-08-02T23:44:28.442643+0000","last_peered":"2024-08-02T23:44:28.442643+0000","last_clean":"2024-08-02T23:44:28.442643+0000","last_became_active":"2024-08-02T23:40:04.556895+0000","last_became_peered":"2024-08-02T23:40:04.556895+0000","last_unstale":"2024-08-02T23:44:28.442643+0000","last_undegraded":"2024-08-02T23:44:28.442643+0000","last_fullsized":"2024-08-02T23:44:28.442643+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-02T23:40:03.497423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-02T23:40:03.497423+0000","last_clean_scrub_stamp":"2024-08-02T23:40:03.497423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":582,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-08-02T23:44:28.444605+0000","last_change":"2024-08-02T23:40:06.538748+0000","last_active":"2024-08-02T23:44:28.444605+0000","last_peered":"2024-08-02T23:44:28.444605+0000","last_clean":"2024-08-02T23:44:28.444605+0000","last_became_active":"2024-08-02T23:40:04.561398+0000","last_became_peered":"2024-08-02T23:40:04.561398+0000","last_unstale":"2024-08-02T23:44:28.444605+0000","last_undegraded":"2024-08-02T23:44:28.444605+0000","last_fullsized":"2024-08-02T23:44:28.444605+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-02T23:40:03.497423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-02T23:40:03.497423+0000","last_clean_scrub_stamp":"2024-08-02T23:40:03.497423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"14'1","reported_seq":583,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-08-02T23:44:28.447433+0000","last_change":"2024-08-02T23:40:06.541978+0000","last_active":"2024-08-02T23:44:28.447433+0000","last_peered":"2024-08-02T23:44:28.447433+0000","last_clean":"2024-08-02T23:44:28.447433+0000","last_became_active":"2024-08-02T23:40:04.550062+0000","last_became_peered":"2024-08-02T23:40:04.550062+0000","last_unstale":"2024-08-02T23:44:28.447433+0000","last_undegraded":"2024-08-02T23:44:28.447433+0000","last_fullsized":"2024-08-02T23:44:28.447433+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-02T23:40:03.497423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-02T23:40:03.497423+0000","last_clean_scrub_stamp":"2024-08-02T23:40:03.497423+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":723,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-08-02T23:44:30.451164+0000","last_change":"2024-08-02T23:40:00.557074+0000","last_active":"2024-08-02T23:44:30.451164+0000","last_peered":"2024-08-02T23:44:30.451164+0000","last_clean":"2024-08-02T23:44:30.451164+0000","last_became_active":"2024-08-02T23:40:00.546868+0000","last_became_peered":"2024-08-02T23:40:00.546868+0000","last_unstale":"2024-08-02T23:44:30.451164+0000","last_undegraded":"2024-08-02T23:44:30.451164+0000","last_fullsized":"2024-08-02T23:44:30.451164+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-02T23:39:59.483999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-02T23:39:59.483999+0000","last_clean_scrub_stamp":"2024-08-02T23:39:59.483999+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":214211493888,"available":214188527616,"internally_reserved":0,"allocated":22966272,"data_stored":22966272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53552873472,"available":53546135552,"internally_reserved":0,"allocated":6737920,"data_stored":6737920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":3,"up_from":10,"seq":42949673230,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":6760,"kb_used_data":6760,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52290968,"statfs":{"total":53552873472,"available":53545951232,"internally_reserved":0,"allocated":6922240,"data_stored":6922240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705934,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":4932,"kb_used_data":4932,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52292796,"statfs":{"total":53552873472,"available":53547823104,"internally_reserved":0,"allocated":5050368,"data_stored":5050368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705934,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":4608,"kb_used_data":4608,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52293120,"statfs":{"total":53552873472,"available":53548154880,"internally_reserved":0,"allocated":4718592,"data_stored":4718592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705934,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":6308,"kb_used_data":6308,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52291420,"statfs":{"total":53552873472,"available":53546414080,"internally_reserved":0,"allocated":6459392,"data_stored":6459392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":3,"total":53552873472,"available":53546135552,"internally_reserved":0,"allocated":6737920,"data_stored":6737920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53552873472,"available":53546414080,"internally_reserved":0,"allocated":6459392,"data_stored":6459392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53552873472,"available":53548154880,"internally_reserved":0,"allocated":4718592,"data_stored":4718592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53552873472,"available":53547823104,"internally_reserved":0,"allocated":5050368,"data_stored":5050368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":53552873472,"available":53546135552,"internally_reserved":0,"allocated":6737920,"data_stored":6737920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-02T23:44:55.251 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-08-02T23:44:55.251 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-08-02T23:44:55.356 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:55.351+0000 7fada9c21640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-02T23:44:55.357 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:55.355+0000 7fada9c21640 1 Processor -- start 2024-08-02T23:44:55.357 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:55.355+0000 7fada9c21640 1 -- start start 2024-08-02T23:44:55.358 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:55.356+0000 7fada9c21640 1 --2- >> v2:172.21.15.73:3300/0 conn(0x7fada41320f0 0x7fada4132500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-02T23:44:55.359 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:55.356+0000 7fada9c21640 1 -- --> v1:172.21.15.73:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fada4073a40 con 0x7fada4132a40 2024-08-02T23:44:55.359 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:55.356+0000 7fada9c21640 1 -- --> v2:172.21.15.73:3300/0 -- mon_getmap magic: 0 -- 0x7fada4073810 con 0x7fada41320f0 2024-08-02T23:44:55.359 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:55.356+0000 7fada37fe640 1 --2- >> v2:172.21.15.73:3300/0 conn(0x7fada41320f0 0x7fada4132500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-02T23:44:55.359 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:55.356+0000 7fada2ffd640 1 --1- >> v1:172.21.15.73:6789/0 conn(0x7fada4132a40 0x7fada412ee40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.73:6789/0 says I am v1:172.21.15.73:43166/0 (socket says 172.21.15.73:43166) 2024-08-02T23:44:55.359 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:55.356+0000 7fada2ffd640 1 -- 172.21.15.73:0/2868284127 learned_addr learned my addr 172.21.15.73:0/2868284127 (peer_addr_for_me v1:172.21.15.73:0/0) 2024-08-02T23:44:55.359 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:55.356+0000 7fada37fe640 1 -- 172.21.15.73:0/2868284127 >> v1:172.21.15.73:6789/0 conn(0x7fada4132a40 legacy=0x7fada412ee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-02T23:44:55.359 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:55.356+0000 7fada37fe640 1 -- 172.21.15.73:0/2868284127 --> v2:172.21.15.73:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fada409dfe0 con 0x7fada41320f0 2024-08-02T23:44:55.360 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:55.356+0000 7fada37fe640 1 --2- 172.21.15.73:0/2868284127 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fada41320f0 0x7fada4132500 crc :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e35e0f0edaf0b5d server_cookie=0 in_seq=0 out_seq=0 2024-08-02T23:44:55.360 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:55.356+0000 7fada27fc640 1 -- 172.21.15.73:0/2868284127 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fad94009ce0 con 0x7fada41320f0 2024-08-02T23:44:55.360 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:55.357+0000 7fada27fc640 1 -- 172.21.15.73:0/2868284127 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7fad94009e40 con 0x7fada41320f0 2024-08-02T23:44:55.360 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:55.357+0000 7fada27fc640 1 -- 172.21.15.73:0/2868284127 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fad9400e360 con 0x7fada41320f0 2024-08-02T23:44:55.362 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:55.357+0000 7fada9c21640 1 -- 172.21.15.73:0/2868284127 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fada41320f0 msgr2=0x7fada4132500 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-02T23:44:55.363 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:55.357+0000 7fada9c21640 1 --2- 172.21.15.73:0/2868284127 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fada41320f0 0x7fada4132500 crc :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:44:55.363 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:55.357+0000 7fada9c21640 1 -- 172.21.15.73:0/2868284127 shutdown_connections 2024-08-02T23:44:55.363 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:55.357+0000 7fada9c21640 1 --2- 172.21.15.73:0/2868284127 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fada41320f0 0x7fada4132500 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:44:55.363 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:55.357+0000 7fada9c21640 1 -- 172.21.15.73:0/2868284127 >> 172.21.15.73:0/2868284127 conn(0x7fada40a0590 msgr2=0x7fada40a09a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-02T23:44:55.363 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:55.357+0000 7fada9c21640 1 -- 172.21.15.73:0/2868284127 shutdown_connections 2024-08-02T23:44:55.363 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:55.357+0000 7fada9c21640 1 -- 172.21.15.73:0/2868284127 wait complete. 2024-08-02T23:44:55.363 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:55.357+0000 7fada9c21640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-02T23:44:55.363 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:55.358+0000 7fada9c21640 1 Processor -- start 2024-08-02T23:44:55.364 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:55.358+0000 7fada9c21640 1 -- start start 2024-08-02T23:44:55.364 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:55.359+0000 7fada9c21640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fada41320f0 0x7fada411dcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-02T23:44:55.364 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:55.359+0000 7fada9c21640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 -- 0x7fada412ebd0 con 0x7fada41320f0 2024-08-02T23:44:55.364 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:55.359+0000 7fada37fe640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fada41320f0 0x7fada411dcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-02T23:44:55.364 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:55.359+0000 7fada37fe640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fada41320f0 0x7fada411dcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:59026/0 (socket says 172.21.15.73:59026) 2024-08-02T23:44:55.364 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:55.359+0000 7fada37fe640 1 -- 172.21.15.73:0/1849994331 learned_addr learned my addr 172.21.15.73:0/1849994331 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-08-02T23:44:55.364 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:55.359+0000 7fada37fe640 1 -- 172.21.15.73:0/1849994331 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fada411d560 con 0x7fada41320f0 2024-08-02T23:44:55.364 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:55.360+0000 7fada37fe640 1 --2- 172.21.15.73:0/1849994331 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fada41320f0 0x7fada411dcf0 crc :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-02T23:44:55.364 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:55.360+0000 7fad8bfff640 1 -- 172.21.15.73:0/1849994331 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fad94014620 con 0x7fada41320f0 2024-08-02T23:44:55.364 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:55.360+0000 7fad8bfff640 1 -- 172.21.15.73:0/1849994331 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7fad94014780 con 0x7fada41320f0 2024-08-02T23:44:55.365 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:55.360+0000 7fada9c21640 1 -- 172.21.15.73:0/1849994331 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fada411c620 con 0x7fada41320f0 2024-08-02T23:44:55.365 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:55.360+0000 7fad8bfff640 1 -- 172.21.15.73:0/1849994331 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fad94014a20 con 0x7fada41320f0 2024-08-02T23:44:55.365 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:55.360+0000 7fada9c21640 1 -- 172.21.15.73:0/1849994331 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fada411cf00 con 0x7fada41320f0 2024-08-02T23:44:55.365 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:55.361+0000 7fad8bfff640 1 -- 172.21.15.73:0/1849994331 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 5) ==== 45563+0+0 (crc 0 0 0) 0x7fad94014b80 con 0x7fada41320f0 2024-08-02T23:44:55.365 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:55.361+0000 7fada9c21640 1 -- 172.21.15.73:0/1849994331 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fada411bef0 con 0x7fada41320f0 2024-08-02T23:44:55.365 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:55.361+0000 7fad8bfff640 1 --2- 172.21.15.73:0/1849994331 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7fad90038390 0x7fad9003a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-02T23:44:55.365 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:55.361+0000 7fad8bfff640 1 -- 172.21.15.73:0/1849994331 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(230..230 src has 1..230) ==== 2700+0+0 (crc 0 0 0) 0x7fad94019040 con 0x7fada41320f0 2024-08-02T23:44:55.366 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:55.361+0000 7fada2ffd640 1 --2- 172.21.15.73:0/1849994331 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7fad90038390 0x7fad9003a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-02T23:44:55.366 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:55.362+0000 7fada2ffd640 1 --2- 172.21.15.73:0/1849994331 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7fad90038390 0x7fad9003a8a0 crc :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-02T23:44:55.367 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:55.365+0000 7fad8bfff640 1 -- 172.21.15.73:0/1849994331 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186747 (crc 0 0 0) 0x7fad94018c00 con 0x7fada41320f0 2024-08-02T23:44:55.537 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:55.535+0000 7fada9c21640 1 -- 172.21.15.73:0/1849994331 --> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fada412ea60 con 0x7fad90038390 2024-08-02T23:44:55.538 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:55.536+0000 7fad8bfff640 1 -- 172.21.15.73:0/1849994331 <== mgr.4100 v2:172.21.15.131:6800/3351532440 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+31247 (crc 0 0 0) 0x7fada412ea60 con 0x7fad90038390 2024-08-02T23:44:55.538 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-08-02T23:44:55.539 INFO:teuthology.orchestra.run.smithi073.stderr:dumped all 2024-08-02T23:44:55.540 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:55.538+0000 7fada9c21640 1 -- 172.21.15.73:0/1849994331 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7fad90038390 msgr2=0x7fad9003a8a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-02T23:44:55.541 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:55.538+0000 7fada9c21640 1 --2- 172.21.15.73:0/1849994331 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7fad90038390 0x7fad9003a8a0 crc :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:44:55.541 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:55.538+0000 7fada9c21640 1 -- 172.21.15.73:0/1849994331 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fada41320f0 msgr2=0x7fada411dcf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-02T23:44:55.541 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:55.538+0000 7fada9c21640 1 --2- 172.21.15.73:0/1849994331 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fada41320f0 0x7fada411dcf0 crc :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:44:55.541 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:55.538+0000 7fada9c21640 1 -- 172.21.15.73:0/1849994331 shutdown_connections 2024-08-02T23:44:55.541 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:55.538+0000 7fada9c21640 1 --2- 172.21.15.73:0/1849994331 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7fad90038390 0x7fad9003a8a0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:44:55.541 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:55.538+0000 7fada9c21640 1 --2- 172.21.15.73:0/1849994331 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fada41320f0 0x7fada411dcf0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:44:55.541 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:55.538+0000 7fada9c21640 1 -- 172.21.15.73:0/1849994331 >> 172.21.15.73:0/1849994331 conn(0x7fada40a0590 msgr2=0x7fada41ba390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-02T23:44:55.541 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:55.538+0000 7fada9c21640 1 -- 172.21.15.73:0/1849994331 shutdown_connections 2024-08-02T23:44:55.541 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:55.538+0000 7fada9c21640 1 -- 172.21.15.73:0/1849994331 wait complete. 2024-08-02T23:44:55.551 INFO:teuthology.orchestra.run.smithi073.stdout:{"pg_ready":true,"pg_map":{"version":370,"stamp":"2024-08-02T23:44:55.430631+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":209190912,"kb_used":22608,"kb_used_data":22608,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":209168304,"statfs":{"total":214211493888,"available":214188343296,"internally_reserved":0,"allocated":23150592,"data_stored":23150592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.947656"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":582,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-08-02T23:44:28.441037+0000","last_change":"2024-08-02T23:40:06.540330+0000","last_active":"2024-08-02T23:44:28.441037+0000","last_peered":"2024-08-02T23:44:28.441037+0000","last_clean":"2024-08-02T23:44:28.441037+0000","last_became_active":"2024-08-02T23:40:04.552702+0000","last_became_peered":"2024-08-02T23:40:04.552702+0000","last_unstale":"2024-08-02T23:44:28.441037+0000","last_undegraded":"2024-08-02T23:44:28.441037+0000","last_fullsized":"2024-08-02T23:44:28.441037+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-02T23:40:03.497423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-02T23:40:03.497423+0000","last_clean_scrub_stamp":"2024-08-02T23:40:03.497423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":582,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-08-02T23:44:28.449743+0000","last_change":"2024-08-02T23:40:06.542935+0000","last_active":"2024-08-02T23:44:28.449743+0000","last_peered":"2024-08-02T23:44:28.449743+0000","last_clean":"2024-08-02T23:44:28.449743+0000","last_became_active":"2024-08-02T23:40:04.561840+0000","last_became_peered":"2024-08-02T23:40:04.561840+0000","last_unstale":"2024-08-02T23:44:28.449743+0000","last_undegraded":"2024-08-02T23:44:28.449743+0000","last_fullsized":"2024-08-02T23:44:28.449743+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-02T23:40:03.497423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-02T23:40:03.497423+0000","last_clean_scrub_stamp":"2024-08-02T23:40:03.497423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":588,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-08-02T23:44:30.466675+0000","last_change":"2024-08-02T23:40:06.540945+0000","last_active":"2024-08-02T23:44:30.466675+0000","last_peered":"2024-08-02T23:44:30.466675+0000","last_clean":"2024-08-02T23:44:30.466675+0000","last_became_active":"2024-08-02T23:40:04.560865+0000","last_became_peered":"2024-08-02T23:40:04.560865+0000","last_unstale":"2024-08-02T23:44:30.466675+0000","last_undegraded":"2024-08-02T23:44:30.466675+0000","last_fullsized":"2024-08-02T23:44:30.466675+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-02T23:40:03.497423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-02T23:40:03.497423+0000","last_clean_scrub_stamp":"2024-08-02T23:40:03.497423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":582,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-08-02T23:44:28.444871+0000","last_change":"2024-08-02T23:40:06.540508+0000","last_active":"2024-08-02T23:44:28.444871+0000","last_peered":"2024-08-02T23:44:28.444871+0000","last_clean":"2024-08-02T23:44:28.444871+0000","last_became_active":"2024-08-02T23:40:04.553411+0000","last_became_peered":"2024-08-02T23:40:04.553411+0000","last_unstale":"2024-08-02T23:44:28.444871+0000","last_undegraded":"2024-08-02T23:44:28.444871+0000","last_fullsized":"2024-08-02T23:44:28.444871+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-02T23:40:03.497423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-02T23:40:03.497423+0000","last_clean_scrub_stamp":"2024-08-02T23:40:03.497423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"16'2","reported_seq":584,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-08-02T23:44:28.446631+0000","last_change":"2024-08-02T23:40:06.538737+0000","last_active":"2024-08-02T23:44:28.446631+0000","last_peered":"2024-08-02T23:44:28.446631+0000","last_clean":"2024-08-02T23:44:28.446631+0000","last_became_active":"2024-08-02T23:40:04.553959+0000","last_became_peered":"2024-08-02T23:40:04.553959+0000","last_unstale":"2024-08-02T23:44:28.446631+0000","last_undegraded":"2024-08-02T23:44:28.446631+0000","last_fullsized":"2024-08-02T23:44:28.446631+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-02T23:40:03.497423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-02T23:40:03.497423+0000","last_clean_scrub_stamp":"2024-08-02T23:40:03.497423+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":582,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-08-02T23:44:28.442643+0000","last_change":"2024-08-02T23:40:06.540125+0000","last_active":"2024-08-02T23:44:28.442643+0000","last_peered":"2024-08-02T23:44:28.442643+0000","last_clean":"2024-08-02T23:44:28.442643+0000","last_became_active":"2024-08-02T23:40:04.556895+0000","last_became_peered":"2024-08-02T23:40:04.556895+0000","last_unstale":"2024-08-02T23:44:28.442643+0000","last_undegraded":"2024-08-02T23:44:28.442643+0000","last_fullsized":"2024-08-02T23:44:28.442643+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-02T23:40:03.497423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-02T23:40:03.497423+0000","last_clean_scrub_stamp":"2024-08-02T23:40:03.497423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":582,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-08-02T23:44:28.444605+0000","last_change":"2024-08-02T23:40:06.538748+0000","last_active":"2024-08-02T23:44:28.444605+0000","last_peered":"2024-08-02T23:44:28.444605+0000","last_clean":"2024-08-02T23:44:28.444605+0000","last_became_active":"2024-08-02T23:40:04.561398+0000","last_became_peered":"2024-08-02T23:40:04.561398+0000","last_unstale":"2024-08-02T23:44:28.444605+0000","last_undegraded":"2024-08-02T23:44:28.444605+0000","last_fullsized":"2024-08-02T23:44:28.444605+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-02T23:40:03.497423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-02T23:40:03.497423+0000","last_clean_scrub_stamp":"2024-08-02T23:40:03.497423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"14'1","reported_seq":583,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-08-02T23:44:28.447433+0000","last_change":"2024-08-02T23:40:06.541978+0000","last_active":"2024-08-02T23:44:28.447433+0000","last_peered":"2024-08-02T23:44:28.447433+0000","last_clean":"2024-08-02T23:44:28.447433+0000","last_became_active":"2024-08-02T23:40:04.550062+0000","last_became_peered":"2024-08-02T23:40:04.550062+0000","last_unstale":"2024-08-02T23:44:28.447433+0000","last_undegraded":"2024-08-02T23:44:28.447433+0000","last_fullsized":"2024-08-02T23:44:28.447433+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-02T23:40:03.497423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-02T23:40:03.497423+0000","last_clean_scrub_stamp":"2024-08-02T23:40:03.497423+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":723,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-08-02T23:44:30.451164+0000","last_change":"2024-08-02T23:40:00.557074+0000","last_active":"2024-08-02T23:44:30.451164+0000","last_peered":"2024-08-02T23:44:30.451164+0000","last_clean":"2024-08-02T23:44:30.451164+0000","last_became_active":"2024-08-02T23:40:00.546868+0000","last_became_peered":"2024-08-02T23:40:00.546868+0000","last_unstale":"2024-08-02T23:44:30.451164+0000","last_undegraded":"2024-08-02T23:44:30.451164+0000","last_fullsized":"2024-08-02T23:44:30.451164+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-02T23:39:59.483999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-02T23:39:59.483999+0000","last_clean_scrub_stamp":"2024-08-02T23:39:59.483999+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":214211493888,"available":214188527616,"internally_reserved":0,"allocated":22966272,"data_stored":22966272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53552873472,"available":53546135552,"internally_reserved":0,"allocated":6737920,"data_stored":6737920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":3,"up_from":10,"seq":42949673230,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":6760,"kb_used_data":6760,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52290968,"statfs":{"total":53552873472,"available":53545951232,"internally_reserved":0,"allocated":6922240,"data_stored":6922240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705934,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":4932,"kb_used_data":4932,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52292796,"statfs":{"total":53552873472,"available":53547823104,"internally_reserved":0,"allocated":5050368,"data_stored":5050368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705934,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":4608,"kb_used_data":4608,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52293120,"statfs":{"total":53552873472,"available":53548154880,"internally_reserved":0,"allocated":4718592,"data_stored":4718592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705934,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":6308,"kb_used_data":6308,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52291420,"statfs":{"total":53552873472,"available":53546414080,"internally_reserved":0,"allocated":6459392,"data_stored":6459392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":3,"total":53552873472,"available":53546135552,"internally_reserved":0,"allocated":6737920,"data_stored":6737920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53552873472,"available":53546414080,"internally_reserved":0,"allocated":6459392,"data_stored":6459392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53552873472,"available":53548154880,"internally_reserved":0,"allocated":4718592,"data_stored":4718592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53552873472,"available":53547823104,"internally_reserved":0,"allocated":5050368,"data_stored":5050368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":53552873472,"available":53546135552,"internally_reserved":0,"allocated":6737920,"data_stored":6737920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-02T23:44:55.552 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-08-02T23:44:55.657 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:55.652+0000 7fd042399640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-02T23:44:55.658 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:55.656+0000 7fd042399640 1 Processor -- start 2024-08-02T23:44:55.658 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:55.657+0000 7fd042399640 1 -- start start 2024-08-02T23:44:55.660 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:55.657+0000 7fd042399640 1 --2- >> v2:172.21.15.73:3300/0 conn(0x7fd03c0a4770 0x7fd03c0a4b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-02T23:44:55.660 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:55.657+0000 7fd042399640 1 -- --> v1:172.21.15.73:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd03c06e030 con 0x7fd03c0a50c0 2024-08-02T23:44:55.660 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:55.657+0000 7fd042399640 1 -- --> v2:172.21.15.73:3300/0 -- mon_getmap magic: 0 -- 0x7fd03c0a0ed0 con 0x7fd03c0a4770 2024-08-02T23:44:55.660 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:55.657+0000 7fd03bfff640 1 --2- >> v2:172.21.15.73:3300/0 conn(0x7fd03c0a4770 0x7fd03c0a4b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-02T23:44:55.660 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:55.657+0000 7fd03b7fe640 1 --1- >> v1:172.21.15.73:6789/0 conn(0x7fd03c0a50c0 0x7fd03c131000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.73:6789/0 says I am v1:172.21.15.73:43174/0 (socket says 172.21.15.73:43174) 2024-08-02T23:44:55.661 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:55.657+0000 7fd03b7fe640 1 -- 172.21.15.73:0/1717495205 learned_addr learned my addr 172.21.15.73:0/1717495205 (peer_addr_for_me v1:172.21.15.73:0/0) 2024-08-02T23:44:55.661 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:55.658+0000 7fd03bfff640 1 -- 172.21.15.73:0/1717495205 >> v1:172.21.15.73:6789/0 conn(0x7fd03c0a50c0 legacy=0x7fd03c131000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-02T23:44:55.661 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:55.658+0000 7fd03affd640 1 -- 172.21.15.73:0/1717495205 <== mon.0 v1:172.21.15.73:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 826256370 0 0) 0x7fd03c06e030 con 0x7fd03c0a50c0 2024-08-02T23:44:55.661 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:55.658+0000 7fd03bfff640 1 -- 172.21.15.73:0/1717495205 --> v2:172.21.15.73:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd03c06e260 con 0x7fd03c0a4770 2024-08-02T23:44:55.661 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:55.658+0000 7fd03bfff640 1 --2- 172.21.15.73:0/1717495205 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd03c0a4770 0x7fd03c0a4b80 crc :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8abea61c2a1eb1f4 server_cookie=0 in_seq=0 out_seq=0 2024-08-02T23:44:55.661 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:55.658+0000 7fd03affd640 1 -- 172.21.15.73:0/1717495205 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fd02c009ce0 con 0x7fd03c0a4770 2024-08-02T23:44:55.661 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:55.658+0000 7fd03affd640 1 -- 172.21.15.73:0/1717495205 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7fd02c009e40 con 0x7fd03c0a4770 2024-08-02T23:44:55.661 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:55.658+0000 7fd03affd640 1 -- 172.21.15.73:0/1717495205 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fd02c00e360 con 0x7fd03c0a4770 2024-08-02T23:44:55.664 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:55.659+0000 7fd042399640 1 -- 172.21.15.73:0/1717495205 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd03c0a4770 msgr2=0x7fd03c0a4b80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-02T23:44:55.664 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:55.659+0000 7fd042399640 1 --2- 172.21.15.73:0/1717495205 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd03c0a4770 0x7fd03c0a4b80 crc :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:44:55.664 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:55.659+0000 7fd042399640 1 -- 172.21.15.73:0/1717495205 shutdown_connections 2024-08-02T23:44:55.664 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:55.659+0000 7fd042399640 1 --2- 172.21.15.73:0/1717495205 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd03c0a4770 0x7fd03c0a4b80 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:44:55.664 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:55.659+0000 7fd042399640 1 -- 172.21.15.73:0/1717495205 >> 172.21.15.73:0/1717495205 conn(0x7fd03c0a3510 msgr2=0x7fd03c0a3920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-02T23:44:55.664 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:55.659+0000 7fd042399640 1 -- 172.21.15.73:0/1717495205 shutdown_connections 2024-08-02T23:44:55.664 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:55.659+0000 7fd042399640 1 -- 172.21.15.73:0/1717495205 wait complete. 2024-08-02T23:44:55.665 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:55.659+0000 7fd042399640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-02T23:44:55.665 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:55.660+0000 7fd042399640 1 Processor -- start 2024-08-02T23:44:55.665 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:55.660+0000 7fd042399640 1 -- start start 2024-08-02T23:44:55.665 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:55.661+0000 7fd042399640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd03c0a4770 0x7fd03c120b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-02T23:44:55.665 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:55.661+0000 7fd042399640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 -- 0x7fd03c121f90 con 0x7fd03c0a4770 2024-08-02T23:44:55.665 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:55.661+0000 7fd03bfff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd03c0a4770 0x7fd03c120b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-02T23:44:55.665 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:55.661+0000 7fd03bfff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd03c0a4770 0x7fd03c120b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:59056/0 (socket says 172.21.15.73:59056) 2024-08-02T23:44:55.665 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:55.661+0000 7fd03bfff640 1 -- 172.21.15.73:0/4170417001 learned_addr learned my addr 172.21.15.73:0/4170417001 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-08-02T23:44:55.665 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:55.661+0000 7fd03bfff640 1 -- 172.21.15.73:0/4170417001 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd03c124c30 con 0x7fd03c0a4770 2024-08-02T23:44:55.666 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:55.661+0000 7fd03bfff640 1 --2- 172.21.15.73:0/4170417001 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd03c0a4770 0x7fd03c120b60 crc :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-02T23:44:55.666 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:55.661+0000 7fd038ff9640 1 -- 172.21.15.73:0/4170417001 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fd02c014540 con 0x7fd03c0a4770 2024-08-02T23:44:55.666 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:55.661+0000 7fd038ff9640 1 -- 172.21.15.73:0/4170417001 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7fd02c0146a0 con 0x7fd03c0a4770 2024-08-02T23:44:55.666 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:55.661+0000 7fd042399640 1 -- 172.21.15.73:0/4170417001 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd03c122740 con 0x7fd03c0a4770 2024-08-02T23:44:55.666 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:55.661+0000 7fd038ff9640 1 -- 172.21.15.73:0/4170417001 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fd02c014940 con 0x7fd03c0a4770 2024-08-02T23:44:55.666 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:55.661+0000 7fd042399640 1 -- 172.21.15.73:0/4170417001 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd03c121c80 con 0x7fd03c0a4770 2024-08-02T23:44:55.667 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:55.662+0000 7fd038ff9640 1 -- 172.21.15.73:0/4170417001 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 5) ==== 45563+0+0 (crc 0 0 0) 0x7fd02c014af0 con 0x7fd03c0a4770 2024-08-02T23:44:55.667 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:55.663+0000 7fd042399640 1 -- 172.21.15.73:0/4170417001 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd03c124460 con 0x7fd03c0a4770 2024-08-02T23:44:55.667 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:55.663+0000 7fd038ff9640 1 --2- 172.21.15.73:0/4170417001 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7fd028038390 0x7fd02803a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-02T23:44:55.668 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:55.663+0000 7fd038ff9640 1 -- 172.21.15.73:0/4170417001 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(230..230 src has 1..230) ==== 2700+0+0 (crc 0 0 0) 0x7fd02c019040 con 0x7fd03c0a4770 2024-08-02T23:44:55.668 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:55.663+0000 7fd03b7fe640 1 --2- 172.21.15.73:0/4170417001 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7fd028038390 0x7fd02803a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-02T23:44:55.668 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:55.663+0000 7fd03b7fe640 1 --2- 172.21.15.73:0/4170417001 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7fd028038390 0x7fd02803a8a0 crc :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-02T23:44:55.669 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:55.668+0000 7fd038ff9640 1 -- 172.21.15.73:0/4170417001 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186747 (crc 0 0 0) 0x7fd02c018190 con 0x7fd03c0a4770 2024-08-02T23:44:55.843 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:55.841+0000 7fd042399640 1 -- 172.21.15.73:0/4170417001 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd03c123680 con 0x7fd03c0a4770 2024-08-02T23:44:55.844 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:55.842+0000 7fd038ff9640 1 -- 172.21.15.73:0/4170417001 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v230) ==== 74+0+8799 (crc 0 0 0) 0x7fd02c010310 con 0x7fd03c0a4770 2024-08-02T23:44:55.844 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-08-02T23:44:55.844 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":230,"fsid":"7caeb70d-fd31-4bdf-94e1-d0f19ea8e3c6","created":"2024-08-02T23:39:52.389003+0000","modified":"2024-08-02T23:44:53.491275+0000","last_up_change":"2024-08-02T23:39:59.483999+0000","last_in_change":"2024-08-02T23:39:54.500028+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":72,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-02T23:39:59.404407+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-08-02T23:40:02.628150+0000","flags":270369,"flags_names":"hashpspool,nopgchange,selfmanaged_snaps,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":16,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bd37ae5d-55cc-4a6d-b943-8de7e40fd1bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":228,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6801","nonce":4128208498}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6800","nonce":4128208498}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6808","nonce":4128208498}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6805","nonce":4128208498}]},"public_addr":"172.21.15.73:6801/4128208498","cluster_addr":"172.21.15.73:6800/4128208498","heartbeat_back_addr":"172.21.15.73:6808/4128208498","heartbeat_front_addr":"172.21.15.73:6805/4128208498","state":["exists","up"]},{"osd":1,"uuid":"6bf29a4c-3240-4fce-8803-66926d0df50c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":228,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6804","nonce":3612805766}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6803","nonce":3612805766}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6807","nonce":3612805766}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6806","nonce":3612805766}]},"public_addr":"172.21.15.73:6804/3612805766","cluster_addr":"172.21.15.73:6803/3612805766","heartbeat_back_addr":"172.21.15.73:6807/3612805766","heartbeat_front_addr":"172.21.15.73:6806/3612805766","state":["exists","up"]},{"osd":2,"uuid":"7d855822-09de-49f2-8811-c47826c38d5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":228,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6802","nonce":3558187202}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6803","nonce":3558187202}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6811","nonce":3558187202}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6810","nonce":3558187202}]},"public_addr":"172.21.15.131:6802/3558187202","cluster_addr":"172.21.15.131:6803/3558187202","heartbeat_back_addr":"172.21.15.131:6811/3558187202","heartbeat_front_addr":"172.21.15.131:6810/3558187202","state":["exists","up"]},{"osd":3,"uuid":"d6e6bd88-9a90-441c-8bbd-8b8d4a2b4f9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":228,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6809","nonce":1209903557}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6808","nonce":1209903557}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6813","nonce":1209903557}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6812","nonce":1209903557}]},"public_addr":"172.21.15.131:6809/1209903557","cluster_addr":"172.21.15.131:6808/1209903557","heartbeat_back_addr":"172.21.15.131:6813/1209903557","heartbeat_front_addr":"172.21.15.131:6812/1209903557","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-02T23:44:55.846 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:55.844+0000 7fd042399640 1 -- 172.21.15.73:0/4170417001 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7fd028038390 msgr2=0x7fd02803a8a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-02T23:44:55.846 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:55.844+0000 7fd042399640 1 --2- 172.21.15.73:0/4170417001 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7fd028038390 0x7fd02803a8a0 crc :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:44:55.847 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:55.844+0000 7fd042399640 1 -- 172.21.15.73:0/4170417001 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd03c0a4770 msgr2=0x7fd03c120b60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-02T23:44:55.847 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:55.844+0000 7fd042399640 1 --2- 172.21.15.73:0/4170417001 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd03c0a4770 0x7fd03c120b60 crc :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:44:55.847 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:55.844+0000 7fd042399640 1 -- 172.21.15.73:0/4170417001 shutdown_connections 2024-08-02T23:44:55.847 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:55.844+0000 7fd042399640 1 --2- 172.21.15.73:0/4170417001 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7fd028038390 0x7fd02803a8a0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:44:55.847 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:55.844+0000 7fd042399640 1 --2- 172.21.15.73:0/4170417001 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd03c0a4770 0x7fd03c120b60 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:44:55.847 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:55.844+0000 7fd042399640 1 -- 172.21.15.73:0/4170417001 >> 172.21.15.73:0/4170417001 conn(0x7fd03c0a3510 msgr2=0x7fd03c0a12e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-02T23:44:55.847 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:55.844+0000 7fd042399640 1 -- 172.21.15.73:0/4170417001 shutdown_connections 2024-08-02T23:44:55.847 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:55.844+0000 7fd042399640 1 -- 172.21.15.73:0/4170417001 wait complete. 2024-08-02T23:44:56.858 INFO:tasks.ceph:Scrubbing osd.0 2024-08-02T23:44:56.859 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-08-02T23:44:56.980 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:56.970+0000 7f609462a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-02T23:44:56.980 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:56.978+0000 7f609462a640 1 Processor -- start 2024-08-02T23:44:56.980 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:56.979+0000 7f609462a640 1 -- start start 2024-08-02T23:44:56.982 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:56.979+0000 7f609462a640 1 --2- >> v2:172.21.15.73:3300/0 conn(0x7f608c0a5290 0x7f608c139530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-02T23:44:56.982 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:56.979+0000 7f609462a640 1 -- --> v1:172.21.15.73:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f608c0a6550 con 0x7f608c0a4770 2024-08-02T23:44:56.982 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:56.980+0000 7f609462a640 1 -- --> v2:172.21.15.73:3300/0 -- mon_getmap magic: 0 -- 0x7f608c09bef0 con 0x7f608c0a5290 2024-08-02T23:44:56.983 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:56.980+0000 7f6091bc5640 1 --2- >> v2:172.21.15.73:3300/0 conn(0x7f608c0a5290 0x7f608c139530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-02T23:44:56.983 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:56.980+0000 7f60923c6640 1 --1- >> v1:172.21.15.73:6789/0 conn(0x7f608c0a4770 0x7f608c0a4b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.73:6789/0 says I am v1:172.21.15.73:53402/0 (socket says 172.21.15.73:53402) 2024-08-02T23:44:56.983 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:56.980+0000 7f60923c6640 1 -- 172.21.15.73:0/603542400 learned_addr learned my addr 172.21.15.73:0/603542400 (peer_addr_for_me v1:172.21.15.73:0/0) 2024-08-02T23:44:56.983 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:56.980+0000 7f6091bc5640 1 -- 172.21.15.73:0/603542400 >> v1:172.21.15.73:6789/0 conn(0x7f608c0a4770 legacy=0x7f608c0a4b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-02T23:44:56.983 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:56.980+0000 7f6091bc5640 1 -- 172.21.15.73:0/603542400 --> v2:172.21.15.73:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f608c09c2e0 con 0x7f608c0a5290 2024-08-02T23:44:56.983 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:56.980+0000 7f6091bc5640 1 --2- 172.21.15.73:0/603542400 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f608c0a5290 0x7f608c139530 crc :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=823cb06933d30a7c server_cookie=0 in_seq=0 out_seq=0 2024-08-02T23:44:56.983 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:56.980+0000 7f60913c4640 1 -- 172.21.15.73:0/603542400 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f6088009ce0 con 0x7f608c0a5290 2024-08-02T23:44:56.983 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:56.980+0000 7f60913c4640 1 -- 172.21.15.73:0/603542400 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f6088009e40 con 0x7f608c0a5290 2024-08-02T23:44:56.984 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:56.980+0000 7f60913c4640 1 -- 172.21.15.73:0/603542400 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f608800e360 con 0x7f608c0a5290 2024-08-02T23:44:56.986 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:56.981+0000 7f609462a640 1 -- 172.21.15.73:0/603542400 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f608c0a5290 msgr2=0x7f608c139530 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-02T23:44:56.986 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:56.981+0000 7f609462a640 1 --2- 172.21.15.73:0/603542400 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f608c0a5290 0x7f608c139530 crc :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:44:56.986 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:56.981+0000 7f609462a640 1 -- 172.21.15.73:0/603542400 shutdown_connections 2024-08-02T23:44:56.986 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:56.981+0000 7f609462a640 1 --2- 172.21.15.73:0/603542400 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f608c0a5290 0x7f608c139530 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:44:56.986 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:56.981+0000 7f609462a640 1 -- 172.21.15.73:0/603542400 >> 172.21.15.73:0/603542400 conn(0x7f608c0a3510 msgr2=0x7f608c0a3920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-02T23:44:56.986 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:56.981+0000 7f609462a640 1 -- 172.21.15.73:0/603542400 shutdown_connections 2024-08-02T23:44:56.986 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:56.981+0000 7f609462a640 1 -- 172.21.15.73:0/603542400 wait complete. 2024-08-02T23:44:56.987 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:56.981+0000 7f609462a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-02T23:44:56.987 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:56.982+0000 7f609462a640 1 Processor -- start 2024-08-02T23:44:56.987 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:56.982+0000 7f609462a640 1 -- start start 2024-08-02T23:44:56.987 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:56.983+0000 7f609462a640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f608c0a4770 0x7f608c11dc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-02T23:44:56.987 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:56.983+0000 7f609462a640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 -- 0x7f608c131dc0 con 0x7f608c0a4770 2024-08-02T23:44:56.987 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:56.983+0000 7f60923c6640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f608c0a4770 0x7f608c11dc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-02T23:44:56.987 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:56.983+0000 7f60923c6640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f608c0a4770 0x7f608c11dc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:56078/0 (socket says 172.21.15.73:56078) 2024-08-02T23:44:56.987 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:56.983+0000 7f60923c6640 1 -- 172.21.15.73:0/3886764427 learned_addr learned my addr 172.21.15.73:0/3886764427 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-08-02T23:44:56.988 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:56.983+0000 7f60923c6640 1 -- 172.21.15.73:0/3886764427 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f608c132a30 con 0x7f608c0a4770 2024-08-02T23:44:56.988 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:56.983+0000 7f60923c6640 1 --2- 172.21.15.73:0/3886764427 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f608c0a4770 0x7f608c11dc30 crc :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-02T23:44:56.988 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:56.983+0000 7f607effd640 1 -- 172.21.15.73:0/3886764427 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f60800059d0 con 0x7f608c0a4770 2024-08-02T23:44:56.988 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:56.984+0000 7f607effd640 1 -- 172.21.15.73:0/3886764427 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f6080005b30 con 0x7f608c0a4770 2024-08-02T23:44:56.988 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:56.984+0000 7f609462a640 1 -- 172.21.15.73:0/3886764427 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f608c1200f0 con 0x7f608c0a4770 2024-08-02T23:44:56.988 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:56.984+0000 7f607effd640 1 -- 172.21.15.73:0/3886764427 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f6080005dd0 con 0x7f608c0a4770 2024-08-02T23:44:56.988 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:56.984+0000 7f609462a640 1 -- 172.21.15.73:0/3886764427 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f608c131ab0 con 0x7f608c0a4770 2024-08-02T23:44:56.990 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:56.984+0000 7f609462a640 1 -- 172.21.15.73:0/3886764427 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f608c121e30 con 0x7f608c0a4770 2024-08-02T23:44:56.990 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:56.984+0000 7f607effd640 1 -- 172.21.15.73:0/3886764427 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 5) ==== 45563+0+0 (crc 0 0 0) 0x7f60800070b0 con 0x7f608c0a4770 2024-08-02T23:44:56.990 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:56.985+0000 7f607effd640 1 --2- 172.21.15.73:0/3886764427 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7f6070038390 0x7f607003a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-02T23:44:56.990 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:56.985+0000 7f607effd640 1 -- 172.21.15.73:0/3886764427 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(230..230 src has 1..230) ==== 2700+0+0 (crc 0 0 0) 0x7f6080047a80 con 0x7f608c0a4770 2024-08-02T23:44:56.990 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:56.985+0000 7f607effd640 1 --2- 172.21.15.73:0/3886764427 >> v2:172.21.15.73:6801/4128208498 conn(0x7f607003d390 0x7f607003f840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-02T23:44:56.990 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:56.985+0000 7f607effd640 1 -- 172.21.15.73:0/3886764427 --> v2:172.21.15.73:6801/4128208498 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f607003fed0 con 0x7f607003d390 2024-08-02T23:44:56.990 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:56.985+0000 7f607effd640 1 -- 172.21.15.73:0/3886764427 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_get_version_reply(handle=1 version=230) ==== 24+0+0 (crc 0 0 0) 0x7f6080047e90 con 0x7f608c0a4770 2024-08-02T23:44:56.991 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:56.985+0000 7f6091bc5640 1 --2- 172.21.15.73:0/3886764427 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7f6070038390 0x7f607003a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-02T23:44:56.991 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:56.985+0000 7f6092bc7640 1 --2- 172.21.15.73:0/3886764427 >> v2:172.21.15.73:6801/4128208498 conn(0x7f607003d390 0x7f607003f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-08-02T23:44:56.991 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:56.986+0000 7f6091bc5640 1 --2- 172.21.15.73:0/3886764427 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7f6070038390 0x7f607003a8a0 crc :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-02T23:44:56.991 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:56.987+0000 7f6092bc7640 1 --2- 172.21.15.73:0/3886764427 >> v2:172.21.15.73:6801/4128208498 conn(0x7f607003d390 0x7f607003f840 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-02T23:44:56.991 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:56.988+0000 7f607effd640 1 -- 172.21.15.73:0/3886764427 <== osd.0 v2:172.21.15.73:6801/4128208498 1 ==== command_reply(tid 1: 0 ) ==== 8+0+5172 (crc 0 0 0) 0x7f607003fed0 con 0x7f607003d390 2024-08-02T23:44:56.994 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:56.992+0000 7f609462a640 1 -- 172.21.15.73:0/3886764427 --> v2:172.21.15.73:6801/4128208498 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f608c09b770 con 0x7f607003d390 2024-08-02T23:44:56.994 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:56.992+0000 7f607effd640 1 -- 172.21.15.73:0/3886764427 <== osd.0 v2:172.21.15.73:6801/4128208498 2 ==== command_reply(tid 2: 0 ) ==== 8+0+22 (crc 0 0 0) 0x7f608400a820 con 0x7f607003d390 2024-08-02T23:44:56.995 INFO:teuthology.orchestra.run.smithi073.stdout:{ 2024-08-02T23:44:56.995 INFO:teuthology.orchestra.run.smithi073.stdout: "success": "" 2024-08-02T23:44:56.995 INFO:teuthology.orchestra.run.smithi073.stdout:} 2024-08-02T23:44:56.996 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:56.993+0000 7f609462a640 1 -- 172.21.15.73:0/3886764427 >> v2:172.21.15.73:6801/4128208498 conn(0x7f607003d390 msgr2=0x7f607003f840 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-02T23:44:56.996 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:56.993+0000 7f609462a640 1 --2- 172.21.15.73:0/3886764427 >> v2:172.21.15.73:6801/4128208498 conn(0x7f607003d390 0x7f607003f840 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:44:56.996 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:56.993+0000 7f609462a640 1 -- 172.21.15.73:0/3886764427 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7f6070038390 msgr2=0x7f607003a8a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-02T23:44:56.996 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:56.993+0000 7f609462a640 1 --2- 172.21.15.73:0/3886764427 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7f6070038390 0x7f607003a8a0 crc :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:44:56.997 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:56.993+0000 7f609462a640 1 -- 172.21.15.73:0/3886764427 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f608c0a4770 msgr2=0x7f608c11dc30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-02T23:44:56.997 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:56.993+0000 7f609462a640 1 --2- 172.21.15.73:0/3886764427 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f608c0a4770 0x7f608c11dc30 crc :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:44:56.997 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:56.993+0000 7f609462a640 1 -- 172.21.15.73:0/3886764427 shutdown_connections 2024-08-02T23:44:56.997 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:56.993+0000 7f609462a640 1 --2- 172.21.15.73:0/3886764427 >> v2:172.21.15.73:6801/4128208498 conn(0x7f607003d390 0x7f607003f840 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:44:56.997 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:56.993+0000 7f609462a640 1 --2- 172.21.15.73:0/3886764427 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7f6070038390 0x7f607003a8a0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:44:56.997 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:56.993+0000 7f609462a640 1 --2- 172.21.15.73:0/3886764427 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f608c0a4770 0x7f608c11dc30 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:44:56.997 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:56.993+0000 7f609462a640 1 -- 172.21.15.73:0/3886764427 >> 172.21.15.73:0/3886764427 conn(0x7f608c0a3510 msgr2=0x7f608c0a2170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-02T23:44:56.997 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:56.993+0000 7f609462a640 1 -- 172.21.15.73:0/3886764427 shutdown_connections 2024-08-02T23:44:56.998 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:56.993+0000 7f609462a640 1 -- 172.21.15.73:0/3886764427 wait complete. 2024-08-02T23:44:57.006 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-08-02T23:44:57.111 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.105+0000 7fdcdfd19640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-02T23:44:57.112 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.110+0000 7fdcdfd19640 1 Processor -- start 2024-08-02T23:44:57.112 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.110+0000 7fdcdfd19640 1 -- start start 2024-08-02T23:44:57.113 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.111+0000 7fdcdfd19640 1 --2- >> v2:172.21.15.73:3300/0 conn(0x7fdcd812d690 0x7fdcd812daa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-02T23:44:57.113 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.111+0000 7fdcdfd19640 1 -- --> v1:172.21.15.73:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdcd806e4f0 con 0x7fdcd81306a0 2024-08-02T23:44:57.113 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.111+0000 7fdcdfd19640 1 -- --> v2:172.21.15.73:3300/0 -- mon_getmap magic: 0 -- 0x7fdcd812d240 con 0x7fdcd812d690 2024-08-02T23:44:57.113 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.111+0000 7fdcddab5640 1 --2- >> v2:172.21.15.73:3300/0 conn(0x7fdcd812d690 0x7fdcd812daa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-02T23:44:57.114 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.111+0000 7fdcdd2b4640 1 --1- >> v1:172.21.15.73:6789/0 conn(0x7fdcd81306a0 0x7fdcd8130ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.73:6789/0 says I am v1:172.21.15.73:53418/0 (socket says 172.21.15.73:53418) 2024-08-02T23:44:57.114 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.111+0000 7fdcdd2b4640 1 -- 172.21.15.73:0/2918835228 learned_addr learned my addr 172.21.15.73:0/2918835228 (peer_addr_for_me v1:172.21.15.73:0/0) 2024-08-02T23:44:57.114 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.111+0000 7fdcddab5640 1 -- 172.21.15.73:0/2918835228 >> v1:172.21.15.73:6789/0 conn(0x7fdcd81306a0 legacy=0x7fdcd8130ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-02T23:44:57.114 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.111+0000 7fdcddab5640 1 -- 172.21.15.73:0/2918835228 --> v2:172.21.15.73:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdcd809e870 con 0x7fdcd812d690 2024-08-02T23:44:57.114 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.111+0000 7fdcddab5640 1 --2- 172.21.15.73:0/2918835228 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fdcd812d690 0x7fdcd812daa0 crc :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a28427329a5b2f7 server_cookie=0 in_seq=0 out_seq=0 2024-08-02T23:44:57.114 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.111+0000 7fdcdcab3640 1 -- 172.21.15.73:0/2918835228 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fdcc8009ce0 con 0x7fdcd812d690 2024-08-02T23:44:57.114 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.111+0000 7fdcdcab3640 1 -- 172.21.15.73:0/2918835228 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7fdcc8009e40 con 0x7fdcd812d690 2024-08-02T23:44:57.115 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.111+0000 7fdcdcab3640 1 -- 172.21.15.73:0/2918835228 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fdcc800e360 con 0x7fdcd812d690 2024-08-02T23:44:57.117 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.112+0000 7fdcdfd19640 1 -- 172.21.15.73:0/2918835228 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fdcd812d690 msgr2=0x7fdcd812daa0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-02T23:44:57.117 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.112+0000 7fdcdfd19640 1 --2- 172.21.15.73:0/2918835228 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fdcd812d690 0x7fdcd812daa0 crc :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:44:57.117 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.112+0000 7fdcdfd19640 1 -- 172.21.15.73:0/2918835228 shutdown_connections 2024-08-02T23:44:57.117 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.112+0000 7fdcdfd19640 1 --2- 172.21.15.73:0/2918835228 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fdcd812d690 0x7fdcd812daa0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:44:57.117 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.112+0000 7fdcdfd19640 1 -- 172.21.15.73:0/2918835228 >> 172.21.15.73:0/2918835228 conn(0x7fdcd80a0590 msgr2=0x7fdcd80a09a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-02T23:44:57.117 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.112+0000 7fdcdfd19640 1 -- 172.21.15.73:0/2918835228 shutdown_connections 2024-08-02T23:44:57.118 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.112+0000 7fdcdfd19640 1 -- 172.21.15.73:0/2918835228 wait complete. 2024-08-02T23:44:57.118 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.112+0000 7fdcdfd19640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-02T23:44:57.118 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.113+0000 7fdcdfd19640 1 Processor -- start 2024-08-02T23:44:57.118 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.113+0000 7fdcdfd19640 1 -- start start 2024-08-02T23:44:57.118 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.114+0000 7fdcdfd19640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fdcd812d690 0x7fdcd81c5bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-02T23:44:57.118 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.114+0000 7fdcdfd19640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 -- 0x7fdcd8120a50 con 0x7fdcd812d690 2024-08-02T23:44:57.118 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.114+0000 7fdcddab5640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fdcd812d690 0x7fdcd81c5bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-02T23:44:57.118 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.114+0000 7fdcddab5640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fdcd812d690 0x7fdcd81c5bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:56096/0 (socket says 172.21.15.73:56096) 2024-08-02T23:44:57.119 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.114+0000 7fdcddab5640 1 -- 172.21.15.73:0/2906664951 learned_addr learned my addr 172.21.15.73:0/2906664951 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-08-02T23:44:57.119 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.114+0000 7fdcddab5640 1 -- 172.21.15.73:0/2906664951 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdcd811cd30 con 0x7fdcd812d690 2024-08-02T23:44:57.119 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.114+0000 7fdcddab5640 1 --2- 172.21.15.73:0/2906664951 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fdcd812d690 0x7fdcd81c5bd0 crc :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-02T23:44:57.119 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.115+0000 7fdcce7fc640 1 -- 172.21.15.73:0/2906664951 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fdcc8014540 con 0x7fdcd812d690 2024-08-02T23:44:57.119 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.115+0000 7fdcce7fc640 1 -- 172.21.15.73:0/2906664951 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7fdcc80146a0 con 0x7fdcd812d690 2024-08-02T23:44:57.119 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.115+0000 7fdcdfd19640 1 -- 172.21.15.73:0/2906664951 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdcd811f4a0 con 0x7fdcd812d690 2024-08-02T23:44:57.119 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.115+0000 7fdcce7fc640 1 -- 172.21.15.73:0/2906664951 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fdcc8014940 con 0x7fdcd812d690 2024-08-02T23:44:57.119 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.115+0000 7fdcdfd19640 1 -- 172.21.15.73:0/2906664951 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdcd811ca60 con 0x7fdcd812d690 2024-08-02T23:44:57.120 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.115+0000 7fdcdfd19640 1 -- 172.21.15.73:0/2906664951 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdcd809ed20 con 0x7fdcd812d690 2024-08-02T23:44:57.121 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.116+0000 7fdcce7fc640 1 -- 172.21.15.73:0/2906664951 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 5) ==== 45563+0+0 (crc 0 0 0) 0x7fdcc8014b30 con 0x7fdcd812d690 2024-08-02T23:44:57.121 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.116+0000 7fdcce7fc640 1 --2- 172.21.15.73:0/2906664951 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7fdcbc038390 0x7fdcbc03a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-02T23:44:57.121 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.116+0000 7fdcce7fc640 1 -- 172.21.15.73:0/2906664951 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(230..230 src has 1..230) ==== 2700+0+0 (crc 0 0 0) 0x7fdcc8019040 con 0x7fdcd812d690 2024-08-02T23:44:57.121 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.116+0000 7fdcdd2b4640 1 --2- 172.21.15.73:0/2906664951 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7fdcbc038390 0x7fdcbc03a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-02T23:44:57.121 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.117+0000 7fdcdd2b4640 1 --2- 172.21.15.73:0/2906664951 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7fdcbc038390 0x7fdcbc03a8a0 crc :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-02T23:44:57.122 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.120+0000 7fdcce7fc640 1 -- 172.21.15.73:0/2906664951 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186747 (crc 0 0 0) 0x7fdcc8018190 con 0x7fdcd812d690 2024-08-02T23:44:57.296 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.294+0000 7fdcdfd19640 1 -- 172.21.15.73:0/2906664951 --> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) -- 0x7fdcd806eaa0 con 0x7fdcbc038390 2024-08-02T23:44:57.297 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.295+0000 7fdcce7fc640 1 -- 172.21.15.73:0/2906664951 <== mgr.4100 v2:172.21.15.131:6800/3351532440 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) ==== 41+0+0 (crc 0 0 0) 0x7fdcd40011a0 con 0x7fdcbc038390 2024-08-02T23:44:57.297 INFO:teuthology.orchestra.run.smithi073.stderr:instructed osd(s) 0 to deep-scrub 2024-08-02T23:44:57.300 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.298+0000 7fdcdfd19640 1 -- 172.21.15.73:0/2906664951 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7fdcbc038390 msgr2=0x7fdcbc03a8a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-02T23:44:57.300 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.298+0000 7fdcdfd19640 1 --2- 172.21.15.73:0/2906664951 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7fdcbc038390 0x7fdcbc03a8a0 crc :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:44:57.300 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.298+0000 7fdcdfd19640 1 -- 172.21.15.73:0/2906664951 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fdcd812d690 msgr2=0x7fdcd81c5bd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-02T23:44:57.300 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.298+0000 7fdcdfd19640 1 --2- 172.21.15.73:0/2906664951 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fdcd812d690 0x7fdcd81c5bd0 crc :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:44:57.301 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.298+0000 7fdcdfd19640 1 -- 172.21.15.73:0/2906664951 shutdown_connections 2024-08-02T23:44:57.301 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.298+0000 7fdcdfd19640 1 --2- 172.21.15.73:0/2906664951 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7fdcbc038390 0x7fdcbc03a8a0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:44:57.301 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.298+0000 7fdcdfd19640 1 --2- 172.21.15.73:0/2906664951 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fdcd812d690 0x7fdcd81c5bd0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:44:57.301 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.298+0000 7fdcdfd19640 1 -- 172.21.15.73:0/2906664951 >> 172.21.15.73:0/2906664951 conn(0x7fdcd80a0590 msgr2=0x7fdcd80a09a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-02T23:44:57.301 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.298+0000 7fdcdfd19640 1 -- 172.21.15.73:0/2906664951 shutdown_connections 2024-08-02T23:44:57.301 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.298+0000 7fdcdfd19640 1 -- 172.21.15.73:0/2906664951 wait complete. 2024-08-02T23:44:57.312 INFO:tasks.ceph:Scrubbing osd.1 2024-08-02T23:44:57.313 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-08-02T23:44:57.420 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.411+0000 7fafd2159640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-02T23:44:57.421 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.419+0000 7fafd2159640 1 Processor -- start 2024-08-02T23:44:57.421 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.420+0000 7fafd2159640 1 -- start start 2024-08-02T23:44:57.423 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.420+0000 7fafd2159640 1 --2- >> v2:172.21.15.73:3300/0 conn(0x7fafcc099790 0x7fafcc09aa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-02T23:44:57.423 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.420+0000 7fafd2159640 1 -- --> v1:172.21.15.73:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fafcc09bae0 con 0x7fafcc099ba0 2024-08-02T23:44:57.423 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.420+0000 7fafd2159640 1 -- --> v2:172.21.15.73:3300/0 -- mon_getmap magic: 0 -- 0x7fafcc09bd10 con 0x7fafcc099790 2024-08-02T23:44:57.423 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.420+0000 7fafcb7fe640 1 --2- >> v2:172.21.15.73:3300/0 conn(0x7fafcc099790 0x7fafcc09aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-02T23:44:57.423 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.420+0000 7fafc3fff640 1 --1- >> v1:172.21.15.73:6789/0 conn(0x7fafcc099ba0 0x7fafcc1306d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.73:6789/0 says I am v1:172.21.15.73:53434/0 (socket says 172.21.15.73:53434) 2024-08-02T23:44:57.423 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.420+0000 7fafc3fff640 1 -- 172.21.15.73:0/683632028 learned_addr learned my addr 172.21.15.73:0/683632028 (peer_addr_for_me v1:172.21.15.73:0/0) 2024-08-02T23:44:57.423 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.421+0000 7fafcb7fe640 1 -- 172.21.15.73:0/683632028 >> v1:172.21.15.73:6789/0 conn(0x7fafcc099ba0 legacy=0x7fafcc1306d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-02T23:44:57.424 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.421+0000 7fafcb7fe640 1 -- 172.21.15.73:0/683632028 --> v2:172.21.15.73:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fafcc09a640 con 0x7fafcc099790 2024-08-02T23:44:57.424 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.421+0000 7fafcaffd640 1 -- 172.21.15.73:0/683632028 <== mon.0 v1:172.21.15.73:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2242860299 0 0) 0x7fafcc09bae0 con 0x7fafcc099ba0 2024-08-02T23:44:57.424 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.421+0000 7fafcb7fe640 1 --2- 172.21.15.73:0/683632028 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fafcc099790 0x7fafcc09aa90 crc :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2da85aaad9106890 server_cookie=0 in_seq=0 out_seq=0 2024-08-02T23:44:57.424 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.421+0000 7fafcaffd640 1 -- 172.21.15.73:0/683632028 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fafbc009ce0 con 0x7fafcc099790 2024-08-02T23:44:57.424 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.421+0000 7fafcaffd640 1 -- 172.21.15.73:0/683632028 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7fafbc009e40 con 0x7fafcc099790 2024-08-02T23:44:57.424 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.421+0000 7fafcaffd640 1 -- 172.21.15.73:0/683632028 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fafbc00e360 con 0x7fafcc099790 2024-08-02T23:44:57.426 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.422+0000 7fafd2159640 1 -- 172.21.15.73:0/683632028 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fafcc099790 msgr2=0x7fafcc09aa90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-02T23:44:57.426 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.422+0000 7fafd2159640 1 --2- 172.21.15.73:0/683632028 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fafcc099790 0x7fafcc09aa90 crc :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:44:57.426 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.422+0000 7fafd2159640 1 -- 172.21.15.73:0/683632028 shutdown_connections 2024-08-02T23:44:57.426 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.422+0000 7fafd2159640 1 --2- 172.21.15.73:0/683632028 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fafcc099790 0x7fafcc09aa90 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:44:57.426 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.422+0000 7fafd2159640 1 -- 172.21.15.73:0/683632028 >> 172.21.15.73:0/683632028 conn(0x7fafcc067aa0 msgr2=0x7fafcc067eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-02T23:44:57.426 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.422+0000 7fafd2159640 1 -- 172.21.15.73:0/683632028 shutdown_connections 2024-08-02T23:44:57.427 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.422+0000 7fafd2159640 1 -- 172.21.15.73:0/683632028 wait complete. 2024-08-02T23:44:57.427 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.422+0000 7fafd2159640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-02T23:44:57.427 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.423+0000 7fafd2159640 1 Processor -- start 2024-08-02T23:44:57.427 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.423+0000 7fafd2159640 1 -- start start 2024-08-02T23:44:57.427 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.424+0000 7fafd2159640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fafcc099790 0x7fafcc12b720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-02T23:44:57.427 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.424+0000 7fafd2159640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 -- 0x7fafcc08ca00 con 0x7fafcc099790 2024-08-02T23:44:57.427 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.424+0000 7fafcb7fe640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fafcc099790 0x7fafcc12b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-02T23:44:57.427 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.424+0000 7fafcb7fe640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fafcc099790 0x7fafcc12b720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:56118/0 (socket says 172.21.15.73:56118) 2024-08-02T23:44:57.428 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.424+0000 7fafcb7fe640 1 -- 172.21.15.73:0/2676091137 learned_addr learned my addr 172.21.15.73:0/2676091137 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-08-02T23:44:57.428 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.424+0000 7fafcb7fe640 1 -- 172.21.15.73:0/2676091137 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fafcc09a020 con 0x7fafcc099790 2024-08-02T23:44:57.428 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.424+0000 7fafcb7fe640 1 --2- 172.21.15.73:0/2676091137 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fafcc099790 0x7fafcc12b720 crc :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-02T23:44:57.429 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.424+0000 7fafc8ff9640 1 -- 172.21.15.73:0/2676091137 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fafbc014620 con 0x7fafcc099790 2024-08-02T23:44:57.430 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.425+0000 7fafc8ff9640 1 -- 172.21.15.73:0/2676091137 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7fafbc014780 con 0x7fafcc099790 2024-08-02T23:44:57.430 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.425+0000 7fafd2159640 1 -- 172.21.15.73:0/2676091137 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fafcc09e280 con 0x7fafcc099790 2024-08-02T23:44:57.430 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.425+0000 7fafc8ff9640 1 -- 172.21.15.73:0/2676091137 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fafbc014a20 con 0x7fafcc099790 2024-08-02T23:44:57.430 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.425+0000 7fafd2159640 1 -- 172.21.15.73:0/2676091137 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fafcc12d790 con 0x7fafcc099790 2024-08-02T23:44:57.430 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.425+0000 7fafd2159640 1 -- 172.21.15.73:0/2676091137 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fafcc09b770 con 0x7fafcc099790 2024-08-02T23:44:57.430 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.425+0000 7fafc8ff9640 1 -- 172.21.15.73:0/2676091137 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 5) ==== 45563+0+0 (crc 0 0 0) 0x7fafbc014bd0 con 0x7fafcc099790 2024-08-02T23:44:57.430 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.426+0000 7fafc8ff9640 1 --2- 172.21.15.73:0/2676091137 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7fafac038390 0x7fafac03a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-02T23:44:57.431 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.426+0000 7fafc8ff9640 1 -- 172.21.15.73:0/2676091137 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(230..230 src has 1..230) ==== 2700+0+0 (crc 0 0 0) 0x7fafbc019040 con 0x7fafcc099790 2024-08-02T23:44:57.431 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.426+0000 7fafc8ff9640 1 --2- 172.21.15.73:0/2676091137 >> v2:172.21.15.73:6804/3612805766 conn(0x7fafac03d390 0x7fafac03f840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-02T23:44:57.431 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.426+0000 7fafc8ff9640 1 -- 172.21.15.73:0/2676091137 --> v2:172.21.15.73:6804/3612805766 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fafac03fed0 con 0x7fafac03d390 2024-08-02T23:44:57.431 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.426+0000 7fafc8ff9640 1 -- 172.21.15.73:0/2676091137 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_get_version_reply(handle=1 version=230) ==== 24+0+0 (crc 0 0 0) 0x7fafbc04b4b0 con 0x7fafcc099790 2024-08-02T23:44:57.431 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.426+0000 7fafcbfff640 1 --2- 172.21.15.73:0/2676091137 >> v2:172.21.15.73:6804/3612805766 conn(0x7fafac03d390 0x7fafac03f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-08-02T23:44:57.431 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.427+0000 7fafc3fff640 1 --2- 172.21.15.73:0/2676091137 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7fafac038390 0x7fafac03a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-02T23:44:57.431 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.427+0000 7fafc3fff640 1 --2- 172.21.15.73:0/2676091137 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7fafac038390 0x7fafac03a8a0 crc :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-02T23:44:57.432 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.428+0000 7fafcbfff640 1 --2- 172.21.15.73:0/2676091137 >> v2:172.21.15.73:6804/3612805766 conn(0x7fafac03d390 0x7fafac03f840 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-02T23:44:57.432 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.429+0000 7fafc8ff9640 1 -- 172.21.15.73:0/2676091137 <== osd.1 v2:172.21.15.73:6804/3612805766 1 ==== command_reply(tid 1: 0 ) ==== 8+0+5172 (crc 0 0 0) 0x7fafac03fed0 con 0x7fafac03d390 2024-08-02T23:44:57.434 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.433+0000 7fafd2159640 1 -- 172.21.15.73:0/2676091137 --> v2:172.21.15.73:6804/3612805766 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fafcc12c9e0 con 0x7fafac03d390 2024-08-02T23:44:57.435 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.434+0000 7fafc8ff9640 1 -- 172.21.15.73:0/2676091137 <== osd.1 v2:172.21.15.73:6804/3612805766 2 ==== command_reply(tid 2: 0 ) ==== 8+0+22 (crc 0 0 0) 0x7fafcc12c9e0 con 0x7fafac03d390 2024-08-02T23:44:57.436 INFO:teuthology.orchestra.run.smithi073.stdout:{ 2024-08-02T23:44:57.436 INFO:teuthology.orchestra.run.smithi073.stdout: "success": "" 2024-08-02T23:44:57.436 INFO:teuthology.orchestra.run.smithi073.stdout:} 2024-08-02T23:44:57.437 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.434+0000 7fafd2159640 1 -- 172.21.15.73:0/2676091137 >> v2:172.21.15.73:6804/3612805766 conn(0x7fafac03d390 msgr2=0x7fafac03f840 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-02T23:44:57.437 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.434+0000 7fafd2159640 1 --2- 172.21.15.73:0/2676091137 >> v2:172.21.15.73:6804/3612805766 conn(0x7fafac03d390 0x7fafac03f840 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:44:57.437 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.434+0000 7fafd2159640 1 -- 172.21.15.73:0/2676091137 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7fafac038390 msgr2=0x7fafac03a8a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-02T23:44:57.437 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.434+0000 7fafd2159640 1 --2- 172.21.15.73:0/2676091137 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7fafac038390 0x7fafac03a8a0 crc :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:44:57.437 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.434+0000 7fafd2159640 1 -- 172.21.15.73:0/2676091137 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fafcc099790 msgr2=0x7fafcc12b720 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-02T23:44:57.437 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.434+0000 7fafd2159640 1 --2- 172.21.15.73:0/2676091137 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fafcc099790 0x7fafcc12b720 crc :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:44:57.438 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.434+0000 7fafd2159640 1 -- 172.21.15.73:0/2676091137 shutdown_connections 2024-08-02T23:44:57.438 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.434+0000 7fafd2159640 1 --2- 172.21.15.73:0/2676091137 >> v2:172.21.15.73:6804/3612805766 conn(0x7fafac03d390 0x7fafac03f840 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:44:57.438 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.434+0000 7fafd2159640 1 --2- 172.21.15.73:0/2676091137 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7fafac038390 0x7fafac03a8a0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:44:57.438 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.434+0000 7fafd2159640 1 --2- 172.21.15.73:0/2676091137 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fafcc099790 0x7fafcc12b720 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:44:57.438 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.434+0000 7fafd2159640 1 -- 172.21.15.73:0/2676091137 >> 172.21.15.73:0/2676091137 conn(0x7fafcc067aa0 msgr2=0x7fafcc09cb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-02T23:44:57.438 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.434+0000 7fafd2159640 1 -- 172.21.15.73:0/2676091137 shutdown_connections 2024-08-02T23:44:57.438 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.434+0000 7fafd2159640 1 -- 172.21.15.73:0/2676091137 wait complete. 2024-08-02T23:44:57.447 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-08-02T23:44:57.553 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.547+0000 7fd9d642a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-02T23:44:57.554 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.552+0000 7fd9d642a640 1 Processor -- start 2024-08-02T23:44:57.554 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.552+0000 7fd9d642a640 1 -- start start 2024-08-02T23:44:57.555 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.553+0000 7fd9d642a640 1 --2- >> v2:172.21.15.73:3300/0 conn(0x7fd9d0134350 0x7fd9d0136770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-02T23:44:57.556 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.553+0000 7fd9d642a640 1 -- --> v1:172.21.15.73:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd9d006fef0 con 0x7fd9d006e880 2024-08-02T23:44:57.556 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.553+0000 7fd9d642a640 1 -- --> v2:172.21.15.73:3300/0 -- mon_getmap magic: 0 -- 0x7fd9d009e5d0 con 0x7fd9d0134350 2024-08-02T23:44:57.556 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.553+0000 7fd9cffff640 1 --1- >> v1:172.21.15.73:6789/0 conn(0x7fd9d006e880 0x7fd9d006ec90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.73:6789/0 says I am v1:172.21.15.73:53448/0 (socket says 172.21.15.73:53448) 2024-08-02T23:44:57.556 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.553+0000 7fd9cffff640 1 -- 172.21.15.73:0/3027060034 learned_addr learned my addr 172.21.15.73:0/3027060034 (peer_addr_for_me v1:172.21.15.73:0/0) 2024-08-02T23:44:57.556 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.553+0000 7fd9cf7fe640 1 --2- 172.21.15.73:0/3027060034 >> v2:172.21.15.73:3300/0 conn(0x7fd9d0134350 0x7fd9d0136770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-02T23:44:57.556 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.553+0000 7fd9ceffd640 1 -- 172.21.15.73:0/3027060034 <== mon.0 v1:172.21.15.73:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 442198980 0 0) 0x7fd9d006fef0 con 0x7fd9d006e880 2024-08-02T23:44:57.556 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.553+0000 7fd9ceffd640 1 -- 172.21.15.73:0/3027060034 --> v1:172.21.15.73:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd9b80034f0 con 0x7fd9d006e880 2024-08-02T23:44:57.556 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.553+0000 7fd9cf7fe640 1 -- 172.21.15.73:0/3027060034 >> v1:172.21.15.73:6789/0 conn(0x7fd9d006e880 legacy=0x7fd9d006ec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-02T23:44:57.556 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.553+0000 7fd9cf7fe640 1 -- 172.21.15.73:0/3027060034 --> v2:172.21.15.73:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9d006e430 con 0x7fd9d0134350 2024-08-02T23:44:57.557 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.554+0000 7fd9cf7fe640 1 --2- 172.21.15.73:0/3027060034 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd9d0134350 0x7fd9d0136770 crc :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9b636ef6ccb3b6a server_cookie=0 in_seq=0 out_seq=0 2024-08-02T23:44:57.557 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.554+0000 7fd9ceffd640 1 -- 172.21.15.73:0/3027060034 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fd9c4009ce0 con 0x7fd9d0134350 2024-08-02T23:44:57.557 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.554+0000 7fd9ceffd640 1 -- 172.21.15.73:0/3027060034 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7fd9c4009e40 con 0x7fd9d0134350 2024-08-02T23:44:57.557 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.554+0000 7fd9ceffd640 1 -- 172.21.15.73:0/3027060034 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fd9c400e360 con 0x7fd9d0134350 2024-08-02T23:44:57.559 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.554+0000 7fd9d642a640 1 -- 172.21.15.73:0/3027060034 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd9d0134350 msgr2=0x7fd9d0136770 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-02T23:44:57.559 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.554+0000 7fd9d642a640 1 --2- 172.21.15.73:0/3027060034 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd9d0134350 0x7fd9d0136770 crc :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:44:57.559 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.554+0000 7fd9d642a640 1 -- 172.21.15.73:0/3027060034 shutdown_connections 2024-08-02T23:44:57.559 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.554+0000 7fd9d642a640 1 --2- 172.21.15.73:0/3027060034 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd9d0134350 0x7fd9d0136770 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:44:57.560 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.554+0000 7fd9d642a640 1 -- 172.21.15.73:0/3027060034 >> 172.21.15.73:0/3027060034 conn(0x7fd9d00a0590 msgr2=0x7fd9d00a09a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-02T23:44:57.560 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.554+0000 7fd9d642a640 1 -- 172.21.15.73:0/3027060034 shutdown_connections 2024-08-02T23:44:57.560 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.554+0000 7fd9d642a640 1 -- 172.21.15.73:0/3027060034 wait complete. 2024-08-02T23:44:57.560 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.555+0000 7fd9d642a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-02T23:44:57.560 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.555+0000 7fd9d642a640 1 Processor -- start 2024-08-02T23:44:57.560 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.556+0000 7fd9d642a640 1 -- start start 2024-08-02T23:44:57.560 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.556+0000 7fd9d642a640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd9d006e880 0x7fd9d011eea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-02T23:44:57.560 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.556+0000 7fd9d642a640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 -- 0x7fd9d0120d10 con 0x7fd9d006e880 2024-08-02T23:44:57.561 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.556+0000 7fd9cffff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd9d006e880 0x7fd9d011eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-02T23:44:57.561 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.556+0000 7fd9cffff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd9d006e880 0x7fd9d011eea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:56128/0 (socket says 172.21.15.73:56128) 2024-08-02T23:44:57.561 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.556+0000 7fd9cffff640 1 -- 172.21.15.73:0/1144241853 learned_addr learned my addr 172.21.15.73:0/1144241853 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-08-02T23:44:57.561 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.557+0000 7fd9cffff640 1 -- 172.21.15.73:0/1144241853 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9d0121330 con 0x7fd9d006e880 2024-08-02T23:44:57.561 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.557+0000 7fd9cffff640 1 --2- 172.21.15.73:0/1144241853 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd9d006e880 0x7fd9d011eea0 crc :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-02T23:44:57.561 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.557+0000 7fd9ccff9640 1 -- 172.21.15.73:0/1144241853 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fd9c00059d0 con 0x7fd9d006e880 2024-08-02T23:44:57.561 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.557+0000 7fd9ccff9640 1 -- 172.21.15.73:0/1144241853 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7fd9c0005b30 con 0x7fd9d006e880 2024-08-02T23:44:57.561 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.557+0000 7fd9d642a640 1 -- 172.21.15.73:0/1144241853 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9d0120360 con 0x7fd9d006e880 2024-08-02T23:44:57.561 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.557+0000 7fd9ccff9640 1 -- 172.21.15.73:0/1144241853 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fd9c0005dd0 con 0x7fd9d006e880 2024-08-02T23:44:57.562 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.557+0000 7fd9d642a640 1 -- 172.21.15.73:0/1144241853 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9d0122420 con 0x7fd9d006e880 2024-08-02T23:44:57.562 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.558+0000 7fd9ccff9640 1 -- 172.21.15.73:0/1144241853 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 5) ==== 45563+0+0 (crc 0 0 0) 0x7fd9c00070b0 con 0x7fd9d006e880 2024-08-02T23:44:57.563 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.558+0000 7fd9d642a640 1 -- 172.21.15.73:0/1144241853 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd9d011eca0 con 0x7fd9d006e880 2024-08-02T23:44:57.563 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.558+0000 7fd9ccff9640 1 --2- 172.21.15.73:0/1144241853 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7fd9bc038390 0x7fd9bc03a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-02T23:44:57.563 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.558+0000 7fd9ccff9640 1 -- 172.21.15.73:0/1144241853 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(230..230 src has 1..230) ==== 2700+0+0 (crc 0 0 0) 0x7fd9c0047a40 con 0x7fd9d006e880 2024-08-02T23:44:57.563 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.559+0000 7fd9cf7fe640 1 --2- 172.21.15.73:0/1144241853 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7fd9bc038390 0x7fd9bc03a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-02T23:44:57.563 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.559+0000 7fd9cf7fe640 1 --2- 172.21.15.73:0/1144241853 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7fd9bc038390 0x7fd9bc03a8a0 crc :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-02T23:44:57.565 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.563+0000 7fd9ccff9640 1 -- 172.21.15.73:0/1144241853 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186747 (crc 0 0 0) 0x7fd9c000d7f0 con 0x7fd9d006e880 2024-08-02T23:44:57.739 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.737+0000 7fd9d642a640 1 -- 172.21.15.73:0/1144241853 --> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) -- 0x7fd9d0072e80 con 0x7fd9bc038390 2024-08-02T23:44:57.740 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.738+0000 7fd9ccff9640 1 -- 172.21.15.73:0/1144241853 <== mgr.4100 v2:172.21.15.131:6800/3351532440 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) ==== 41+0+0 (crc 0 0 0) 0x7fd9d0072e80 con 0x7fd9bc038390 2024-08-02T23:44:57.740 INFO:teuthology.orchestra.run.smithi073.stderr:instructed osd(s) 1 to deep-scrub 2024-08-02T23:44:57.743 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.740+0000 7fd9d642a640 1 -- 172.21.15.73:0/1144241853 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7fd9bc038390 msgr2=0x7fd9bc03a8a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-02T23:44:57.743 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.740+0000 7fd9d642a640 1 --2- 172.21.15.73:0/1144241853 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7fd9bc038390 0x7fd9bc03a8a0 crc :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:44:57.743 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.740+0000 7fd9d642a640 1 -- 172.21.15.73:0/1144241853 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd9d006e880 msgr2=0x7fd9d011eea0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-02T23:44:57.743 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.740+0000 7fd9d642a640 1 --2- 172.21.15.73:0/1144241853 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd9d006e880 0x7fd9d011eea0 crc :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:44:57.743 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.740+0000 7fd9d642a640 1 -- 172.21.15.73:0/1144241853 shutdown_connections 2024-08-02T23:44:57.744 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.740+0000 7fd9d642a640 1 --2- 172.21.15.73:0/1144241853 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7fd9bc038390 0x7fd9bc03a8a0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:44:57.744 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.740+0000 7fd9d642a640 1 --2- 172.21.15.73:0/1144241853 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd9d006e880 0x7fd9d011eea0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:44:57.744 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.740+0000 7fd9d642a640 1 -- 172.21.15.73:0/1144241853 >> 172.21.15.73:0/1144241853 conn(0x7fd9d00a0590 msgr2=0x7fd9d01b9470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-02T23:44:57.744 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.740+0000 7fd9d642a640 1 -- 172.21.15.73:0/1144241853 shutdown_connections 2024-08-02T23:44:57.744 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.741+0000 7fd9d642a640 1 -- 172.21.15.73:0/1144241853 wait complete. 2024-08-02T23:44:57.755 INFO:tasks.ceph:Scrubbing osd.2 2024-08-02T23:44:57.756 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-08-02T23:44:57.864 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.855+0000 7f1ffcb50640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-02T23:44:57.865 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.863+0000 7f1ffcb50640 1 Processor -- start 2024-08-02T23:44:57.865 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.863+0000 7f1ffcb50640 1 -- start start 2024-08-02T23:44:57.866 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.864+0000 7f1ffcb50640 1 --2- >> v2:172.21.15.73:3300/0 conn(0x7f1ff8073c70 0x7f1ff8099790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-02T23:44:57.866 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.864+0000 7f1ffcb50640 1 -- --> v1:172.21.15.73:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1ff806d6e0 con 0x7f1ff8099cd0 2024-08-02T23:44:57.866 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.864+0000 7f1ffcb50640 1 -- --> v2:172.21.15.73:3300/0 -- mon_getmap magic: 0 -- 0x7f1ff809c440 con 0x7f1ff8073c70 2024-08-02T23:44:57.867 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.864+0000 7f1ff659c640 1 --2- >> v2:172.21.15.73:3300/0 conn(0x7f1ff8073c70 0x7f1ff8099790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-02T23:44:57.867 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.864+0000 7f1feffff640 1 --1- >> v1:172.21.15.73:6789/0 conn(0x7f1ff8099cd0 0x7f1ff8132400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.73:6789/0 says I am v1:172.21.15.73:53458/0 (socket says 172.21.15.73:53458) 2024-08-02T23:44:57.867 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.864+0000 7f1feffff640 1 -- 172.21.15.73:0/1653766719 learned_addr learned my addr 172.21.15.73:0/1653766719 (peer_addr_for_me v1:172.21.15.73:0/0) 2024-08-02T23:44:57.867 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.864+0000 7f1ff659c640 1 -- 172.21.15.73:0/1653766719 >> v1:172.21.15.73:6789/0 conn(0x7f1ff8099cd0 legacy=0x7f1ff8132400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-02T23:44:57.867 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.864+0000 7f1ff659c640 1 -- 172.21.15.73:0/1653766719 --> v2:172.21.15.73:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ff8070710 con 0x7f1ff8073c70 2024-08-02T23:44:57.867 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.864+0000 7f1ff659c640 1 --2- 172.21.15.73:0/1653766719 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1ff8073c70 0x7f1ff8099790 crc :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a97820a1f56f0fda server_cookie=0 in_seq=0 out_seq=0 2024-08-02T23:44:57.867 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.864+0000 7f1ff5d9b640 1 -- 172.21.15.73:0/1653766719 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f1fe8009ce0 con 0x7f1ff8073c70 2024-08-02T23:44:57.867 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.864+0000 7f1ff5d9b640 1 -- 172.21.15.73:0/1653766719 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f1fe8009e40 con 0x7f1ff8073c70 2024-08-02T23:44:57.867 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.864+0000 7f1ff5d9b640 1 -- 172.21.15.73:0/1653766719 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f1fe800e360 con 0x7f1ff8073c70 2024-08-02T23:44:57.869 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.865+0000 7f1ffcb50640 1 -- 172.21.15.73:0/1653766719 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1ff8073c70 msgr2=0x7f1ff8099790 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-02T23:44:57.870 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.865+0000 7f1ffcb50640 1 --2- 172.21.15.73:0/1653766719 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1ff8073c70 0x7f1ff8099790 crc :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:44:57.870 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.865+0000 7f1ffcb50640 1 -- 172.21.15.73:0/1653766719 shutdown_connections 2024-08-02T23:44:57.870 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.865+0000 7f1ffcb50640 1 --2- 172.21.15.73:0/1653766719 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1ff8073c70 0x7f1ff8099790 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:44:57.870 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.865+0000 7f1ffcb50640 1 -- 172.21.15.73:0/1653766719 >> 172.21.15.73:0/1653766719 conn(0x7f1ff8067aa0 msgr2=0x7f1ff8067eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-02T23:44:57.870 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.865+0000 7f1ffcb50640 1 -- 172.21.15.73:0/1653766719 shutdown_connections 2024-08-02T23:44:57.870 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.865+0000 7f1ffcb50640 1 -- 172.21.15.73:0/1653766719 wait complete. 2024-08-02T23:44:57.870 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.865+0000 7f1ffcb50640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-02T23:44:57.871 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.866+0000 7f1ffcb50640 1 Processor -- start 2024-08-02T23:44:57.871 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.866+0000 7f1ffcb50640 1 -- start start 2024-08-02T23:44:57.871 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.867+0000 7f1ffcb50640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1ff8099cd0 0x7f1ff81315c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-02T23:44:57.871 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.867+0000 7f1ffcb50640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 -- 0x7f1ff806e1c0 con 0x7f1ff8099cd0 2024-08-02T23:44:57.871 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.867+0000 7f1ff659c640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1ff8099cd0 0x7f1ff81315c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-02T23:44:57.871 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.867+0000 7f1ff659c640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1ff8099cd0 0x7f1ff81315c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:56148/0 (socket says 172.21.15.73:56148) 2024-08-02T23:44:57.871 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.867+0000 7f1ff659c640 1 -- 172.21.15.73:0/2124644641 learned_addr learned my addr 172.21.15.73:0/2124644641 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-08-02T23:44:57.871 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.867+0000 7f1ff659c640 1 -- 172.21.15.73:0/2124644641 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ff8098b70 con 0x7f1ff8099cd0 2024-08-02T23:44:57.872 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.867+0000 7f1ff659c640 1 --2- 172.21.15.73:0/2124644641 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1ff8099cd0 0x7f1ff81315c0 crc :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-02T23:44:57.872 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.867+0000 7f1feeffd640 1 -- 172.21.15.73:0/2124644641 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f1fe8014540 con 0x7f1ff8099cd0 2024-08-02T23:44:57.872 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.868+0000 7f1feeffd640 1 -- 172.21.15.73:0/2124644641 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f1fe80146a0 con 0x7f1ff8099cd0 2024-08-02T23:44:57.872 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.868+0000 7f1feeffd640 1 -- 172.21.15.73:0/2124644641 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f1fe8014940 con 0x7f1ff8099cd0 2024-08-02T23:44:57.872 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.868+0000 7f1ffcb50640 1 -- 172.21.15.73:0/2124644641 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ff806f550 con 0x7f1ff8099cd0 2024-08-02T23:44:57.872 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.868+0000 7f1ffcb50640 1 -- 172.21.15.73:0/2124644641 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ff806fc50 con 0x7f1ff8099cd0 2024-08-02T23:44:57.873 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.868+0000 7f1ffcb50640 1 -- 172.21.15.73:0/2124644641 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f1ff809d070 con 0x7f1ff8099cd0 2024-08-02T23:44:57.873 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.868+0000 7f1feeffd640 1 -- 172.21.15.73:0/2124644641 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 5) ==== 45563+0+0 (crc 0 0 0) 0x7f1fe8014b80 con 0x7f1ff8099cd0 2024-08-02T23:44:57.873 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.869+0000 7f1feeffd640 1 --2- 172.21.15.73:0/2124644641 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7f1fd8038340 0x7f1fd803a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-02T23:44:57.873 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.869+0000 7f1feeffd640 1 -- 172.21.15.73:0/2124644641 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(230..230 src has 1..230) ==== 2700+0+0 (crc 0 0 0) 0x7f1fe8019040 con 0x7f1ff8099cd0 2024-08-02T23:44:57.874 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.869+0000 7f1feeffd640 1 --2- 172.21.15.73:0/2124644641 >> v2:172.21.15.131:6802/3558187202 conn(0x7f1fd803d340 0x7f1fd803f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-02T23:44:57.874 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.869+0000 7f1feeffd640 1 -- 172.21.15.73:0/2124644641 --> v2:172.21.15.131:6802/3558187202 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1fd803fe80 con 0x7f1fd803d340 2024-08-02T23:44:57.874 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.869+0000 7f1feeffd640 1 -- 172.21.15.73:0/2124644641 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_get_version_reply(handle=1 version=230) ==== 24+0+0 (crc 0 0 0) 0x7f1fe804b720 con 0x7f1ff8099cd0 2024-08-02T23:44:57.874 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.869+0000 7f1feffff640 1 --2- 172.21.15.73:0/2124644641 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7f1fd8038340 0x7f1fd803a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-02T23:44:57.874 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.869+0000 7f1feffff640 1 --2- 172.21.15.73:0/2124644641 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7f1fd8038340 0x7f1fd803a850 crc :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-02T23:44:57.874 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.869+0000 7f1ff6d9d640 1 --2- 172.21.15.73:0/2124644641 >> v2:172.21.15.131:6802/3558187202 conn(0x7f1fd803d340 0x7f1fd803f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-08-02T23:44:57.875 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.872+0000 7f1ff6d9d640 1 --2- 172.21.15.73:0/2124644641 >> v2:172.21.15.131:6802/3558187202 conn(0x7f1fd803d340 0x7f1fd803f7f0 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-02T23:44:57.875 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.873+0000 7f1feeffd640 1 -- 172.21.15.73:0/2124644641 <== osd.2 v2:172.21.15.131:6802/3558187202 1 ==== command_reply(tid 1: 0 ) ==== 8+0+5172 (crc 0 0 0) 0x7f1fd803fe80 con 0x7f1fd803d340 2024-08-02T23:44:57.878 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.876+0000 7f1ffcb50640 1 -- 172.21.15.73:0/2124644641 --> v2:172.21.15.131:6802/3558187202 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f1ff8070110 con 0x7f1fd803d340 2024-08-02T23:44:57.880 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.878+0000 7f1feeffd640 1 -- 172.21.15.73:0/2124644641 <== osd.2 v2:172.21.15.131:6802/3558187202 2 ==== command_reply(tid 2: 0 ) ==== 8+0+22 (crc 0 0 0) 0x7f1ff8070110 con 0x7f1fd803d340 2024-08-02T23:44:57.880 INFO:teuthology.orchestra.run.smithi073.stdout:{ 2024-08-02T23:44:57.880 INFO:teuthology.orchestra.run.smithi073.stdout: "success": "" 2024-08-02T23:44:57.880 INFO:teuthology.orchestra.run.smithi073.stdout:} 2024-08-02T23:44:57.881 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.878+0000 7f1ffcb50640 1 -- 172.21.15.73:0/2124644641 >> v2:172.21.15.131:6802/3558187202 conn(0x7f1fd803d340 msgr2=0x7f1fd803f7f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-02T23:44:57.881 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.878+0000 7f1ffcb50640 1 --2- 172.21.15.73:0/2124644641 >> v2:172.21.15.131:6802/3558187202 conn(0x7f1fd803d340 0x7f1fd803f7f0 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:44:57.881 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.878+0000 7f1ffcb50640 1 -- 172.21.15.73:0/2124644641 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7f1fd8038340 msgr2=0x7f1fd803a850 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-02T23:44:57.882 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.878+0000 7f1ffcb50640 1 --2- 172.21.15.73:0/2124644641 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7f1fd8038340 0x7f1fd803a850 crc :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:44:57.882 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.878+0000 7f1ffcb50640 1 -- 172.21.15.73:0/2124644641 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1ff8099cd0 msgr2=0x7f1ff81315c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-02T23:44:57.882 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.878+0000 7f1ffcb50640 1 --2- 172.21.15.73:0/2124644641 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1ff8099cd0 0x7f1ff81315c0 crc :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:44:57.882 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.878+0000 7f1ffcb50640 1 -- 172.21.15.73:0/2124644641 shutdown_connections 2024-08-02T23:44:57.882 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.878+0000 7f1ffcb50640 1 --2- 172.21.15.73:0/2124644641 >> v2:172.21.15.131:6802/3558187202 conn(0x7f1fd803d340 0x7f1fd803f7f0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:44:57.882 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.878+0000 7f1ffcb50640 1 --2- 172.21.15.73:0/2124644641 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7f1fd8038340 0x7f1fd803a850 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:44:57.882 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.878+0000 7f1ffcb50640 1 --2- 172.21.15.73:0/2124644641 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1ff8099cd0 0x7f1ff81315c0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:44:57.882 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.878+0000 7f1ffcb50640 1 -- 172.21.15.73:0/2124644641 >> 172.21.15.73:0/2124644641 conn(0x7f1ff8067aa0 msgr2=0x7f1ff809ebd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-02T23:44:57.883 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.878+0000 7f1ffcb50640 1 -- 172.21.15.73:0/2124644641 shutdown_connections 2024-08-02T23:44:57.883 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.878+0000 7f1ffcb50640 1 -- 172.21.15.73:0/2124644641 wait complete. 2024-08-02T23:44:57.890 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-08-02T23:44:57.995 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.989+0000 7fd45b8ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-02T23:44:57.996 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.994+0000 7fd45b8ad640 1 Processor -- start 2024-08-02T23:44:57.996 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.994+0000 7fd45b8ad640 1 -- start start 2024-08-02T23:44:57.997 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.995+0000 7fd45b8ad640 1 --2- >> v2:172.21.15.73:3300/0 conn(0x7fd454134350 0x7fd454136770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-02T23:44:57.997 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.995+0000 7fd45b8ad640 1 -- --> v1:172.21.15.73:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd45406fef0 con 0x7fd45406e880 2024-08-02T23:44:57.998 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.995+0000 7fd45b8ad640 1 -- --> v2:172.21.15.73:3300/0 -- mon_getmap magic: 0 -- 0x7fd45409e5d0 con 0x7fd454134350 2024-08-02T23:44:57.998 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.995+0000 7fd458e48640 1 --2- >> v2:172.21.15.73:3300/0 conn(0x7fd454134350 0x7fd454136770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-02T23:44:57.998 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.995+0000 7fd459649640 1 --1- >> v1:172.21.15.73:6789/0 conn(0x7fd45406e880 0x7fd45406ec90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.73:6789/0 says I am v1:172.21.15.73:53462/0 (socket says 172.21.15.73:53462) 2024-08-02T23:44:57.998 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.995+0000 7fd459649640 1 -- 172.21.15.73:0/1801248490 learned_addr learned my addr 172.21.15.73:0/1801248490 (peer_addr_for_me v1:172.21.15.73:0/0) 2024-08-02T23:44:57.998 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.995+0000 7fd458e48640 1 -- 172.21.15.73:0/1801248490 >> v1:172.21.15.73:6789/0 conn(0x7fd45406e880 legacy=0x7fd45406ec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-02T23:44:57.998 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.995+0000 7fd458e48640 1 -- 172.21.15.73:0/1801248490 --> v2:172.21.15.73:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd45406e430 con 0x7fd454134350 2024-08-02T23:44:57.998 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.996+0000 7fd458e48640 1 --2- 172.21.15.73:0/1801248490 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd454134350 0x7fd454136770 crc :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d0a88f52923107 server_cookie=0 in_seq=0 out_seq=0 2024-08-02T23:44:57.998 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.996+0000 7fd44bfff640 1 -- 172.21.15.73:0/1801248490 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fd444009ce0 con 0x7fd454134350 2024-08-02T23:44:57.999 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.996+0000 7fd44bfff640 1 -- 172.21.15.73:0/1801248490 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7fd444009e40 con 0x7fd454134350 2024-08-02T23:44:57.999 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.996+0000 7fd44bfff640 1 -- 172.21.15.73:0/1801248490 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fd44400e360 con 0x7fd454134350 2024-08-02T23:44:58.001 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.996+0000 7fd45b8ad640 1 -- 172.21.15.73:0/1801248490 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd454134350 msgr2=0x7fd454136770 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-02T23:44:58.001 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.996+0000 7fd45b8ad640 1 --2- 172.21.15.73:0/1801248490 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd454134350 0x7fd454136770 crc :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:44:58.001 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.996+0000 7fd45b8ad640 1 -- 172.21.15.73:0/1801248490 shutdown_connections 2024-08-02T23:44:58.001 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.996+0000 7fd45b8ad640 1 --2- 172.21.15.73:0/1801248490 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd454134350 0x7fd454136770 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:44:58.001 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.996+0000 7fd45b8ad640 1 -- 172.21.15.73:0/1801248490 >> 172.21.15.73:0/1801248490 conn(0x7fd4540a0590 msgr2=0x7fd4540a09a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-02T23:44:58.001 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.996+0000 7fd45b8ad640 1 -- 172.21.15.73:0/1801248490 shutdown_connections 2024-08-02T23:44:58.001 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.996+0000 7fd45b8ad640 1 -- 172.21.15.73:0/1801248490 wait complete. 2024-08-02T23:44:58.001 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.997+0000 7fd45b8ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-02T23:44:58.002 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.997+0000 7fd45b8ad640 1 Processor -- start 2024-08-02T23:44:58.002 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.998+0000 7fd45b8ad640 1 -- start start 2024-08-02T23:44:58.002 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.998+0000 7fd45b8ad640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd45406e880 0x7fd45411b280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-02T23:44:58.002 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.998+0000 7fd45b8ad640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 -- 0x7fd45411c490 con 0x7fd45406e880 2024-08-02T23:44:58.002 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.998+0000 7fd459649640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd45406e880 0x7fd45411b280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-02T23:44:58.002 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.998+0000 7fd459649640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd45406e880 0x7fd45411b280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:56166/0 (socket says 172.21.15.73:56166) 2024-08-02T23:44:58.002 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.998+0000 7fd459649640 1 -- 172.21.15.73:0/752757088 learned_addr learned my addr 172.21.15.73:0/752757088 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-08-02T23:44:58.002 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.999+0000 7fd459649640 1 -- 172.21.15.73:0/752757088 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd45411acc0 con 0x7fd45406e880 2024-08-02T23:44:58.003 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.999+0000 7fd459649640 1 --2- 172.21.15.73:0/752757088 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd45406e880 0x7fd45411b280 crc :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-02T23:44:58.003 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.999+0000 7fd449ffb640 1 -- 172.21.15.73:0/752757088 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fd4500059d0 con 0x7fd45406e880 2024-08-02T23:44:58.003 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.999+0000 7fd449ffb640 1 -- 172.21.15.73:0/752757088 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7fd450005b30 con 0x7fd45406e880 2024-08-02T23:44:58.003 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.999+0000 7fd449ffb640 1 -- 172.21.15.73:0/752757088 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fd450005dd0 con 0x7fd45406e880 2024-08-02T23:44:58.003 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.999+0000 7fd45b8ad640 1 -- 172.21.15.73:0/752757088 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd45411a9b0 con 0x7fd45406e880 2024-08-02T23:44:58.003 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:57.999+0000 7fd45b8ad640 1 -- 172.21.15.73:0/752757088 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd45411dd00 con 0x7fd45406e880 2024-08-02T23:44:58.004 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.000+0000 7fd449ffb640 1 -- 172.21.15.73:0/752757088 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 5) ==== 45563+0+0 (crc 0 0 0) 0x7fd4500070b0 con 0x7fd45406e880 2024-08-02T23:44:58.004 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.000+0000 7fd449ffb640 1 --2- 172.21.15.73:0/752757088 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7fd438038390 0x7fd43803a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-02T23:44:58.004 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.000+0000 7fd449ffb640 1 -- 172.21.15.73:0/752757088 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(230..230 src has 1..230) ==== 2700+0+0 (crc 0 0 0) 0x7fd450047a70 con 0x7fd45406e880 2024-08-02T23:44:58.004 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.000+0000 7fd45b8ad640 1 -- 172.21.15.73:0/752757088 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd45411e980 con 0x7fd45406e880 2024-08-02T23:44:58.004 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.000+0000 7fd458e48640 1 --2- 172.21.15.73:0/752757088 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7fd438038390 0x7fd43803a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-02T23:44:58.005 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.001+0000 7fd458e48640 1 --2- 172.21.15.73:0/752757088 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7fd438038390 0x7fd43803a8a0 crc :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-02T23:44:58.007 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.005+0000 7fd449ffb640 1 -- 172.21.15.73:0/752757088 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186747 (crc 0 0 0) 0x7fd45000d330 con 0x7fd45406e880 2024-08-02T23:44:58.182 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.180+0000 7fd45b8ad640 1 -- 172.21.15.73:0/752757088 --> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) -- 0x7fd45409bfc0 con 0x7fd438038390 2024-08-02T23:44:58.182 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.181+0000 7fd449ffb640 1 -- 172.21.15.73:0/752757088 <== mgr.4100 v2:172.21.15.131:6800/3351532440 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) ==== 41+0+0 (crc 0 0 0) 0x7fd45409bfc0 con 0x7fd438038390 2024-08-02T23:44:58.183 INFO:teuthology.orchestra.run.smithi073.stderr:instructed osd(s) 2 to deep-scrub 2024-08-02T23:44:58.185 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.183+0000 7fd45b8ad640 1 -- 172.21.15.73:0/752757088 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7fd438038390 msgr2=0x7fd43803a8a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-02T23:44:58.185 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.183+0000 7fd45b8ad640 1 --2- 172.21.15.73:0/752757088 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7fd438038390 0x7fd43803a8a0 crc :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:44:58.185 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.183+0000 7fd45b8ad640 1 -- 172.21.15.73:0/752757088 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd45406e880 msgr2=0x7fd45411b280 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-02T23:44:58.185 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.183+0000 7fd45b8ad640 1 --2- 172.21.15.73:0/752757088 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd45406e880 0x7fd45411b280 crc :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:44:58.186 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.183+0000 7fd45b8ad640 1 -- 172.21.15.73:0/752757088 shutdown_connections 2024-08-02T23:44:58.186 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.183+0000 7fd45b8ad640 1 --2- 172.21.15.73:0/752757088 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7fd438038390 0x7fd43803a8a0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:44:58.186 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.183+0000 7fd45b8ad640 1 --2- 172.21.15.73:0/752757088 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd45406e880 0x7fd45411b280 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:44:58.186 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.183+0000 7fd45b8ad640 1 -- 172.21.15.73:0/752757088 >> 172.21.15.73:0/752757088 conn(0x7fd4540a0590 msgr2=0x7fd454136770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-02T23:44:58.186 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.183+0000 7fd45b8ad640 1 -- 172.21.15.73:0/752757088 shutdown_connections 2024-08-02T23:44:58.186 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.183+0000 7fd45b8ad640 1 -- 172.21.15.73:0/752757088 wait complete. 2024-08-02T23:44:58.196 INFO:tasks.ceph:Scrubbing osd.3 2024-08-02T23:44:58.196 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-08-02T23:44:58.303 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.294+0000 7f3a73559640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-02T23:44:58.304 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.302+0000 7f3a73559640 1 Processor -- start 2024-08-02T23:44:58.304 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.302+0000 7f3a73559640 1 -- start start 2024-08-02T23:44:58.305 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.303+0000 7f3a73559640 1 --2- >> v2:172.21.15.73:3300/0 conn(0x7f3a6c09aa90 0x7f3a6c09aea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-02T23:44:58.306 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.303+0000 7f3a73559640 1 -- --> v1:172.21.15.73:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3a6c09bae0 con 0x7f3a6c1306d0 2024-08-02T23:44:58.306 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.303+0000 7f3a73559640 1 -- --> v2:172.21.15.73:3300/0 -- mon_getmap magic: 0 -- 0x7f3a6c09bd10 con 0x7f3a6c09aa90 2024-08-02T23:44:58.306 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.303+0000 7f3a712f5640 1 --2- >> v2:172.21.15.73:3300/0 conn(0x7f3a6c09aa90 0x7f3a6c09aea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-02T23:44:58.306 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.303+0000 7f3a70af4640 1 --1- >> v1:172.21.15.73:6789/0 conn(0x7f3a6c1306d0 0x7f3a6c130ae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.73:6789/0 says I am v1:172.21.15.73:53466/0 (socket says 172.21.15.73:53466) 2024-08-02T23:44:58.306 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.303+0000 7f3a70af4640 1 -- 172.21.15.73:0/2299428067 learned_addr learned my addr 172.21.15.73:0/2299428067 (peer_addr_for_me v1:172.21.15.73:0/0) 2024-08-02T23:44:58.306 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.303+0000 7f3a712f5640 1 -- 172.21.15.73:0/2299428067 >> v1:172.21.15.73:6789/0 conn(0x7f3a6c1306d0 legacy=0x7f3a6c130ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-02T23:44:58.306 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.304+0000 7f3a712f5640 1 -- 172.21.15.73:0/2299428067 --> v2:172.21.15.73:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a6c09a640 con 0x7f3a6c09aa90 2024-08-02T23:44:58.307 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.304+0000 7f3a712f5640 1 --2- 172.21.15.73:0/2299428067 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3a6c09aa90 0x7f3a6c09aea0 crc :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17687b393b4578a server_cookie=0 in_seq=0 out_seq=0 2024-08-02T23:44:58.307 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.304+0000 7f3a63fff640 1 -- 172.21.15.73:0/2299428067 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f3a6800ac70 con 0x7f3a6c09aa90 2024-08-02T23:44:58.307 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.304+0000 7f3a63fff640 1 -- 172.21.15.73:0/2299428067 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f3a6800add0 con 0x7f3a6c09aa90 2024-08-02T23:44:58.307 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.304+0000 7f3a63fff640 1 -- 172.21.15.73:0/2299428067 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f3a6800f470 con 0x7f3a6c09aa90 2024-08-02T23:44:58.310 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.304+0000 7f3a73559640 1 -- 172.21.15.73:0/2299428067 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3a6c09aa90 msgr2=0x7f3a6c09aea0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-02T23:44:58.310 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.304+0000 7f3a73559640 1 --2- 172.21.15.73:0/2299428067 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3a6c09aa90 0x7f3a6c09aea0 crc :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:44:58.310 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.304+0000 7f3a73559640 1 -- 172.21.15.73:0/2299428067 shutdown_connections 2024-08-02T23:44:58.310 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.304+0000 7f3a73559640 1 --2- 172.21.15.73:0/2299428067 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3a6c09aa90 0x7f3a6c09aea0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:44:58.310 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.304+0000 7f3a73559640 1 -- 172.21.15.73:0/2299428067 >> 172.21.15.73:0/2299428067 conn(0x7f3a6c067aa0 msgr2=0x7f3a6c067eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-02T23:44:58.311 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.305+0000 7f3a73559640 1 -- 172.21.15.73:0/2299428067 shutdown_connections 2024-08-02T23:44:58.311 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.305+0000 7f3a73559640 1 -- 172.21.15.73:0/2299428067 wait complete. 2024-08-02T23:44:58.311 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.305+0000 7f3a73559640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-02T23:44:58.311 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.306+0000 7f3a73559640 1 Processor -- start 2024-08-02T23:44:58.311 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.306+0000 7f3a73559640 1 -- start start 2024-08-02T23:44:58.311 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.306+0000 7f3a73559640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3a6c09aa90 0x7f3a6c12a160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-02T23:44:58.311 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.306+0000 7f3a73559640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 -- 0x7f3a6c098b60 con 0x7f3a6c09aa90 2024-08-02T23:44:58.311 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.306+0000 7f3a712f5640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3a6c09aa90 0x7f3a6c12a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-02T23:44:58.312 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.306+0000 7f3a712f5640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3a6c09aa90 0x7f3a6c12a160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:56186/0 (socket says 172.21.15.73:56186) 2024-08-02T23:44:58.312 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.306+0000 7f3a712f5640 1 -- 172.21.15.73:0/1972736055 learned_addr learned my addr 172.21.15.73:0/1972736055 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-08-02T23:44:58.312 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.307+0000 7f3a712f5640 1 -- 172.21.15.73:0/1972736055 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a6c09a090 con 0x7f3a6c09aa90 2024-08-02T23:44:58.312 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.307+0000 7f3a712f5640 1 --2- 172.21.15.73:0/1972736055 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3a6c09aa90 0x7f3a6c12a160 crc :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-02T23:44:58.312 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.307+0000 7f3a61ffb640 1 -- 172.21.15.73:0/1972736055 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f3a6800fe10 con 0x7f3a6c09aa90 2024-08-02T23:44:58.312 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.307+0000 7f3a61ffb640 1 -- 172.21.15.73:0/1972736055 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f3a68012430 con 0x7f3a6c09aa90 2024-08-02T23:44:58.312 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.307+0000 7f3a61ffb640 1 -- 172.21.15.73:0/1972736055 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f3a680126d0 con 0x7f3a6c09aa90 2024-08-02T23:44:58.312 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.307+0000 7f3a73559640 1 -- 172.21.15.73:0/1972736055 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a6c12c580 con 0x7f3a6c09aa90 2024-08-02T23:44:58.313 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.307+0000 7f3a73559640 1 -- 172.21.15.73:0/1972736055 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a6c12d460 con 0x7f3a6c09aa90 2024-08-02T23:44:58.313 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.308+0000 7f3a61ffb640 1 -- 172.21.15.73:0/1972736055 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 5) ==== 45563+0+0 (crc 0 0 0) 0x7f3a68012880 con 0x7f3a6c09aa90 2024-08-02T23:44:58.313 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.308+0000 7f3a73559640 1 -- 172.21.15.73:0/1972736055 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f3a6c09e690 con 0x7f3a6c09aa90 2024-08-02T23:44:58.313 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.308+0000 7f3a61ffb640 1 --2- 172.21.15.73:0/1972736055 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7f3a50038390 0x7f3a5003a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-02T23:44:58.313 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.308+0000 7f3a61ffb640 1 -- 172.21.15.73:0/1972736055 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(230..230 src has 1..230) ==== 2700+0+0 (crc 0 0 0) 0x7f3a68018040 con 0x7f3a6c09aa90 2024-08-02T23:44:58.313 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.308+0000 7f3a61ffb640 1 --2- 172.21.15.73:0/1972736055 >> v2:172.21.15.131:6809/1209903557 conn(0x7f3a5003d390 0x7f3a5003f840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-02T23:44:58.313 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.308+0000 7f3a61ffb640 1 -- 172.21.15.73:0/1972736055 --> v2:172.21.15.131:6809/1209903557 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3a5003fed0 con 0x7f3a5003d390 2024-08-02T23:44:58.313 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.308+0000 7f3a61ffb640 1 -- 172.21.15.73:0/1972736055 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_get_version_reply(handle=1 version=230) ==== 24+0+0 (crc 0 0 0) 0x7f3a68048db0 con 0x7f3a6c09aa90 2024-08-02T23:44:58.313 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.308+0000 7f3a70af4640 1 --2- 172.21.15.73:0/1972736055 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7f3a50038390 0x7f3a5003a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-02T23:44:58.314 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.309+0000 7f3a70af4640 1 --2- 172.21.15.73:0/1972736055 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7f3a50038390 0x7f3a5003a8a0 crc :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-02T23:44:58.314 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.309+0000 7f3a71af6640 1 --2- 172.21.15.73:0/1972736055 >> v2:172.21.15.131:6809/1209903557 conn(0x7f3a5003d390 0x7f3a5003f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-08-02T23:44:58.314 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.311+0000 7f3a71af6640 1 --2- 172.21.15.73:0/1972736055 >> v2:172.21.15.131:6809/1209903557 conn(0x7f3a5003d390 0x7f3a5003f840 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-02T23:44:58.315 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.313+0000 7f3a61ffb640 1 -- 172.21.15.73:0/1972736055 <== osd.3 v2:172.21.15.131:6809/1209903557 1 ==== command_reply(tid 1: 0 ) ==== 8+0+5172 (crc 0 0 0) 0x7f3a5003fed0 con 0x7f3a5003d390 2024-08-02T23:44:58.318 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.316+0000 7f3a73559640 1 -- 172.21.15.73:0/1972736055 --> v2:172.21.15.131:6809/1209903557 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f3a6c130050 con 0x7f3a5003d390 2024-08-02T23:44:58.319 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.317+0000 7f3a61ffb640 1 -- 172.21.15.73:0/1972736055 <== osd.3 v2:172.21.15.131:6809/1209903557 2 ==== command_reply(tid 2: 0 ) ==== 8+0+22 (crc 0 0 0) 0x7f3a6400a820 con 0x7f3a5003d390 2024-08-02T23:44:58.320 INFO:teuthology.orchestra.run.smithi073.stdout:{ 2024-08-02T23:44:58.320 INFO:teuthology.orchestra.run.smithi073.stdout: "success": "" 2024-08-02T23:44:58.320 INFO:teuthology.orchestra.run.smithi073.stdout:} 2024-08-02T23:44:58.321 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.318+0000 7f3a73559640 1 -- 172.21.15.73:0/1972736055 >> v2:172.21.15.131:6809/1209903557 conn(0x7f3a5003d390 msgr2=0x7f3a5003f840 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-02T23:44:58.321 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.318+0000 7f3a73559640 1 --2- 172.21.15.73:0/1972736055 >> v2:172.21.15.131:6809/1209903557 conn(0x7f3a5003d390 0x7f3a5003f840 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:44:58.321 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.318+0000 7f3a73559640 1 -- 172.21.15.73:0/1972736055 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7f3a50038390 msgr2=0x7f3a5003a8a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-02T23:44:58.321 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.318+0000 7f3a73559640 1 --2- 172.21.15.73:0/1972736055 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7f3a50038390 0x7f3a5003a8a0 crc :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:44:58.321 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.318+0000 7f3a73559640 1 -- 172.21.15.73:0/1972736055 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3a6c09aa90 msgr2=0x7f3a6c12a160 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-02T23:44:58.322 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.318+0000 7f3a73559640 1 --2- 172.21.15.73:0/1972736055 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3a6c09aa90 0x7f3a6c12a160 crc :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:44:58.322 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.318+0000 7f3a73559640 1 -- 172.21.15.73:0/1972736055 shutdown_connections 2024-08-02T23:44:58.322 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.318+0000 7f3a73559640 1 --2- 172.21.15.73:0/1972736055 >> v2:172.21.15.131:6809/1209903557 conn(0x7f3a5003d390 0x7f3a5003f840 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:44:58.322 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.318+0000 7f3a73559640 1 --2- 172.21.15.73:0/1972736055 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7f3a50038390 0x7f3a5003a8a0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:44:58.322 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.318+0000 7f3a73559640 1 --2- 172.21.15.73:0/1972736055 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3a6c09aa90 0x7f3a6c12a160 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:44:58.322 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.318+0000 7f3a73559640 1 -- 172.21.15.73:0/1972736055 >> 172.21.15.73:0/1972736055 conn(0x7f3a6c067aa0 msgr2=0x7f3a6c09e810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-02T23:44:58.322 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.318+0000 7f3a73559640 1 -- 172.21.15.73:0/1972736055 shutdown_connections 2024-08-02T23:44:58.323 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.318+0000 7f3a73559640 1 -- 172.21.15.73:0/1972736055 wait complete. 2024-08-02T23:44:58.330 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-08-02T23:44:58.435 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.429+0000 7f853baad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-02T23:44:58.435 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.434+0000 7f853baad640 1 Processor -- start 2024-08-02T23:44:58.436 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.434+0000 7f853baad640 1 -- start start 2024-08-02T23:44:58.437 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.435+0000 7f853baad640 1 --2- >> v2:172.21.15.73:3300/0 conn(0x7f8534070cf0 0x7f8534071100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-02T23:44:58.437 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.435+0000 7f853baad640 1 -- --> v1:172.21.15.73:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8534070830 con 0x7f8534131070 2024-08-02T23:44:58.437 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.435+0000 7f853baad640 1 -- --> v2:172.21.15.73:3300/0 -- mon_getmap magic: 0 -- 0x7f8534072aa0 con 0x7f8534070cf0 2024-08-02T23:44:58.437 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.435+0000 7f8539849640 1 --2- >> v2:172.21.15.73:3300/0 conn(0x7f8534070cf0 0x7f8534071100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-02T23:44:58.438 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.435+0000 7f8539048640 1 --1- >> v1:172.21.15.73:6789/0 conn(0x7f8534131070 0x7f8534131480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.73:6789/0 says I am v1:172.21.15.73:53476/0 (socket says 172.21.15.73:53476) 2024-08-02T23:44:58.438 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.435+0000 7f8539048640 1 -- 172.21.15.73:0/4185107700 learned_addr learned my addr 172.21.15.73:0/4185107700 (peer_addr_for_me v1:172.21.15.73:0/0) 2024-08-02T23:44:58.438 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.435+0000 7f8539849640 1 -- 172.21.15.73:0/4185107700 >> v1:172.21.15.73:6789/0 conn(0x7f8534131070 legacy=0x7f8534131480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-02T23:44:58.438 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.435+0000 7f8539849640 1 -- 172.21.15.73:0/4185107700 --> v2:172.21.15.73:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8534070520 con 0x7f8534070cf0 2024-08-02T23:44:58.438 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.435+0000 7f8539849640 1 --2- 172.21.15.73:0/4185107700 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8534070cf0 0x7f8534071100 crc :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4ee6adead9d762f server_cookie=0 in_seq=0 out_seq=0 2024-08-02T23:44:58.438 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.435+0000 7f8538847640 1 -- 172.21.15.73:0/4185107700 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f852400e040 con 0x7f8534070cf0 2024-08-02T23:44:58.438 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.435+0000 7f8538847640 1 -- 172.21.15.73:0/4185107700 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f8524007670 con 0x7f8534070cf0 2024-08-02T23:44:58.439 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.435+0000 7f8538847640 1 -- 172.21.15.73:0/4185107700 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f8524007910 con 0x7f8534070cf0 2024-08-02T23:44:58.441 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.436+0000 7f853baad640 1 -- 172.21.15.73:0/4185107700 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8534070cf0 msgr2=0x7f8534071100 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-02T23:44:58.441 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.436+0000 7f853baad640 1 --2- 172.21.15.73:0/4185107700 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8534070cf0 0x7f8534071100 crc :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:44:58.441 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.436+0000 7f853baad640 1 -- 172.21.15.73:0/4185107700 shutdown_connections 2024-08-02T23:44:58.441 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.436+0000 7f853baad640 1 --2- 172.21.15.73:0/4185107700 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8534070cf0 0x7f8534071100 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:44:58.441 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.436+0000 7f853baad640 1 -- 172.21.15.73:0/4185107700 >> 172.21.15.73:0/4185107700 conn(0x7f85340a02d0 msgr2=0x7f85340a06e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-02T23:44:58.441 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.436+0000 7f853baad640 1 -- 172.21.15.73:0/4185107700 shutdown_connections 2024-08-02T23:44:58.441 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.436+0000 7f853baad640 1 -- 172.21.15.73:0/4185107700 wait complete. 2024-08-02T23:44:58.441 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.436+0000 7f853baad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-02T23:44:58.442 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.437+0000 7f853baad640 1 Processor -- start 2024-08-02T23:44:58.442 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.437+0000 7f853baad640 1 -- start start 2024-08-02T23:44:58.442 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.438+0000 7f853baad640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8534070cf0 0x7f853411dde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-02T23:44:58.442 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.438+0000 7f853baad640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 -- 0x7f853411f9f0 con 0x7f8534070cf0 2024-08-02T23:44:58.442 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.438+0000 7f8539849640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8534070cf0 0x7f853411dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-02T23:44:58.442 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.438+0000 7f8539849640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8534070cf0 0x7f853411dde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:56202/0 (socket says 172.21.15.73:56202) 2024-08-02T23:44:58.442 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.438+0000 7f8539849640 1 -- 172.21.15.73:0/3892525639 learned_addr learned my addr 172.21.15.73:0/3892525639 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-08-02T23:44:58.442 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.438+0000 7f8539849640 1 -- 172.21.15.73:0/3892525639 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f853412e7a0 con 0x7f8534070cf0 2024-08-02T23:44:58.442 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.438+0000 7f8539849640 1 --2- 172.21.15.73:0/3892525639 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8534070cf0 0x7f853411dde0 crc :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-02T23:44:58.443 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.439+0000 7f85327fc640 1 -- 172.21.15.73:0/3892525639 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f852400e040 con 0x7f8534070cf0 2024-08-02T23:44:58.443 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.439+0000 7f85327fc640 1 -- 172.21.15.73:0/3892525639 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f8524012a80 con 0x7f8534070cf0 2024-08-02T23:44:58.443 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.439+0000 7f853baad640 1 -- 172.21.15.73:0/3892525639 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8534120ee0 con 0x7f8534070cf0 2024-08-02T23:44:58.443 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.439+0000 7f85327fc640 1 -- 172.21.15.73:0/3892525639 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f8524012d20 con 0x7f8534070cf0 2024-08-02T23:44:58.443 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.439+0000 7f853baad640 1 -- 172.21.15.73:0/3892525639 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f85341202d0 con 0x7f8534070cf0 2024-08-02T23:44:58.444 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.439+0000 7f85327fc640 1 -- 172.21.15.73:0/3892525639 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 5) ==== 45563+0+0 (crc 0 0 0) 0x7f85240153f0 con 0x7f8534070cf0 2024-08-02T23:44:58.444 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.440+0000 7f85327fc640 1 --2- 172.21.15.73:0/3892525639 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7f8520038390 0x7f852003a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-02T23:44:58.444 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.440+0000 7f85327fc640 1 -- 172.21.15.73:0/3892525639 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(230..230 src has 1..230) ==== 2700+0+0 (crc 0 0 0) 0x7f8524049490 con 0x7f8534070cf0 2024-08-02T23:44:58.444 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.440+0000 7f853baad640 1 -- 172.21.15.73:0/3892525639 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f853409fbd0 con 0x7f8534070cf0 2024-08-02T23:44:58.444 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.440+0000 7f8539048640 1 --2- 172.21.15.73:0/3892525639 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7f8520038390 0x7f852003a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-02T23:44:58.445 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.440+0000 7f8539048640 1 --2- 172.21.15.73:0/3892525639 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7f8520038390 0x7f852003a8a0 crc :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-02T23:44:58.446 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.445+0000 7f85327fc640 1 -- 172.21.15.73:0/3892525639 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186747 (crc 0 0 0) 0x7f852400e1a0 con 0x7f8534070cf0 2024-08-02T23:44:58.625 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.623+0000 7f853baad640 1 -- 172.21.15.73:0/3892525639 --> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) -- 0x7f85341220e0 con 0x7f8520038390 2024-08-02T23:44:58.625 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.623+0000 7f85327fc640 1 -- 172.21.15.73:0/3892525639 <== mgr.4100 v2:172.21.15.131:6800/3351532440 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) ==== 41+0+0 (crc 0 0 0) 0x7f8528001200 con 0x7f8520038390 2024-08-02T23:44:58.625 INFO:teuthology.orchestra.run.smithi073.stderr:instructed osd(s) 3 to deep-scrub 2024-08-02T23:44:58.627 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.625+0000 7f853baad640 1 -- 172.21.15.73:0/3892525639 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7f8520038390 msgr2=0x7f852003a8a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-02T23:44:58.628 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.625+0000 7f853baad640 1 --2- 172.21.15.73:0/3892525639 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7f8520038390 0x7f852003a8a0 crc :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:44:58.628 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.625+0000 7f853baad640 1 -- 172.21.15.73:0/3892525639 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8534070cf0 msgr2=0x7f853411dde0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-02T23:44:58.628 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.625+0000 7f853baad640 1 --2- 172.21.15.73:0/3892525639 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8534070cf0 0x7f853411dde0 crc :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:44:58.628 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.625+0000 7f853baad640 1 -- 172.21.15.73:0/3892525639 shutdown_connections 2024-08-02T23:44:58.628 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.625+0000 7f853baad640 1 --2- 172.21.15.73:0/3892525639 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7f8520038390 0x7f852003a8a0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:44:58.628 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.625+0000 7f853baad640 1 --2- 172.21.15.73:0/3892525639 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8534070cf0 0x7f853411dde0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:44:58.628 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.625+0000 7f853baad640 1 -- 172.21.15.73:0/3892525639 >> 172.21.15.73:0/3892525639 conn(0x7f85340a02d0 msgr2=0x7f853412f9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-02T23:44:58.628 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.625+0000 7f853baad640 1 -- 172.21.15.73:0/3892525639 shutdown_connections 2024-08-02T23:44:58.629 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.625+0000 7f853baad640 1 -- 172.21.15.73:0/3892525639 wait complete. 2024-08-02T23:44:58.639 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-08-02T23:44:58.743 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.738+0000 7f1f1926d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-02T23:44:58.743 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.742+0000 7f1f1926d640 1 Processor -- start 2024-08-02T23:44:58.743 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.742+0000 7f1f1926d640 1 -- start start 2024-08-02T23:44:58.745 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.742+0000 7f1f1926d640 1 --2- >> v2:172.21.15.73:3300/0 conn(0x7f1f1412f290 0x7f1f1412f6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-02T23:44:58.745 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.742+0000 7f1f1926d640 1 -- --> v1:172.21.15.73:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1f140a3650 con 0x7f1f141321f0 2024-08-02T23:44:58.745 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.742+0000 7f1f1926d640 1 -- --> v2:172.21.15.73:3300/0 -- mon_getmap magic: 0 -- 0x7f1f14071d10 con 0x7f1f1412f290 2024-08-02T23:44:58.745 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.743+0000 7f1f12ffd640 1 --2- >> v2:172.21.15.73:3300/0 conn(0x7f1f1412f290 0x7f1f1412f6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-02T23:44:58.746 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.743+0000 7f1f127fc640 1 --1- >> v1:172.21.15.73:6789/0 conn(0x7f1f141321f0 0x7f1f14132600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.73:6789/0 says I am v1:172.21.15.73:53480/0 (socket says 172.21.15.73:53480) 2024-08-02T23:44:58.746 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.743+0000 7f1f127fc640 1 -- 172.21.15.73:0/3991910701 learned_addr learned my addr 172.21.15.73:0/3991910701 (peer_addr_for_me v1:172.21.15.73:0/0) 2024-08-02T23:44:58.746 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.743+0000 7f1f12ffd640 1 -- 172.21.15.73:0/3991910701 >> v1:172.21.15.73:6789/0 conn(0x7f1f141321f0 legacy=0x7f1f14132600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-02T23:44:58.746 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.743+0000 7f1f12ffd640 1 -- 172.21.15.73:0/3991910701 --> v2:172.21.15.73:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f14073280 con 0x7f1f1412f290 2024-08-02T23:44:58.746 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.743+0000 7f1f11ffb640 1 -- 172.21.15.73:0/3991910701 <== mon.0 v1:172.21.15.73:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 547422146 0 0) 0x7f1f140a3650 con 0x7f1f141321f0 2024-08-02T23:44:58.746 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.743+0000 7f1f12ffd640 1 --2- 172.21.15.73:0/3991910701 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1f1412f290 0x7f1f1412f6a0 crc :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9aa4ad3a78416c94 server_cookie=0 in_seq=0 out_seq=0 2024-08-02T23:44:58.746 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.743+0000 7f1f11ffb640 1 -- 172.21.15.73:0/3991910701 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f1f04009ce0 con 0x7f1f1412f290 2024-08-02T23:44:58.746 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.743+0000 7f1f11ffb640 1 -- 172.21.15.73:0/3991910701 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f1f04009e40 con 0x7f1f1412f290 2024-08-02T23:44:58.747 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.743+0000 7f1f11ffb640 1 -- 172.21.15.73:0/3991910701 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f1f0400e360 con 0x7f1f1412f290 2024-08-02T23:44:58.749 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.744+0000 7f1f1926d640 1 -- 172.21.15.73:0/3991910701 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1f1412f290 msgr2=0x7f1f1412f6a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-02T23:44:58.749 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.744+0000 7f1f1926d640 1 --2- 172.21.15.73:0/3991910701 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1f1412f290 0x7f1f1412f6a0 crc :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:44:58.749 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.744+0000 7f1f1926d640 1 -- 172.21.15.73:0/3991910701 shutdown_connections 2024-08-02T23:44:58.749 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.744+0000 7f1f1926d640 1 --2- 172.21.15.73:0/3991910701 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1f1412f290 0x7f1f1412f6a0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:44:58.749 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.744+0000 7f1f1926d640 1 -- 172.21.15.73:0/3991910701 >> 172.21.15.73:0/3991910701 conn(0x7f1f140a13e0 msgr2=0x7f1f140a17f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-02T23:44:58.749 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.744+0000 7f1f1926d640 1 -- 172.21.15.73:0/3991910701 shutdown_connections 2024-08-02T23:44:58.749 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.744+0000 7f1f1926d640 1 -- 172.21.15.73:0/3991910701 wait complete. 2024-08-02T23:44:58.750 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.744+0000 7f1f1926d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-02T23:44:58.750 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.745+0000 7f1f1926d640 1 Processor -- start 2024-08-02T23:44:58.750 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.745+0000 7f1f1926d640 1 -- start start 2024-08-02T23:44:58.750 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.746+0000 7f1f1926d640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1f1412f290 0x7f1f1406ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-02T23:44:58.750 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.746+0000 7f1f1926d640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 -- 0x7f1f141204d0 con 0x7f1f1412f290 2024-08-02T23:44:58.750 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.746+0000 7f1f12ffd640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1f1412f290 0x7f1f1406ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-02T23:44:58.750 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.746+0000 7f1f12ffd640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1f1412f290 0x7f1f1406ccf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:56218/0 (socket says 172.21.15.73:56218) 2024-08-02T23:44:58.750 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.746+0000 7f1f12ffd640 1 -- 172.21.15.73:0/1382056196 learned_addr learned my addr 172.21.15.73:0/1382056196 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-08-02T23:44:58.751 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.746+0000 7f1f12ffd640 1 -- 172.21.15.73:0/1382056196 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f1412e980 con 0x7f1f1412f290 2024-08-02T23:44:58.751 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.746+0000 7f1f12ffd640 1 --2- 172.21.15.73:0/1382056196 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1f1412f290 0x7f1f1406ccf0 crc :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-02T23:44:58.751 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.746+0000 7f1efb7fe640 1 -- 172.21.15.73:0/1382056196 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f1f04014620 con 0x7f1f1412f290 2024-08-02T23:44:58.751 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.746+0000 7f1efb7fe640 1 -- 172.21.15.73:0/1382056196 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f1f04014780 con 0x7f1f1412f290 2024-08-02T23:44:58.751 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.747+0000 7f1f1926d640 1 -- 172.21.15.73:0/1382056196 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1f140a0610 con 0x7f1f1412f290 2024-08-02T23:44:58.751 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.747+0000 7f1efb7fe640 1 -- 172.21.15.73:0/1382056196 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f1f04014a20 con 0x7f1f1412f290 2024-08-02T23:44:58.751 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.747+0000 7f1f1926d640 1 -- 172.21.15.73:0/1382056196 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1f141201c0 con 0x7f1f1412f290 2024-08-02T23:44:58.752 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.747+0000 7f1efb7fe640 1 -- 172.21.15.73:0/1382056196 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 5) ==== 45563+0+0 (crc 0 0 0) 0x7f1f04014bd0 con 0x7f1f1412f290 2024-08-02T23:44:58.752 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.748+0000 7f1efb7fe640 1 --2- 172.21.15.73:0/1382056196 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7f1f00038390 0x7f1f0003a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-02T23:44:58.752 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.748+0000 7f1efb7fe640 1 -- 172.21.15.73:0/1382056196 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(230..230 src has 1..230) ==== 2700+0+0 (crc 0 0 0) 0x7f1f04019040 con 0x7f1f1412f290 2024-08-02T23:44:58.752 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.748+0000 7f1f1926d640 1 -- 172.21.15.73:0/1382056196 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1f1406e7d0 con 0x7f1f1412f290 2024-08-02T23:44:58.752 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.748+0000 7f1f127fc640 1 --2- 172.21.15.73:0/1382056196 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7f1f00038390 0x7f1f0003a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-02T23:44:58.753 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.748+0000 7f1f127fc640 1 --2- 172.21.15.73:0/1382056196 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7f1f00038390 0x7f1f0003a8a0 crc :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-02T23:44:58.754 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.753+0000 7f1efb7fe640 1 -- 172.21.15.73:0/1382056196 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186747 (crc 0 0 0) 0x7f1f04010310 con 0x7f1f1412f290 2024-08-02T23:44:58.930 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.928+0000 7f1f1926d640 1 -- 172.21.15.73:0/1382056196 --> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f1f141211c0 con 0x7f1f00038390 2024-08-02T23:44:58.931 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.929+0000 7f1efb7fe640 1 -- 172.21.15.73:0/1382056196 <== mgr.4100 v2:172.21.15.131:6800/3351532440 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+31247 (crc 0 0 0) 0x7f1f141211c0 con 0x7f1f00038390 2024-08-02T23:44:58.932 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-08-02T23:44:58.932 INFO:teuthology.orchestra.run.smithi073.stderr:dumped all 2024-08-02T23:44:58.934 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.931+0000 7f1f1926d640 1 -- 172.21.15.73:0/1382056196 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7f1f00038390 msgr2=0x7f1f0003a8a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-02T23:44:58.934 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.931+0000 7f1f1926d640 1 --2- 172.21.15.73:0/1382056196 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7f1f00038390 0x7f1f0003a8a0 crc :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:44:58.934 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.931+0000 7f1f1926d640 1 -- 172.21.15.73:0/1382056196 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1f1412f290 msgr2=0x7f1f1406ccf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-02T23:44:58.934 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.931+0000 7f1f1926d640 1 --2- 172.21.15.73:0/1382056196 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1f1412f290 0x7f1f1406ccf0 crc :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:44:58.934 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.931+0000 7f1f1926d640 1 -- 172.21.15.73:0/1382056196 shutdown_connections 2024-08-02T23:44:58.934 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.931+0000 7f1f1926d640 1 --2- 172.21.15.73:0/1382056196 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7f1f00038390 0x7f1f0003a8a0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:44:58.934 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.931+0000 7f1f1926d640 1 --2- 172.21.15.73:0/1382056196 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1f1412f290 0x7f1f1406ccf0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:44:58.935 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.931+0000 7f1f1926d640 1 -- 172.21.15.73:0/1382056196 >> 172.21.15.73:0/1382056196 conn(0x7f1f140a13e0 msgr2=0x7f1f140702a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-02T23:44:58.935 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.932+0000 7f1f1926d640 1 -- 172.21.15.73:0/1382056196 shutdown_connections 2024-08-02T23:44:58.935 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:44:58.932+0000 7f1f1926d640 1 -- 172.21.15.73:0/1382056196 wait complete. 2024-08-02T23:44:58.944 INFO:teuthology.orchestra.run.smithi073.stdout:{"pg_ready":true,"pg_map":{"version":371,"stamp":"2024-08-02T23:44:57.430982+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":209190912,"kb_used":22608,"kb_used_data":22608,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":209168304,"statfs":{"total":214211493888,"available":214188343296,"internally_reserved":0,"allocated":23150592,"data_stored":23150592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001231"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":582,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-08-02T23:44:28.441037+0000","last_change":"2024-08-02T23:40:06.540330+0000","last_active":"2024-08-02T23:44:28.441037+0000","last_peered":"2024-08-02T23:44:28.441037+0000","last_clean":"2024-08-02T23:44:28.441037+0000","last_became_active":"2024-08-02T23:40:04.552702+0000","last_became_peered":"2024-08-02T23:40:04.552702+0000","last_unstale":"2024-08-02T23:44:28.441037+0000","last_undegraded":"2024-08-02T23:44:28.441037+0000","last_fullsized":"2024-08-02T23:44:28.441037+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-02T23:40:03.497423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-02T23:40:03.497423+0000","last_clean_scrub_stamp":"2024-08-02T23:40:03.497423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":582,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-08-02T23:44:28.449743+0000","last_change":"2024-08-02T23:40:06.542935+0000","last_active":"2024-08-02T23:44:28.449743+0000","last_peered":"2024-08-02T23:44:28.449743+0000","last_clean":"2024-08-02T23:44:28.449743+0000","last_became_active":"2024-08-02T23:40:04.561840+0000","last_became_peered":"2024-08-02T23:40:04.561840+0000","last_unstale":"2024-08-02T23:44:28.449743+0000","last_undegraded":"2024-08-02T23:44:28.449743+0000","last_fullsized":"2024-08-02T23:44:28.449743+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-02T23:40:03.497423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-02T23:40:03.497423+0000","last_clean_scrub_stamp":"2024-08-02T23:40:03.497423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":588,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-08-02T23:44:30.466675+0000","last_change":"2024-08-02T23:40:06.540945+0000","last_active":"2024-08-02T23:44:30.466675+0000","last_peered":"2024-08-02T23:44:30.466675+0000","last_clean":"2024-08-02T23:44:30.466675+0000","last_became_active":"2024-08-02T23:40:04.560865+0000","last_became_peered":"2024-08-02T23:40:04.560865+0000","last_unstale":"2024-08-02T23:44:30.466675+0000","last_undegraded":"2024-08-02T23:44:30.466675+0000","last_fullsized":"2024-08-02T23:44:30.466675+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-02T23:40:03.497423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-02T23:40:03.497423+0000","last_clean_scrub_stamp":"2024-08-02T23:40:03.497423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":582,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-08-02T23:44:28.444871+0000","last_change":"2024-08-02T23:40:06.540508+0000","last_active":"2024-08-02T23:44:28.444871+0000","last_peered":"2024-08-02T23:44:28.444871+0000","last_clean":"2024-08-02T23:44:28.444871+0000","last_became_active":"2024-08-02T23:40:04.553411+0000","last_became_peered":"2024-08-02T23:40:04.553411+0000","last_unstale":"2024-08-02T23:44:28.444871+0000","last_undegraded":"2024-08-02T23:44:28.444871+0000","last_fullsized":"2024-08-02T23:44:28.444871+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-02T23:40:03.497423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-02T23:40:03.497423+0000","last_clean_scrub_stamp":"2024-08-02T23:40:03.497423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"16'2","reported_seq":584,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-08-02T23:44:28.446631+0000","last_change":"2024-08-02T23:40:06.538737+0000","last_active":"2024-08-02T23:44:28.446631+0000","last_peered":"2024-08-02T23:44:28.446631+0000","last_clean":"2024-08-02T23:44:28.446631+0000","last_became_active":"2024-08-02T23:40:04.553959+0000","last_became_peered":"2024-08-02T23:40:04.553959+0000","last_unstale":"2024-08-02T23:44:28.446631+0000","last_undegraded":"2024-08-02T23:44:28.446631+0000","last_fullsized":"2024-08-02T23:44:28.446631+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-02T23:40:03.497423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-02T23:40:03.497423+0000","last_clean_scrub_stamp":"2024-08-02T23:40:03.497423+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":582,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-08-02T23:44:28.442643+0000","last_change":"2024-08-02T23:40:06.540125+0000","last_active":"2024-08-02T23:44:28.442643+0000","last_peered":"2024-08-02T23:44:28.442643+0000","last_clean":"2024-08-02T23:44:28.442643+0000","last_became_active":"2024-08-02T23:40:04.556895+0000","last_became_peered":"2024-08-02T23:40:04.556895+0000","last_unstale":"2024-08-02T23:44:28.442643+0000","last_undegraded":"2024-08-02T23:44:28.442643+0000","last_fullsized":"2024-08-02T23:44:28.442643+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-02T23:40:03.497423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-02T23:40:03.497423+0000","last_clean_scrub_stamp":"2024-08-02T23:40:03.497423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":582,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-08-02T23:44:28.444605+0000","last_change":"2024-08-02T23:40:06.538748+0000","last_active":"2024-08-02T23:44:28.444605+0000","last_peered":"2024-08-02T23:44:28.444605+0000","last_clean":"2024-08-02T23:44:28.444605+0000","last_became_active":"2024-08-02T23:40:04.561398+0000","last_became_peered":"2024-08-02T23:40:04.561398+0000","last_unstale":"2024-08-02T23:44:28.444605+0000","last_undegraded":"2024-08-02T23:44:28.444605+0000","last_fullsized":"2024-08-02T23:44:28.444605+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-02T23:40:03.497423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-02T23:40:03.497423+0000","last_clean_scrub_stamp":"2024-08-02T23:40:03.497423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"14'1","reported_seq":583,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-08-02T23:44:28.447433+0000","last_change":"2024-08-02T23:40:06.541978+0000","last_active":"2024-08-02T23:44:28.447433+0000","last_peered":"2024-08-02T23:44:28.447433+0000","last_clean":"2024-08-02T23:44:28.447433+0000","last_became_active":"2024-08-02T23:40:04.550062+0000","last_became_peered":"2024-08-02T23:40:04.550062+0000","last_unstale":"2024-08-02T23:44:28.447433+0000","last_undegraded":"2024-08-02T23:44:28.447433+0000","last_fullsized":"2024-08-02T23:44:28.447433+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-02T23:40:03.497423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-02T23:40:03.497423+0000","last_clean_scrub_stamp":"2024-08-02T23:40:03.497423+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":723,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-08-02T23:44:30.451164+0000","last_change":"2024-08-02T23:40:00.557074+0000","last_active":"2024-08-02T23:44:30.451164+0000","last_peered":"2024-08-02T23:44:30.451164+0000","last_clean":"2024-08-02T23:44:30.451164+0000","last_became_active":"2024-08-02T23:40:00.546868+0000","last_became_peered":"2024-08-02T23:40:00.546868+0000","last_unstale":"2024-08-02T23:44:30.451164+0000","last_undegraded":"2024-08-02T23:44:30.451164+0000","last_fullsized":"2024-08-02T23:44:30.451164+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-02T23:39:59.483999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-02T23:39:59.483999+0000","last_clean_scrub_stamp":"2024-08-02T23:39:59.483999+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":214211493888,"available":214188527616,"internally_reserved":0,"allocated":22966272,"data_stored":22966272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53552873472,"available":53546135552,"internally_reserved":0,"allocated":6737920,"data_stored":6737920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":3,"up_from":10,"seq":42949673230,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":6760,"kb_used_data":6760,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52290968,"statfs":{"total":53552873472,"available":53545951232,"internally_reserved":0,"allocated":6922240,"data_stored":6922240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705934,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":4932,"kb_used_data":4932,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52292796,"statfs":{"total":53552873472,"available":53547823104,"internally_reserved":0,"allocated":5050368,"data_stored":5050368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705934,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":4608,"kb_used_data":4608,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52293120,"statfs":{"total":53552873472,"available":53548154880,"internally_reserved":0,"allocated":4718592,"data_stored":4718592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705934,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":6308,"kb_used_data":6308,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52291420,"statfs":{"total":53552873472,"available":53546414080,"internally_reserved":0,"allocated":6459392,"data_stored":6459392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":3,"total":53552873472,"available":53546135552,"internally_reserved":0,"allocated":6737920,"data_stored":6737920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53552873472,"available":53546414080,"internally_reserved":0,"allocated":6459392,"data_stored":6459392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53552873472,"available":53548154880,"internally_reserved":0,"allocated":4718592,"data_stored":4718592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53552873472,"available":53547823104,"internally_reserved":0,"allocated":5050368,"data_stored":5050368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":53552873472,"available":53546135552,"internally_reserved":0,"allocated":6737920,"data_stored":6737920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-02T23:44:58.947 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-08-02T23:40:03.497423+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=2, tm_hour=23, tm_min=40, tm_sec=3, tm_wday=4, tm_yday=215, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=2, tm_hour=23, tm_min=44, tm_sec=55, tm_wday=4, tm_yday=215, tm_isdst=0) 2024-08-02T23:44:58.947 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-08-02T23:40:03.497423+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=2, tm_hour=23, tm_min=40, tm_sec=3, tm_wday=4, tm_yday=215, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=2, tm_hour=23, tm_min=44, tm_sec=55, tm_wday=4, tm_yday=215, tm_isdst=0) 2024-08-02T23:44:58.947 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-08-02T23:40:03.497423+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=2, tm_hour=23, tm_min=40, tm_sec=3, tm_wday=4, tm_yday=215, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=2, tm_hour=23, tm_min=44, tm_sec=55, tm_wday=4, tm_yday=215, tm_isdst=0) 2024-08-02T23:44:58.948 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-08-02T23:40:03.497423+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=2, tm_hour=23, tm_min=40, tm_sec=3, tm_wday=4, tm_yday=215, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=2, tm_hour=23, tm_min=44, tm_sec=55, tm_wday=4, tm_yday=215, tm_isdst=0) 2024-08-02T23:44:58.948 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-08-02T23:40:03.497423+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=2, tm_hour=23, tm_min=40, tm_sec=3, tm_wday=4, tm_yday=215, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=2, tm_hour=23, tm_min=44, tm_sec=55, tm_wday=4, tm_yday=215, tm_isdst=0) 2024-08-02T23:44:58.948 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-08-02T23:40:03.497423+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=2, tm_hour=23, tm_min=40, tm_sec=3, tm_wday=4, tm_yday=215, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=2, tm_hour=23, tm_min=44, tm_sec=55, tm_wday=4, tm_yday=215, tm_isdst=0) 2024-08-02T23:44:58.948 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-08-02T23:40:03.497423+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=2, tm_hour=23, tm_min=40, tm_sec=3, tm_wday=4, tm_yday=215, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=2, tm_hour=23, tm_min=44, tm_sec=55, tm_wday=4, tm_yday=215, tm_isdst=0) 2024-08-02T23:44:58.948 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-08-02T23:40:03.497423+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=2, tm_hour=23, tm_min=40, tm_sec=3, tm_wday=4, tm_yday=215, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=2, tm_hour=23, tm_min=44, tm_sec=55, tm_wday=4, tm_yday=215, tm_isdst=0) 2024-08-02T23:44:58.948 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-08-02T23:39:59.483999+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=2, tm_hour=23, tm_min=39, tm_sec=59, tm_wday=4, tm_yday=215, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=2, tm_hour=23, tm_min=44, tm_sec=55, tm_wday=4, tm_yday=215, tm_isdst=0) 2024-08-02T23:44:58.949 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-08-02T23:45:18.950 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-08-02T23:45:19.068 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:45:19.063+0000 7f6d5486d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-02T23:45:19.069 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:45:19.067+0000 7f6d5486d640 1 Processor -- start 2024-08-02T23:45:19.069 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:45:19.067+0000 7f6d5486d640 1 -- start start 2024-08-02T23:45:19.071 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:45:19.068+0000 7f6d5486d640 1 --2- >> v2:172.21.15.73:3300/0 conn(0x7f6d5006e880 0x7f6d5006ec90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-02T23:45:19.071 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:45:19.068+0000 7f6d5486d640 1 -- --> v1:172.21.15.73:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6d5006fef0 con 0x7f6d5006f1d0 2024-08-02T23:45:19.071 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:45:19.068+0000 7f6d5486d640 1 -- --> v2:172.21.15.73:3300/0 -- mon_getmap magic: 0 -- 0x7f6d5009e5d0 con 0x7f6d5006e880 2024-08-02T23:45:19.073 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:45:19.068+0000 7f6d4e59c640 1 --2- >> v2:172.21.15.73:3300/0 conn(0x7f6d5006e880 0x7f6d5006ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-02T23:45:19.073 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:45:19.068+0000 7f6d45d9b640 1 --1- >> v1:172.21.15.73:6789/0 conn(0x7f6d5006f1d0 0x7f6d50131040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.73:6789/0 says I am v1:172.21.15.73:36262/0 (socket says 172.21.15.73:36262) 2024-08-02T23:45:19.074 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:45:19.068+0000 7f6d45d9b640 1 -- 172.21.15.73:0/308044945 learned_addr learned my addr 172.21.15.73:0/308044945 (peer_addr_for_me v1:172.21.15.73:0/0) 2024-08-02T23:45:19.074 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:45:19.068+0000 7f6d4e59c640 1 -- 172.21.15.73:0/308044945 >> v1:172.21.15.73:6789/0 conn(0x7f6d5006f1d0 legacy=0x7f6d50131040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-02T23:45:19.074 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:45:19.068+0000 7f6d4e59c640 1 -- 172.21.15.73:0/308044945 --> v2:172.21.15.73:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d5006e430 con 0x7f6d5006e880 2024-08-02T23:45:19.074 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:45:19.069+0000 7f6d4e59c640 1 --2- 172.21.15.73:0/308044945 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f6d5006e880 0x7f6d5006ec90 crc :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b19dad19872ad940 server_cookie=0 in_seq=0 out_seq=0 2024-08-02T23:45:19.074 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:45:19.069+0000 7f6d4dd9b640 1 -- 172.21.15.73:0/308044945 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f6d40009ce0 con 0x7f6d5006e880 2024-08-02T23:45:19.074 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:45:19.069+0000 7f6d4dd9b640 1 -- 172.21.15.73:0/308044945 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f6d40009e40 con 0x7f6d5006e880 2024-08-02T23:45:19.074 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:45:19.069+0000 7f6d4dd9b640 1 -- 172.21.15.73:0/308044945 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f6d4000e360 con 0x7f6d5006e880 2024-08-02T23:45:19.077 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:45:19.069+0000 7f6d5486d640 1 -- 172.21.15.73:0/308044945 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f6d5006e880 msgr2=0x7f6d5006ec90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-02T23:45:19.077 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:45:19.069+0000 7f6d5486d640 1 --2- 172.21.15.73:0/308044945 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f6d5006e880 0x7f6d5006ec90 crc :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:45:19.077 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:45:19.069+0000 7f6d5486d640 1 -- 172.21.15.73:0/308044945 shutdown_connections 2024-08-02T23:45:19.077 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:45:19.069+0000 7f6d5486d640 1 --2- 172.21.15.73:0/308044945 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f6d5006e880 0x7f6d5006ec90 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:45:19.077 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:45:19.069+0000 7f6d5486d640 1 -- 172.21.15.73:0/308044945 >> 172.21.15.73:0/308044945 conn(0x7f6d500a0590 msgr2=0x7f6d500a09a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-02T23:45:19.078 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:45:19.069+0000 7f6d5486d640 1 -- 172.21.15.73:0/308044945 shutdown_connections 2024-08-02T23:45:19.078 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:45:19.070+0000 7f6d5486d640 1 -- 172.21.15.73:0/308044945 wait complete. 2024-08-02T23:45:19.078 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:45:19.070+0000 7f6d5486d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-02T23:45:19.078 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:45:19.071+0000 7f6d5486d640 1 Processor -- start 2024-08-02T23:45:19.078 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:45:19.071+0000 7f6d5486d640 1 -- start start 2024-08-02T23:45:19.078 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:45:19.072+0000 7f6d5486d640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f6d5006e880 0x7f6d5011dcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-02T23:45:19.078 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:45:19.072+0000 7f6d5486d640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 -- 0x7f6d50124c80 con 0x7f6d5006e880 2024-08-02T23:45:19.079 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:45:19.072+0000 7f6d4e59c640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f6d5006e880 0x7f6d5011dcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-02T23:45:19.079 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:45:19.072+0000 7f6d4e59c640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f6d5006e880 0x7f6d5011dcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:50576/0 (socket says 172.21.15.73:50576) 2024-08-02T23:45:19.079 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:45:19.072+0000 7f6d4e59c640 1 -- 172.21.15.73:0/299036417 learned_addr learned my addr 172.21.15.73:0/299036417 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-08-02T23:45:19.079 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:45:19.072+0000 7f6d4e59c640 1 -- 172.21.15.73:0/299036417 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d5011f220 con 0x7f6d5006e880 2024-08-02T23:45:19.079 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:45:19.072+0000 7f6d4e59c640 1 --2- 172.21.15.73:0/299036417 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f6d5006e880 0x7f6d5011dcf0 crc :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-02T23:45:19.079 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:45:19.072+0000 7f6d477fe640 1 -- 172.21.15.73:0/299036417 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f6d40014540 con 0x7f6d5006e880 2024-08-02T23:45:19.079 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:45:19.072+0000 7f6d477fe640 1 -- 172.21.15.73:0/299036417 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f6d400146a0 con 0x7f6d5006e880 2024-08-02T23:45:19.079 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:45:19.072+0000 7f6d5486d640 1 -- 172.21.15.73:0/299036417 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d5006f840 con 0x7f6d5006e880 2024-08-02T23:45:19.080 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:45:19.072+0000 7f6d477fe640 1 -- 172.21.15.73:0/299036417 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f6d40014940 con 0x7f6d5006e880 2024-08-02T23:45:19.080 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:45:19.072+0000 7f6d5486d640 1 -- 172.21.15.73:0/299036417 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d50131930 con 0x7f6d5006e880 2024-08-02T23:45:19.080 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:45:19.073+0000 7f6d477fe640 1 -- 172.21.15.73:0/299036417 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 5) ==== 45563+0+0 (crc 0 0 0) 0x7f6d40014af0 con 0x7f6d5006e880 2024-08-02T23:45:19.080 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:45:19.074+0000 7f6d5486d640 1 -- 172.21.15.73:0/299036417 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6d5006ec90 con 0x7f6d5006e880 2024-08-02T23:45:19.080 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:45:19.074+0000 7f6d477fe640 1 --2- 172.21.15.73:0/299036417 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7f6d30038390 0x7f6d3003a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-02T23:45:19.080 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:45:19.074+0000 7f6d477fe640 1 -- 172.21.15.73:0/299036417 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(230..230 src has 1..230) ==== 2700+0+0 (crc 0 0 0) 0x7f6d40019040 con 0x7f6d5006e880 2024-08-02T23:45:19.080 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:45:19.074+0000 7f6d45d9b640 1 --2- 172.21.15.73:0/299036417 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7f6d30038390 0x7f6d3003a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-02T23:45:19.080 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:45:19.075+0000 7f6d45d9b640 1 --2- 172.21.15.73:0/299036417 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7f6d30038390 0x7f6d3003a8a0 crc :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-02T23:45:19.081 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:45:19.078+0000 7f6d477fe640 1 -- 172.21.15.73:0/299036417 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186747 (crc 0 0 0) 0x7f6d40018190 con 0x7f6d5006e880 2024-08-02T23:45:19.251 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:45:19.249+0000 7f6d5486d640 1 -- 172.21.15.73:0/299036417 --> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f6d50070890 con 0x7f6d30038390 2024-08-02T23:45:19.252 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:45:19.250+0000 7f6d477fe640 1 -- 172.21.15.73:0/299036417 <== mgr.4100 v2:172.21.15.131:6800/3351532440 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+31252 (crc 0 0 0) 0x7f6d50070890 con 0x7f6d30038390 2024-08-02T23:45:19.253 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-08-02T23:45:19.253 INFO:teuthology.orchestra.run.smithi073.stderr:dumped all 2024-08-02T23:45:19.254 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:45:19.252+0000 7f6d5486d640 1 -- 172.21.15.73:0/299036417 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7f6d30038390 msgr2=0x7f6d3003a8a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-02T23:45:19.254 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:45:19.252+0000 7f6d5486d640 1 --2- 172.21.15.73:0/299036417 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7f6d30038390 0x7f6d3003a8a0 crc :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:45:19.255 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:45:19.252+0000 7f6d5486d640 1 -- 172.21.15.73:0/299036417 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f6d5006e880 msgr2=0x7f6d5011dcf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-02T23:45:19.255 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:45:19.252+0000 7f6d5486d640 1 --2- 172.21.15.73:0/299036417 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f6d5006e880 0x7f6d5011dcf0 crc :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:45:19.255 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:45:19.252+0000 7f6d5486d640 1 -- 172.21.15.73:0/299036417 shutdown_connections 2024-08-02T23:45:19.255 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:45:19.252+0000 7f6d5486d640 1 --2- 172.21.15.73:0/299036417 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7f6d30038390 0x7f6d3003a8a0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:45:19.255 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:45:19.252+0000 7f6d5486d640 1 --2- 172.21.15.73:0/299036417 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f6d5006e880 0x7f6d5011dcf0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:45:19.255 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:45:19.252+0000 7f6d5486d640 1 -- 172.21.15.73:0/299036417 >> 172.21.15.73:0/299036417 conn(0x7f6d500a0590 msgr2=0x7f6d5009a9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-02T23:45:19.255 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:45:19.252+0000 7f6d5486d640 1 -- 172.21.15.73:0/299036417 shutdown_connections 2024-08-02T23:45:19.256 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:45:19.252+0000 7f6d5486d640 1 -- 172.21.15.73:0/299036417 wait complete. 2024-08-02T23:45:19.266 INFO:teuthology.orchestra.run.smithi073.stdout:{"pg_ready":true,"pg_map":{"version":381,"stamp":"2024-08-02T23:45:17.435072+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":209190912,"kb_used":12948,"kb_used_data":12948,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":209177964,"statfs":{"total":214211493888,"available":214198235136,"internally_reserved":0,"allocated":13258752,"data_stored":13258752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002402"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":659,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-08-02T23:44:57.741365+0000","last_change":"2024-08-02T23:44:57.751772+0000","last_active":"2024-08-02T23:44:57.741365+0000","last_peered":"2024-08-02T23:44:57.741365+0000","last_clean":"2024-08-02T23:44:57.741365+0000","last_became_active":"2024-08-02T23:40:04.552702+0000","last_became_peered":"2024-08-02T23:40:04.552702+0000","last_unstale":"2024-08-02T23:44:57.741365+0000","last_undegraded":"2024-08-02T23:44:57.741365+0000","last_fullsized":"2024-08-02T23:44:57.741365+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-02T23:44:57.751685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-02T23:44:57.751685+0000","last_clean_scrub_stamp":"2024-08-02T23:40:03.497423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":659,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-08-02T23:44:58.184630+0000","last_change":"2024-08-02T23:44:58.192043+0000","last_active":"2024-08-02T23:44:58.184630+0000","last_peered":"2024-08-02T23:44:58.184630+0000","last_clean":"2024-08-02T23:44:58.184630+0000","last_became_active":"2024-08-02T23:40:04.561840+0000","last_became_peered":"2024-08-02T23:40:04.561840+0000","last_unstale":"2024-08-02T23:44:58.184630+0000","last_undegraded":"2024-08-02T23:44:58.184630+0000","last_fullsized":"2024-08-02T23:44:58.184630+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-02T23:44:58.191912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-02T23:44:58.191912+0000","last_clean_scrub_stamp":"2024-08-02T23:40:03.497423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":659,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-08-02T23:44:58.627091+0000","last_change":"2024-08-02T23:44:58.634938+0000","last_active":"2024-08-02T23:44:58.627091+0000","last_peered":"2024-08-02T23:44:58.627091+0000","last_clean":"2024-08-02T23:44:58.627091+0000","last_became_active":"2024-08-02T23:40:04.560865+0000","last_became_peered":"2024-08-02T23:40:04.560865+0000","last_unstale":"2024-08-02T23:44:58.627091+0000","last_undegraded":"2024-08-02T23:44:58.627091+0000","last_fullsized":"2024-08-02T23:44:58.627091+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-02T23:44:58.634786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-02T23:44:58.634786+0000","last_clean_scrub_stamp":"2024-08-02T23:40:03.497423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":659,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-08-02T23:44:58.183169+0000","last_change":"2024-08-02T23:44:58.192069+0000","last_active":"2024-08-02T23:44:58.183169+0000","last_peered":"2024-08-02T23:44:58.183169+0000","last_clean":"2024-08-02T23:44:58.183169+0000","last_became_active":"2024-08-02T23:40:04.553411+0000","last_became_peered":"2024-08-02T23:40:04.553411+0000","last_unstale":"2024-08-02T23:44:58.183169+0000","last_undegraded":"2024-08-02T23:44:58.183169+0000","last_fullsized":"2024-08-02T23:44:58.183169+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-02T23:44:58.191952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-02T23:44:58.191952+0000","last_clean_scrub_stamp":"2024-08-02T23:40:03.497423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"16'2","reported_seq":661,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-08-02T23:44:57.297326+0000","last_change":"2024-08-02T23:44:57.311194+0000","last_active":"2024-08-02T23:44:57.297326+0000","last_peered":"2024-08-02T23:44:57.297326+0000","last_clean":"2024-08-02T23:44:57.297326+0000","last_became_active":"2024-08-02T23:40:04.553959+0000","last_became_peered":"2024-08-02T23:40:04.553959+0000","last_unstale":"2024-08-02T23:44:57.297326+0000","last_undegraded":"2024-08-02T23:44:57.297326+0000","last_fullsized":"2024-08-02T23:44:57.297326+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"16'2","last_scrub_stamp":"2024-08-02T23:44:57.311071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-02T23:44:57.311071+0000","last_clean_scrub_stamp":"2024-08-02T23:40:03.497423+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":659,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-08-02T23:44:57.740689+0000","last_change":"2024-08-02T23:44:57.750408+0000","last_active":"2024-08-02T23:44:57.740689+0000","last_peered":"2024-08-02T23:44:57.740689+0000","last_clean":"2024-08-02T23:44:57.740689+0000","last_became_active":"2024-08-02T23:40:04.556895+0000","last_became_peered":"2024-08-02T23:40:04.556895+0000","last_unstale":"2024-08-02T23:44:57.740689+0000","last_undegraded":"2024-08-02T23:44:57.740689+0000","last_fullsized":"2024-08-02T23:44:57.740689+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-02T23:44:57.750312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-02T23:44:57.750312+0000","last_clean_scrub_stamp":"2024-08-02T23:40:03.497423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":659,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-08-02T23:44:57.297373+0000","last_change":"2024-08-02T23:44:57.303721+0000","last_active":"2024-08-02T23:44:57.297373+0000","last_peered":"2024-08-02T23:44:57.297373+0000","last_clean":"2024-08-02T23:44:57.297373+0000","last_became_active":"2024-08-02T23:40:04.561398+0000","last_became_peered":"2024-08-02T23:40:04.561398+0000","last_unstale":"2024-08-02T23:44:57.297373+0000","last_undegraded":"2024-08-02T23:44:57.297373+0000","last_fullsized":"2024-08-02T23:44:57.297373+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-02T23:44:57.303634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-02T23:44:57.303634+0000","last_clean_scrub_stamp":"2024-08-02T23:40:03.497423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"14'1","reported_seq":660,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-08-02T23:44:57.741497+0000","last_change":"2024-08-02T23:44:57.754289+0000","last_active":"2024-08-02T23:44:57.741497+0000","last_peered":"2024-08-02T23:44:57.741497+0000","last_clean":"2024-08-02T23:44:57.741497+0000","last_became_active":"2024-08-02T23:40:04.550062+0000","last_became_peered":"2024-08-02T23:40:04.550062+0000","last_unstale":"2024-08-02T23:44:57.741497+0000","last_undegraded":"2024-08-02T23:44:57.741497+0000","last_fullsized":"2024-08-02T23:44:57.741497+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"14'1","last_scrub_stamp":"2024-08-02T23:44:57.754196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-02T23:44:57.754196+0000","last_clean_scrub_stamp":"2024-08-02T23:40:03.497423+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":794,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-08-02T23:44:58.625831+0000","last_change":"2024-08-02T23:44:58.643559+0000","last_active":"2024-08-02T23:44:58.625831+0000","last_peered":"2024-08-02T23:44:58.625831+0000","last_clean":"2024-08-02T23:44:58.625831+0000","last_became_active":"2024-08-02T23:40:00.546868+0000","last_became_peered":"2024-08-02T23:40:00.546868+0000","last_unstale":"2024-08-02T23:44:58.625831+0000","last_undegraded":"2024-08-02T23:44:58.625831+0000","last_fullsized":"2024-08-02T23:44:58.625831+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"11'97","last_scrub_stamp":"2024-08-02T23:44:58.643404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-02T23:44:58.643404+0000","last_clean_scrub_stamp":"2024-08-02T23:39:59.483999+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":214211493888,"available":214198235136,"internally_reserved":0,"allocated":13258752,"data_stored":13258752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53552873472,"available":53549301760,"internally_reserved":0,"allocated":3571712,"data_stored":3571712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":3,"up_from":10,"seq":42949673260,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":3488,"kb_used_data":3488,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52294240,"statfs":{"total":53552873472,"available":53549301760,"internally_reserved":0,"allocated":3571712,"data_stored":3571712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705964,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":2852,"kb_used_data":2852,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52294876,"statfs":{"total":53552873472,"available":53549953024,"internally_reserved":0,"allocated":2920448,"data_stored":2920448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705964,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":2844,"kb_used_data":2844,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52294884,"statfs":{"total":53552873472,"available":53549961216,"internally_reserved":0,"allocated":2912256,"data_stored":2912256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705964,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":3764,"kb_used_data":3764,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52293964,"statfs":{"total":53552873472,"available":53549019136,"internally_reserved":0,"allocated":3854336,"data_stored":3854336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":3,"total":53552873472,"available":53549301760,"internally_reserved":0,"allocated":3571712,"data_stored":3571712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53552873472,"available":53549019136,"internally_reserved":0,"allocated":3854336,"data_stored":3854336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53552873472,"available":53549961216,"internally_reserved":0,"allocated":2912256,"data_stored":2912256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53552873472,"available":53549953024,"internally_reserved":0,"allocated":2920448,"data_stored":2920448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":53552873472,"available":53549301760,"internally_reserved":0,"allocated":3571712,"data_stored":3571712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-02T23:45:19.267 DEBUG:teuthology.orchestra.run.smithi073:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-08-02T23:45:19.371 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:45:19.363+0000 7fa365b99640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-02T23:45:19.372 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:45:19.370+0000 7fa365b99640 1 Processor -- start 2024-08-02T23:45:19.372 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:45:19.370+0000 7fa365b99640 1 -- start start 2024-08-02T23:45:19.373 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:45:19.371+0000 7fa365b99640 1 --2- >> v2:172.21.15.73:3300/0 conn(0x7fa36009edc0 0x7fa360135de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-02T23:45:19.373 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:45:19.371+0000 7fa365b99640 1 -- --> v1:172.21.15.73:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa36009f770 con 0x7fa36009e2a0 2024-08-02T23:45:19.374 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:45:19.371+0000 7fa365b99640 1 -- --> v2:172.21.15.73:3300/0 -- mon_getmap magic: 0 -- 0x7fa36006ea90 con 0x7fa36009edc0 2024-08-02T23:45:19.374 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:45:19.371+0000 7fa35effd640 1 --2- >> v2:172.21.15.73:3300/0 conn(0x7fa36009edc0 0x7fa360135de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-02T23:45:19.374 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:45:19.371+0000 7fa35f7fe640 1 --1- >> v1:172.21.15.73:6789/0 conn(0x7fa36009e2a0 0x7fa36009e6b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.73:6789/0 says I am v1:172.21.15.73:36278/0 (socket says 172.21.15.73:36278) 2024-08-02T23:45:19.374 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:45:19.371+0000 7fa35f7fe640 1 -- 172.21.15.73:0/20993943 learned_addr learned my addr 172.21.15.73:0/20993943 (peer_addr_for_me v1:172.21.15.73:0/0) 2024-08-02T23:45:19.374 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:45:19.371+0000 7fa35effd640 1 -- 172.21.15.73:0/20993943 >> v1:172.21.15.73:6789/0 conn(0x7fa36009e2a0 legacy=0x7fa36009e6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-02T23:45:19.374 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:45:19.371+0000 7fa35effd640 1 -- 172.21.15.73:0/20993943 --> v2:172.21.15.73:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa36009f970 con 0x7fa36009edc0 2024-08-02T23:45:19.374 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:45:19.371+0000 7fa35effd640 1 --2- 172.21.15.73:0/20993943 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa36009edc0 0x7fa360135de0 crc :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68c0e54e7d1b27e3 server_cookie=0 in_seq=0 out_seq=0 2024-08-02T23:45:19.375 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:45:19.371+0000 7fa35e7fc640 1 -- 172.21.15.73:0/20993943 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fa354009ce0 con 0x7fa36009edc0 2024-08-02T23:45:19.375 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:45:19.371+0000 7fa35e7fc640 1 -- 172.21.15.73:0/20993943 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7fa354009e40 con 0x7fa36009edc0 2024-08-02T23:45:19.375 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:45:19.371+0000 7fa35e7fc640 1 -- 172.21.15.73:0/20993943 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fa35400e360 con 0x7fa36009edc0 2024-08-02T23:45:19.378 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:45:19.372+0000 7fa365b99640 1 -- 172.21.15.73:0/20993943 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa36009edc0 msgr2=0x7fa360135de0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-02T23:45:19.378 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:45:19.372+0000 7fa365b99640 1 --2- 172.21.15.73:0/20993943 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa36009edc0 0x7fa360135de0 crc :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:45:19.378 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:45:19.372+0000 7fa365b99640 1 -- 172.21.15.73:0/20993943 shutdown_connections 2024-08-02T23:45:19.378 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:45:19.372+0000 7fa365b99640 1 --2- 172.21.15.73:0/20993943 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa36009edc0 0x7fa360135de0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:45:19.378 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:45:19.372+0000 7fa365b99640 1 -- 172.21.15.73:0/20993943 >> 172.21.15.73:0/20993943 conn(0x7fa3600a12a0 msgr2=0x7fa3600a16b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-02T23:45:19.379 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:45:19.372+0000 7fa365b99640 1 -- 172.21.15.73:0/20993943 shutdown_connections 2024-08-02T23:45:19.379 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:45:19.372+0000 7fa365b99640 1 -- 172.21.15.73:0/20993943 wait complete. 2024-08-02T23:45:19.379 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:45:19.372+0000 7fa365b99640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-02T23:45:19.379 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:45:19.373+0000 7fa365b99640 1 Processor -- start 2024-08-02T23:45:19.379 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:45:19.373+0000 7fa365b99640 1 -- start start 2024-08-02T23:45:19.379 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:45:19.374+0000 7fa365b99640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa36009e2a0 0x7fa36011ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-02T23:45:19.379 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:45:19.374+0000 7fa365b99640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 -- 0x7fa36009d710 con 0x7fa36009e2a0 2024-08-02T23:45:19.379 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:45:19.374+0000 7fa35f7fe640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa36009e2a0 0x7fa36011ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-02T23:45:19.380 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:45:19.374+0000 7fa35f7fe640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa36009e2a0 0x7fa36011ffa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:50586/0 (socket says 172.21.15.73:50586) 2024-08-02T23:45:19.380 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:45:19.374+0000 7fa35f7fe640 1 -- 172.21.15.73:0/1140372305 learned_addr learned my addr 172.21.15.73:0/1140372305 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-08-02T23:45:19.380 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:45:19.374+0000 7fa35f7fe640 1 -- 172.21.15.73:0/1140372305 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa360136820 con 0x7fa36009e2a0 2024-08-02T23:45:19.380 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:45:19.374+0000 7fa35f7fe640 1 --2- 172.21.15.73:0/1140372305 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa36009e2a0 0x7fa36011ffa0 crc :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-02T23:45:19.380 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:45:19.375+0000 7fa347fff640 1 -- 172.21.15.73:0/1140372305 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fa3500070b0 con 0x7fa36009e2a0 2024-08-02T23:45:19.380 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:45:19.375+0000 7fa347fff640 1 -- 172.21.15.73:0/1140372305 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7fa350007210 con 0x7fa36009e2a0 2024-08-02T23:45:19.380 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:45:19.375+0000 7fa365b99640 1 -- 172.21.15.73:0/1140372305 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa3601235a0 con 0x7fa36009e2a0 2024-08-02T23:45:19.380 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:45:19.375+0000 7fa347fff640 1 -- 172.21.15.73:0/1140372305 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fa3500074b0 con 0x7fa36009e2a0 2024-08-02T23:45:19.380 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:45:19.375+0000 7fa365b99640 1 -- 172.21.15.73:0/1140372305 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa360124040 con 0x7fa36009e2a0 2024-08-02T23:45:19.381 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:45:19.375+0000 7fa347fff640 1 -- 172.21.15.73:0/1140372305 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 5) ==== 45563+0+0 (crc 0 0 0) 0x7fa3500076a0 con 0x7fa36009e2a0 2024-08-02T23:45:19.381 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:45:19.376+0000 7fa347fff640 1 --2- 172.21.15.73:0/1140372305 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7fa34c038390 0x7fa34c03a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-02T23:45:19.381 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:45:19.376+0000 7fa347fff640 1 -- 172.21.15.73:0/1140372305 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(230..230 src has 1..230) ==== 2700+0+0 (crc 0 0 0) 0x7fa350048580 con 0x7fa36009e2a0 2024-08-02T23:45:19.381 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:45:19.376+0000 7fa365b99640 1 -- 172.21.15.73:0/1140372305 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa360069280 con 0x7fa36009e2a0 2024-08-02T23:45:19.381 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:45:19.376+0000 7fa35effd640 1 --2- 172.21.15.73:0/1140372305 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7fa34c038390 0x7fa34c03a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-02T23:45:19.381 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:45:19.376+0000 7fa35effd640 1 --2- 172.21.15.73:0/1140372305 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7fa34c038390 0x7fa34c03a8a0 crc :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-02T23:45:19.382 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:45:19.381+0000 7fa347fff640 1 -- 172.21.15.73:0/1140372305 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186747 (crc 0 0 0) 0x7fa3500094a0 con 0x7fa36009e2a0 2024-08-02T23:45:19.557 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:45:19.556+0000 7fa365b99640 1 -- 172.21.15.73:0/1140372305 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) -- 0x7fa3601224a0 con 0x7fa36009e2a0 2024-08-02T23:45:19.559 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:45:19.558+0000 7fa347fff640 1 -- 172.21.15.73:0/1140372305 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v2)=0 v2) ==== 123+0+0 (crc 0 0 0) 0x7fa3500090b0 con 0x7fa36009e2a0 2024-08-02T23:45:19.560 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:45:19.558+0000 7fa347fff640 1 -- 172.21.15.73:0/1140372305 <== mon.0 v2:172.21.15.73:3300/0 8 ==== config(1 keys) ==== 35+0+0 (crc 0 0 0) 0x7fa3500094a0 con 0x7fa36009e2a0 2024-08-02T23:45:19.563 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:45:19.560+0000 7fa365b99640 1 -- 172.21.15.73:0/1140372305 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7fa34c038390 msgr2=0x7fa34c03a8a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-02T23:45:19.563 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:45:19.560+0000 7fa365b99640 1 --2- 172.21.15.73:0/1140372305 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7fa34c038390 0x7fa34c03a8a0 crc :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:45:19.563 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:45:19.560+0000 7fa365b99640 1 -- 172.21.15.73:0/1140372305 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa36009e2a0 msgr2=0x7fa36011ffa0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-02T23:45:19.563 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:45:19.560+0000 7fa365b99640 1 --2- 172.21.15.73:0/1140372305 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa36009e2a0 0x7fa36011ffa0 crc :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:45:19.563 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:45:19.560+0000 7fa365b99640 1 -- 172.21.15.73:0/1140372305 shutdown_connections 2024-08-02T23:45:19.563 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:45:19.560+0000 7fa365b99640 1 --2- 172.21.15.73:0/1140372305 >> [v2:172.21.15.131:6800/3351532440,v1:172.21.15.131:6801/3351532440] conn(0x7fa34c038390 0x7fa34c03a8a0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:45:19.563 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:45:19.560+0000 7fa365b99640 1 --2- 172.21.15.73:0/1140372305 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa36009e2a0 0x7fa36011ffa0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-02T23:45:19.564 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:45:19.560+0000 7fa365b99640 1 -- 172.21.15.73:0/1140372305 >> 172.21.15.73:0/1140372305 conn(0x7fa3600a12a0 msgr2=0x7fa36009d920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-02T23:45:19.564 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:45:19.561+0000 7fa365b99640 1 -- 172.21.15.73:0/1140372305 shutdown_connections 2024-08-02T23:45:19.564 INFO:teuthology.orchestra.run.smithi073.stderr:2024-08-02T23:45:19.561+0000 7fa365b99640 1 -- 172.21.15.73:0/1140372305 wait complete. 2024-08-02T23:45:19.574 INFO:teuthology.misc:Shutting down mds daemons... 2024-08-02T23:45:19.575 INFO:teuthology.misc:Shutting down osd daemons... 2024-08-02T23:45:19.575 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-08-02T23:45:19.575 INFO:teuthology.orchestra.run:waiting for 300 2024-08-02T23:45:19.674 INFO:tasks.ceph.osd.0:Stopped 2024-08-02T23:45:19.674 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-08-02T23:45:19.674 INFO:teuthology.orchestra.run:waiting for 300 2024-08-02T23:45:19.769 INFO:tasks.ceph.osd.1:Stopped 2024-08-02T23:45:19.769 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-08-02T23:45:19.769 INFO:teuthology.orchestra.run:waiting for 300 2024-08-02T23:45:19.864 INFO:tasks.ceph.osd.2:Stopped 2024-08-02T23:45:19.864 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-08-02T23:45:19.864 INFO:teuthology.orchestra.run:waiting for 300 2024-08-02T23:45:19.962 INFO:tasks.ceph.osd.3:Stopped 2024-08-02T23:45:19.963 INFO:teuthology.misc:Shutting down mgr daemons... 2024-08-02T23:45:19.963 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-08-02T23:45:19.963 INFO:teuthology.orchestra.run:waiting for 300 2024-08-02T23:45:19.990 INFO:tasks.ceph.mgr.x:Stopped 2024-08-02T23:45:19.990 INFO:teuthology.misc:Shutting down mon daemons... 2024-08-02T23:45:19.991 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-08-02T23:45:19.991 INFO:teuthology.orchestra.run:waiting for 300 2024-08-02T23:45:20.003 INFO:tasks.ceph.mon.a:Stopped 2024-08-02T23:45:20.003 INFO:tasks.ceph:Checking cluster log for badness... 2024-08-02T23:45:20.003 DEBUG:teuthology.orchestra.run.smithi073:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'reached quota' | egrep -v 'but it is still running' | egrep -v 'overall HEALTH_' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(PG_DEGRADED\)' | head -n 1 2024-08-02T23:45:20.076 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi073.front.sepia.ceph.com 2024-08-02T23:45:20.076 DEBUG:teuthology.orchestra.run.smithi073:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-08-02T23:45:20.202 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi073.front.sepia.ceph.com 2024-08-02T23:45:20.203 DEBUG:teuthology.orchestra.run.smithi073:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-08-02T23:45:20.331 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi131.front.sepia.ceph.com 2024-08-02T23:45:20.331 DEBUG:teuthology.orchestra.run.smithi131:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-08-02T23:45:20.447 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi131.front.sepia.ceph.com 2024-08-02T23:45:20.448 DEBUG:teuthology.orchestra.run.smithi131:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-08-02T23:45:20.519 INFO:tasks.ceph:Archiving mon data... 2024-08-02T23:45:20.520 DEBUG:teuthology.misc:Transferring archived files from smithi073:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/yuriw-2024-08-02_15:39:04-crimson-rados-squid-release-distro-default-smithi/7833276/data/mon.a.tgz 2024-08-02T23:45:20.521 DEBUG:teuthology.orchestra.run.smithi073:> mktemp 2024-08-02T23:45:20.536 INFO:teuthology.orchestra.run.smithi073.stdout:/tmp/tmp.6vYFk7VW18 2024-08-02T23:45:20.537 DEBUG:teuthology.orchestra.run.smithi073:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.6vYFk7VW18 2024-08-02T23:45:20.695 DEBUG:teuthology.orchestra.run.smithi073:> sudo chmod 0666 /tmp/tmp.6vYFk7VW18 2024-08-02T23:45:20.778 DEBUG:teuthology.orchestra.remote:smithi073:/tmp/tmp.6vYFk7VW18 is 371KB 2024-08-02T23:45:20.874 DEBUG:teuthology.orchestra.run.smithi073:> rm -fr /tmp/tmp.6vYFk7VW18 2024-08-02T23:45:20.889 INFO:tasks.ceph:Cleaning ceph cluster... 2024-08-02T23:45:20.889 DEBUG:teuthology.orchestra.run.smithi073:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-08-02T23:45:20.931 DEBUG:teuthology.orchestra.run.smithi131:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-08-02T23:45:21.009 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-08-02T23:45:21.027 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-08-02T23:45:21.027 INFO:tasks.ceph:Archiving crash dumps... 2024-08-02T23:45:21.029 DEBUG:teuthology.misc:Transferring archived files from smithi073:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2024-08-02_15:39:04-crimson-rados-squid-release-distro-default-smithi/7833276/remote/smithi073/crash 2024-08-02T23:45:21.029 DEBUG:teuthology.orchestra.run.smithi073:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-08-02T23:45:21.074 DEBUG:teuthology.misc:Transferring archived files from smithi131:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2024-08-02_15:39:04-crimson-rados-squid-release-distro-default-smithi/7833276/remote/smithi131/crash 2024-08-02T23:45:21.075 DEBUG:teuthology.orchestra.run.smithi131:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-08-02T23:45:21.118 INFO:tasks.ceph:Compressing logs... 2024-08-02T23:45:21.118 DEBUG:teuthology.orchestra.run.smithi073:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-08-02T23:45:21.121 DEBUG:teuthology.orchestra.run.smithi131:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-08-02T23:45:21.158 INFO:teuthology.orchestra.run.smithi073.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30637.log 2024-08-02T23:45:21.159 INFO:teuthology.orchestra.run.smithi073.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30529.log 2024-08-02T23:45:21.159 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.30637.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32687.log 2024-08-02T23:45:21.159 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.30529.log: 85.5%gzip -- replaced with /var/log/ceph/ceph-client.admin.30637.log.gz 2024-08-02T23:45:21.159 INFO:teuthology.orchestra.run.smithi073.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.32351.log 2024-08-02T23:45:21.159 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.32687.log: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.30529.log.gz 2024-08-02T23:45:21.159 INFO:teuthology.orchestra.run.smithi073.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-08-02T23:45:21.159 INFO:teuthology.orchestra.run.smithi073.stderr: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.32687.log.gz 2024-08-02T23:45:21.160 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.32351.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32729.log 2024-08-02T23:45:21.160 INFO:teuthology.orchestra.run.smithi073.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.32351.log.gz 2024-08-02T23:45:21.160 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29963.log 2024-08-02T23:45:21.160 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.32729.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-08-02T23:45:21.160 INFO:teuthology.orchestra.run.smithi073.stderr: 85.5% -- replaced with /var/log/ceph/ceph-client.admin.32729.log.gz 2024-08-02T23:45:21.160 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.29963.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32435.log 2024-08-02T23:45:21.160 INFO:teuthology.orchestra.run.smithi073.stderr: 84.8% -- replaced with /var/log/ceph/ceph-client.admin.29963.log.gz 2024-08-02T23:45:21.161 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30527.log 2024-08-02T23:45:21.161 INFO:teuthology.orchestra.run.smithi073.stderr: 87.4% -- replaced with /var/log/ceph/ceph.log.gz 2024-08-02T23:45:21.161 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.32435.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32811.log 2024-08-02T23:45:21.161 INFO:teuthology.orchestra.run.smithi073.stderr: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.32435.log.gz 2024-08-02T23:45:21.161 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.30527.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30358.log 2024-08-02T23:45:21.161 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.32811.log: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.30527.log.gz 2024-08-02T23:45:21.161 INFO:teuthology.orchestra.run.smithi073.stderr:gzip -5 88.9% --verbose -- -- replaced with /var/log/ceph/ceph.audit.log.gz /var/log/ceph/ceph-client.admin.30217.log 2024-08-02T23:45:21.161 INFO:teuthology.orchestra.run.smithi073.stderr: 2024-08-02T23:45:21.162 INFO:teuthology.orchestra.run.smithi073.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.32811.log.gz 2024-08-02T23:45:21.162 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.30358.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-08-02T23:45:21.162 INFO:teuthology.orchestra.run.smithi073.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.30358.log.gz 2024-08-02T23:45:21.162 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.30217.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-08-02T23:45:21.162 INFO:teuthology.orchestra.run.smithi073.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.30217.log.gz 2024-08-02T23:45:21.162 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30259.log 2024-08-02T23:45:21.162 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32645.log 2024-08-02T23:45:21.162 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.30259.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32393.log 2024-08-02T23:45:21.163 INFO:teuthology.orchestra.run.smithi073.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.30259.log.gz 2024-08-02T23:45:21.163 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.32645.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29936.log 2024-08-02T23:45:21.163 INFO:teuthology.orchestra.run.smithi073.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.32645.log.gz 2024-08-02T23:45:21.163 INFO:teuthology.orchestra.run.smithi073.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32519.log 2024-08-02T23:45:21.163 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.32393.log: /var/log/ceph/ceph-client.admin.29936.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-08-02T23:45:21.163 INFO:teuthology.orchestra.run.smithi073.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.32393.log.gz 2024-08-02T23:45:21.163 INFO:teuthology.orchestra.run.smithi073.stderr: 84.8% -- replaced with /var/log/ceph/ceph-client.admin.29936.log.gz 2024-08-02T23:45:21.163 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.32519.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30679.log 2024-08-02T23:45:21.164 INFO:teuthology.orchestra.run.smithi073.stderr: 85.5% -- replaced with /var/log/ceph/ceph-client.admin.32519.log.gz 2024-08-02T23:45:21.164 INFO:teuthology.orchestra.run.smithi073.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30941.log 2024-08-02T23:45:21.164 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-mon.a.log: /var/log/ceph/ceph-client.admin.30679.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32771.log 2024-08-02T23:45:21.164 INFO:teuthology.orchestra.run.smithi073.stderr: 85.5% -- replaced with /var/log/ceph/ceph-client.admin.30679.log.gz 2024-08-02T23:45:21.164 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.30941.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30361.log 2024-08-02T23:45:21.164 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.32771.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.28911.log 2024-08-02T23:45:21.164 INFO:teuthology.orchestra.run.smithi073.stderr: 85.6%/var/log/ceph/ceph-client.admin.30361.log: -- replaced with /var/log/ceph/ceph-client.admin.32771.log.gz 2024-08-02T23:45:21.165 INFO:teuthology.orchestra.run.smithi073.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30360.log 2024-08-02T23:45:21.165 INFO:teuthology.orchestra.run.smithi073.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.30361.log.gz 2024-08-02T23:45:21.165 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph.tmp-client.admin.28911.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30088.log 2024-08-02T23:45:21.165 INFO:teuthology.orchestra.run.smithi073.stderr: 67.7% -- replaced with /var/log/ceph/ceph.tmp-client.admin.28911.log.gz 2024-08-02T23:45:21.165 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.30360.log: gzip -5 --verbose -- 87.3% /var/log/ceph/ceph-client.admin.32603.log -- replaced with /var/log/ceph/ceph-client.admin.30360.log.gz 2024-08-02T23:45:21.165 INFO:teuthology.orchestra.run.smithi073.stderr: 2024-08-02T23:45:21.165 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.30088.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30124.log 2024-08-02T23:45:21.165 INFO:teuthology.orchestra.run.smithi073.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.30088.log.gz 2024-08-02T23:45:21.165 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.32603.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32561.log 2024-08-02T23:45:21.166 INFO:teuthology.orchestra.run.smithi073.stderr: 85.5% -- replaced with /var/log/ceph/ceph-client.admin.32603.log.gz 2024-08-02T23:45:21.166 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.30124.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32309.log 2024-08-02T23:45:21.166 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.32561.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.30124.log.gz 2024-08-02T23:45:21.166 INFO:teuthology.orchestra.run.smithi073.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30031.log 2024-08-02T23:45:21.166 INFO:teuthology.orchestra.run.smithi073.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.32561.log.gz 2024-08-02T23:45:21.166 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.32309.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29701.log 2024-08-02T23:45:21.166 INFO:teuthology.orchestra.run.smithi073.stderr: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.32309.log.gz 2024-08-02T23:45:21.167 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.30031.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31410.log 2024-08-02T23:45:21.167 INFO:teuthology.orchestra.run.smithi073.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.30031.log.gz 2024-08-02T23:45:21.167 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.29701.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32477.log 2024-08-02T23:45:21.167 INFO:teuthology.orchestra.run.smithi073.stderr: 84.4% -- replaced with /var/log/ceph/ceph-client.admin.29701.log.gz 2024-08-02T23:45:21.167 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.31410.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30721.log 2024-08-02T23:45:21.167 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.32477.log: gzip -5 --verbose 92.4% -- /var/log/ceph/ceph-client.admin.32225.log 2024-08-02T23:45:21.167 INFO:teuthology.orchestra.run.smithi073.stderr: -- replaced with /var/log/ceph/ceph-client.admin.30941.log.gz 2024-08-02T23:45:21.167 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.30721.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.32477.log.gz 2024-08-02T23:45:21.168 INFO:teuthology.orchestra.run.smithi073.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30526.log 2024-08-02T23:45:21.168 INFO:teuthology.orchestra.run.smithi073.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.30721.log.gz 2024-08-02T23:45:21.168 INFO:teuthology.orchestra.run.smithi073.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32267.log 2024-08-02T23:45:21.168 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.32225.log: /var/log/ceph/ceph-client.admin.30526.log: gzip 85.5% -- replaced with /var/log/ceph/ceph-client.admin.32225.log.gz 2024-08-02T23:45:21.168 INFO:teuthology.orchestra.run.smithi073.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.30528.log 2024-08-02T23:45:21.168 INFO:teuthology.orchestra.run.smithi073.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.30526.log.gz 2024-08-02T23:45:21.168 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.32267.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30175.log 2024-08-02T23:45:21.168 INFO:teuthology.orchestra.run.smithi073.stderr: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.32267.log.gz 2024-08-02T23:45:21.169 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.30528.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30359.log 2024-08-02T23:45:21.169 INFO:teuthology.orchestra.run.smithi073.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.30528.log.gz 2024-08-02T23:45:21.169 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.30175.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29780.log 2024-08-02T23:45:21.169 INFO:teuthology.orchestra.run.smithi073.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.30175.log.gz 2024-08-02T23:45:21.169 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.30359.log: /var/log/ceph/ceph-client.admin.29780.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.30359.log.gz 2024-08-02T23:45:21.169 INFO:teuthology.orchestra.run.smithi073.stderr: 84.8% -- replaced with /var/log/ceph/ceph-client.admin.29780.log.gz 2024-08-02T23:45:21.170 INFO:teuthology.orchestra.run.smithi073.stderr: 93.4% -- replaced with /var/log/ceph/ceph-client.admin.31410.log.gz 2024-08-02T23:45:21.196 INFO:teuthology.orchestra.run.smithi131.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29487.log 2024-08-02T23:45:21.198 INFO:teuthology.orchestra.run.smithi131.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-08-02T23:45:21.198 INFO:teuthology.orchestra.run.smithi131.stderr:gzip/var/log/ceph/ceph-client.admin.29487.log: -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-08-02T23:45:21.198 INFO:teuthology.orchestra.run.smithi131.stderr: 84.9% -- replaced with /var/log/ceph/ceph-client.admin.29487.log.gz 2024-08-02T23:45:21.198 INFO:teuthology.orchestra.run.smithi131.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-08-02T23:45:21.198 INFO:teuthology.orchestra.run.smithi131.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29447.log 2024-08-02T23:45:21.198 INFO:teuthology.orchestra.run.smithi131.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29407.log 2024-08-02T23:45:21.199 INFO:teuthology.orchestra.run.smithi131.stderr:/var/log/ceph/ceph-client.admin.29447.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29527.log 2024-08-02T23:45:21.200 INFO:teuthology.orchestra.run.smithi131.stderr: 85.2% -- replaced with /var/log/ceph/ceph-client.admin.29447.log.gz 2024-08-02T23:45:21.200 INFO:teuthology.orchestra.run.smithi131.stderr:/var/log/ceph/ceph-client.admin.29407.log: /var/log/ceph/ceph-client.admin.29527.log: 85.0% -- replaced with /var/log/ceph/ceph-client.admin.29407.log.gz 2024-08-02T23:45:21.200 INFO:teuthology.orchestra.run.smithi131.stderr: 85.1% -- replaced with /var/log/ceph/ceph-client.admin.29527.log.gz 2024-08-02T23:45:21.228 INFO:teuthology.orchestra.run.smithi131.stderr: 92.5% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-08-02T23:45:21.419 INFO:teuthology.orchestra.run.smithi073.stderr: 91.4% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-08-02T23:45:22.084 INFO:teuthology.orchestra.run.smithi131.stderr: 94.3% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-08-02T23:45:22.096 INFO:teuthology.orchestra.run.smithi073.stderr: 94.4% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-08-02T23:45:22.120 INFO:teuthology.orchestra.run.smithi131.stderr: 94.4% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-08-02T23:45:22.122 INFO:teuthology.orchestra.run.smithi131.stderr: 2024-08-02T23:45:22.122 INFO:teuthology.orchestra.run.smithi131.stderr:real 0m0.936s 2024-08-02T23:45:22.122 INFO:teuthology.orchestra.run.smithi131.stderr:user 0m1.777s 2024-08-02T23:45:22.122 INFO:teuthology.orchestra.run.smithi131.stderr:sys 0m0.088s 2024-08-02T23:45:22.235 INFO:teuthology.orchestra.run.smithi073.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-08-02T23:45:22.237 INFO:teuthology.orchestra.run.smithi073.stderr: 2024-08-02T23:45:22.237 INFO:teuthology.orchestra.run.smithi073.stderr:real 0m1.104s 2024-08-02T23:45:22.237 INFO:teuthology.orchestra.run.smithi073.stderr:user 0m2.254s 2024-08-02T23:45:22.237 INFO:teuthology.orchestra.run.smithi073.stderr:sys 0m0.126s 2024-08-02T23:45:22.237 INFO:tasks.ceph:Archiving logs... 2024-08-02T23:45:22.238 DEBUG:teuthology.misc:Transferring archived files from smithi073:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-08-02_15:39:04-crimson-rados-squid-release-distro-default-smithi/7833276/remote/smithi073/log 2024-08-02T23:45:22.239 DEBUG:teuthology.orchestra.run.smithi073:> sudo tar c -f - -C /var/log/ceph -- . 2024-08-02T23:45:22.630 DEBUG:teuthology.misc:Transferring archived files from smithi131:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-08-02_15:39:04-crimson-rados-squid-release-distro-default-smithi/7833276/remote/smithi131/log 2024-08-02T23:45:22.631 DEBUG:teuthology.orchestra.run.smithi131:> sudo tar c -f - -C /var/log/ceph -- . 2024-08-02T23:45:22.899 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-08-02T23:45:22.908 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-08-02T23:45:22.908 DEBUG:teuthology.orchestra.run.smithi073:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-08-02T23:45:22.911 DEBUG:teuthology.orchestra.run.smithi131:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-08-02T23:45:23.002 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-08-02T23:45:23.011 INFO:teuthology.task.clock:Checking final clock skew... 2024-08-02T23:45:23.011 DEBUG:teuthology.orchestra.run.smithi073:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-08-02T23:45:23.014 DEBUG:teuthology.orchestra.run.smithi131:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-08-02T23:45:23.025 INFO:teuthology.orchestra.run.smithi073.stderr:bash: line 1: ntpq: command not found 2024-08-02T23:45:23.028 INFO:teuthology.orchestra.run.smithi131.stderr:bash: line 1: ntpq: command not found 2024-08-02T23:45:23.029 INFO:teuthology.orchestra.run.smithi073.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-08-02T23:45:23.029 INFO:teuthology.orchestra.run.smithi073.stdout:=============================================================================== 2024-08-02T23:45:23.029 INFO:teuthology.orchestra.run.smithi073.stdout:^* hv01.front.sepia.ceph.com 3 6 377 5 -214us[ -209us] +/- 45ms 2024-08-02T23:45:23.029 INFO:teuthology.orchestra.run.smithi073.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 4 +706us[ +706us] +/- 70ms 2024-08-02T23:45:23.030 INFO:teuthology.orchestra.run.smithi073.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 6 -353us[ -348us] +/- 87ms 2024-08-02T23:45:23.030 INFO:teuthology.orchestra.run.smithi073.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-08-02T23:45:23.032 INFO:teuthology.orchestra.run.smithi131.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-08-02T23:45:23.032 INFO:teuthology.orchestra.run.smithi131.stdout:=============================================================================== 2024-08-02T23:45:23.032 INFO:teuthology.orchestra.run.smithi131.stdout:^* hv01.front.sepia.ceph.com 3 6 377 4 -193us[ -178us] +/- 45ms 2024-08-02T23:45:23.032 INFO:teuthology.orchestra.run.smithi131.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 4 +659us[ +675us] +/- 70ms 2024-08-02T23:45:23.032 INFO:teuthology.orchestra.run.smithi131.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 4 -340us[ -340us] +/- 87ms 2024-08-02T23:45:23.033 INFO:teuthology.orchestra.run.smithi131.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-08-02T23:45:23.033 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-08-02T23:45:23.041 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-08-02T23:45:23.042 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-08-02T23:45:23.050 DEBUG:teuthology.orchestra.run.smithi073:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-08-02T23:45:23.130 DEBUG:teuthology.orchestra.run.smithi131:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-08-02T23:45:23.189 DEBUG:teuthology.orchestra.run.smithi073:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-08-02T23:45:23.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-02T23:45:23.216 DEBUG:teuthology.orchestra.run.smithi131:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-08-02T23:45:23.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-02T23:45:23.244 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-08-02T23:45:23.253 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-08-02T23:45:23.261 INFO:teuthology.task.internal:Duration was 813.378666 seconds 2024-08-02T23:45:23.261 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-08-02T23:45:23.269 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-08-02T23:45:23.270 DEBUG:teuthology.orchestra.run.smithi073:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-08-02T23:45:23.272 DEBUG:teuthology.orchestra.run.smithi131:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-08-02T23:45:23.308 INFO:teuthology.orchestra.run.smithi073.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-08-02T23:45:23.323 INFO:teuthology.orchestra.run.smithi131.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-08-02T23:45:23.728 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-08-02T23:45:23.728 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi073.front.sepia.ceph.com 2024-08-02T23:45:23.729 DEBUG:teuthology.orchestra.run.smithi073:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-08-02T23:45:23.789 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi131.front.sepia.ceph.com 2024-08-02T23:45:23.790 DEBUG:teuthology.orchestra.run.smithi131:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-08-02T23:45:23.813 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-08-02T23:45:23.814 DEBUG:teuthology.orchestra.run.smithi073:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-08-02T23:45:23.832 DEBUG:teuthology.orchestra.run.smithi131:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-08-02T23:45:23.882 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-08-02T23:45:23.882 DEBUG:teuthology.orchestra.run.smithi073:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-08-02T23:45:23.899 DEBUG:teuthology.orchestra.run.smithi131:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-08-02T23:45:24.019 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-08-02T23:45:24.029 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-08-02T23:45:24.029 DEBUG:teuthology.orchestra.run.smithi073:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-08-02T23:45:24.054 DEBUG:teuthology.orchestra.run.smithi131:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-08-02T23:45:24.080 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-08-02T23:45:24.089 DEBUG:teuthology.orchestra.run.smithi073:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-08-02T23:45:24.096 DEBUG:teuthology.orchestra.run.smithi131:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-08-02T23:45:24.117 INFO:teuthology.orchestra.run.smithi073.stdout:kernel.core_pattern = core 2024-08-02T23:45:24.145 INFO:teuthology.orchestra.run.smithi131.stdout:kernel.core_pattern = core 2024-08-02T23:45:24.172 DEBUG:teuthology.orchestra.run.smithi073:> test -e /home/ubuntu/cephtest/archive/coredump 2024-08-02T23:45:24.195 WARNING:teuthology.task.internal:Found coredumps on ubuntu@smithi073.front.sepia.ceph.com, flagging run as failed 2024-08-02T23:45:24.195 DEBUG:teuthology.orchestra.run.smithi131:> test -e /home/ubuntu/cephtest/archive/coredump 2024-08-02T23:45:24.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-02T23:45:24.227 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-08-02T23:45:24.237 INFO:teuthology.task.internal:Transferring archived files... 2024-08-02T23:45:24.238 DEBUG:teuthology.misc:Transferring archived files from smithi073:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-08-02_15:39:04-crimson-rados-squid-release-distro-default-smithi/7833276/remote/smithi073 2024-08-02T23:45:24.238 DEBUG:teuthology.orchestra.run.smithi073:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-08-02T23:45:30.822 INFO:teuthology.task.internal:Transferring binaries for coredumps... 2024-08-02T23:45:31.342 INFO:teuthology.task.internal: core looks like: /home/teuthworker/archive/yuriw-2024-08-02_15:39:04-crimson-rados-squid-release-distro-default-smithi/7833276/remote/smithi073/coredump/1722642293.31410.core.gz: gzip compressed data, was "1722642293.31410.core", last modified: Fri Aug 2 23:45:24 2024, max compression, original size modulo 2^32 138678272 2024-08-02T23:45:31.343 INFO:teuthology.task.internal:core is compressed, try accessing gzip file ... 2024-08-02T23:45:32.042 INFO:teuthology.task.internal: core looks like: /tmp/tmpf_20_tke: ELF 64-bit LSB core file, x86-64, version 1 (SYSV), SVR4-style, from 'ceph_test_rados_api_aio_pp', real uid: 1000, effective uid: 1000, real gid: 1316, effective gid: 1316, execfn: '/usr/bin/ceph_test_rados_api_aio_pp', platform: 'x86_64' 2024-08-02T23:45:32.253 INFO:teuthology.task.internal: dump_program: ceph_test_rados_api_aio_pp 2024-08-02T23:45:32.254 DEBUG:teuthology.orchestra.run.smithi073:> which ceph_test_rados_api_aio_pp 2024-08-02T23:45:32.373 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph_test_rados_api_aio_pp 2024-08-02T23:45:32.395 DEBUG:teuthology.orchestra.remote:smithi073:/usr/bin/ceph_test_rados_api_aio_pp is 1MB 2024-08-02T23:45:32.476 DEBUG:teuthology.orchestra.run.smithi073:> mktemp 2024-08-02T23:45:32.493 INFO:teuthology.orchestra.run.smithi073.stdout:/tmp/tmp.KO7Ab9xCQM 2024-08-02T23:45:32.494 DEBUG:teuthology.orchestra.run.smithi073:> sudo cp /usr/bin/ceph_test_rados_api_aio_pp.debug /tmp/tmp.KO7Ab9xCQM 2024-08-02T23:45:32.559 INFO:teuthology.orchestra.run.smithi073.stderr:cp: cannot stat '/usr/bin/ceph_test_rados_api_aio_pp.debug': No such file or directory 2024-08-02T23:45:32.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-02T23:45:32.561 ERROR:teuthology.run_tasks:Manager failed: internal.archive Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/teuthology/task/internal/__init__.py", line 420, in archive fetch_binaries_for_coredumps(path, rem) File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/teuthology/task/internal/__init__.py", line 363, in fetch_binaries_for_coredumps remote.get_file(debug_path, coredump_path) File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/teuthology/orchestra/remote.py", line 622, in get_file self.run(args=args) File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi073 with status 1: 'sudo cp /usr/bin/ceph_test_rados_api_aio_pp.debug /tmp/tmp.KO7Ab9xCQM' 2024-08-02T23:45:32.570 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-08-02T23:45:32.581 INFO:teuthology.task.internal:Not uploading archives. 2024-08-02T23:45:32.581 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-08-02T23:45:32.589 INFO:teuthology.task.internal:Tidying up after the test... 2024-08-02T23:45:32.590 DEBUG:teuthology.orchestra.run.smithi073:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-08-02T23:45:32.603 DEBUG:teuthology.orchestra.run.smithi131:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-08-02T23:45:32.617 INFO:teuthology.orchestra.run.smithi073.stdout: 265306 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Aug 2 23:45 /home/ubuntu/cephtest 2024-08-02T23:45:32.618 INFO:teuthology.orchestra.run.smithi073.stdout: 403394 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Aug 2 23:40 /home/ubuntu/cephtest/mnt.0 2024-08-02T23:45:32.618 INFO:teuthology.orchestra.run.smithi073.stdout: 403395 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Aug 2 23:40 /home/ubuntu/cephtest/mnt.0/client.0 2024-08-02T23:45:32.618 INFO:teuthology.orchestra.run.smithi073.stdout: 417889 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Aug 2 23:40 /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-08-02T23:45:32.618 INFO:teuthology.orchestra.run.smithi073.stdout: 265307 4 drwxr-xr-x 6 ubuntu ubuntu 4096 Aug 2 23:45 /home/ubuntu/cephtest/archive 2024-08-02T23:45:32.618 INFO:teuthology.orchestra.run.smithi073.stdout: 403393 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Aug 2 23:45 /home/ubuntu/cephtest/archive/audit 2024-08-02T23:45:32.618 INFO:teuthology.orchestra.run.smithi073.stdout: 403397 120 -rw------- 1 ubuntu ubuntu 121143 Aug 2 23:45 /home/ubuntu/cephtest/archive/audit/audit.log.gz 2024-08-02T23:45:32.619 INFO:teuthology.orchestra.run.smithi073.stdout: 265311 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Aug 2 23:45 /home/ubuntu/cephtest/archive/syslog 2024-08-02T23:45:32.619 INFO:teuthology.orchestra.run.smithi073.stdout: 262209 4 -rw-rw-rw- 1 ubuntu ubuntu 29 Aug 2 23:31 /home/ubuntu/cephtest/archive/syslog/misc.log.gz 2024-08-02T23:45:32.619 INFO:teuthology.orchestra.run.smithi073.stdout: 265312 184 -rw-r--r-- 1 ubuntu ubuntu 187659 Aug 2 23:45 /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-08-02T23:45:32.619 INFO:teuthology.orchestra.run.smithi073.stdout: 265313 4 -rw-rw-rw- 1 ubuntu ubuntu 29 Aug 2 23:31 /home/ubuntu/cephtest/archive/syslog/kern.log.gz 2024-08-02T23:45:32.619 INFO:teuthology.orchestra.run.smithi073.stdout: 265310 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Aug 2 23:44 /home/ubuntu/cephtest/archive/coredump 2024-08-02T23:45:32.619 INFO:teuthology.orchestra.run.smithi073.stdout: 265443 86740 -rw------- 1 ubuntu ubuntu 138678272 Aug 2 23:44 /home/ubuntu/cephtest/archive/coredump/1722642293.31410.core 2024-08-02T23:45:32.619 INFO:teuthology.orchestra.run.smithi073.stdout: 417890 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Aug 2 23:40 /home/ubuntu/cephtest/archive/unit_test_xml_report 2024-08-02T23:45:32.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-02T23:45:32.620 INFO:teuthology.orchestra.run.smithi073.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2024-08-02T23:45:32.621 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/teuthology/task/internal/__init__.py", line 48, in base yield File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/teuthology/task/internal/__init__.py", line 420, in archive fetch_binaries_for_coredumps(path, rem) File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/teuthology/task/internal/__init__.py", line 363, in fetch_binaries_for_coredumps remote.get_file(debug_path, coredump_path) File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/teuthology/orchestra/remote.py", line 622, in get_file self.run(args=args) File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi073 with status 1: 'sudo cp /usr/bin/ceph_test_rados_api_aio_pp.debug /tmp/tmp.KO7Ab9xCQM' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.10/contextlib.py", line 153, in __exit__ self.gen.throw(typ, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/teuthology/task/internal/__init__.py", line 53, in base run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi073 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2024-08-02T23:45:32.631 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-08-02T23:45:32.640 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-08-02T23:45:32.650 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: CommandFailedError: Command failed on smithi073 with status 1: 'sudo cp /usr/bin/ceph_test_rados_api_aio_pp.debug /tmp/tmp.KO7Ab9xCQM' 2024-08-02T23:45:32.656 INFO:teuthology.run:Summary data: description: crimson-rados/basic/{clusters/fixed-2 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/seastore tasks/rados_api_tests} duration: 813.3786659240723 failure_reason: 'Command failed (workunit test rados/test.sh) on smithi073 with status 134: ''mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=26c3fb8e197dcf7a49a54d1f4c8a7362ee35a8ea TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 CRIMSON_COMPAT=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh --crimson''' flavor: crimson owner: scheduled_yuriw@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=d16edf7c858f4f85a8823bcf33b92306 status: fail success: false 2024-08-02T23:45:32.656 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-08-02T23:45:32.663 INFO:teuthology.orchestra.run.smithi131.stdout: 266031 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Aug 2 23:45 /home/ubuntu/cephtest 2024-08-02T23:45:32.663 INFO:teuthology.orchestra.run.smithi131.stdout: 266032 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Aug 2 23:45 /home/ubuntu/cephtest/archive 2024-08-02T23:45:32.664 INFO:teuthology.orchestra.run.smithi131.stdout: 403393 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Aug 2 23:45 /home/ubuntu/cephtest/archive/audit 2024-08-02T23:45:32.664 INFO:teuthology.orchestra.run.smithi131.stdout: 403676 116 -rw------- 1 ubuntu ubuntu 116469 Aug 2 23:45 /home/ubuntu/cephtest/archive/audit/audit.log.gz 2024-08-02T23:45:32.664 INFO:teuthology.orchestra.run.smithi131.stdout: 266043 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Aug 2 23:45 /home/ubuntu/cephtest/archive/syslog 2024-08-02T23:45:32.664 INFO:teuthology.orchestra.run.smithi131.stdout: 262209 4 -rw-rw-rw- 1 ubuntu ubuntu 29 Aug 2 23:31 /home/ubuntu/cephtest/archive/syslog/misc.log.gz 2024-08-02T23:45:32.664 INFO:teuthology.orchestra.run.smithi131.stdout: 265393 184 -rw-r--r-- 1 ubuntu ubuntu 187547 Aug 2 23:45 /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-08-02T23:45:32.664 INFO:teuthology.orchestra.run.smithi131.stdout: 265392 4 -rw-rw-rw- 1 ubuntu ubuntu 29 Aug 2 23:31 /home/ubuntu/cephtest/archive/syslog/kern.log.gz 2024-08-02T23:45:32.665 INFO:teuthology.orchestra.run.smithi131.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2024-08-02T23:45:32.792 INFO:teuthology.run:FAIL