2024-09-16T07:43:02.606 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-16T07:43:02.622 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-16T07:43:02.715 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/matan-2024-09-16_07:15:55-crimson-rados-wip-matanb-crimson-testing-59543-distro-crimson-smithi/7906886 branch: wip-matanb-crimson-testing-59543 description: crimson-rados/basic/{clusters/fixed-2 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/seastore tasks/rados_api_tests} email: null first_in_suite: false job_id: '7906886' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: matan-2024-09-16_07:15:55-crimson-rados-wip-matanb-crimson-testing-59543-distro-crimson-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-matanb-crimson-testing-59543 ceph: conf: client: debug ms: 1 global: enable experimental unrecoverable data corrupting features: crimson ms client mode: crc ms cluster mode: crc ms mon client mode: crc ms mon cluster mode: crc ms mon service mode: crc ms service mode: crc mgr: debug mgr: 20 debug ms: 1 mgr stats period: 30 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon warn on pool no app: false osd pool default crimson: true osd: crimson alien thread cpu cores: 6-7 crimson osd obc lru size: 10 debug ms: 1 debug osd: 20 osd blocked scrub grace period: 3600 osd class default list: '*' osd class load list: '*' osd objectstore: seastore osd shutdown pgref assert: true osd.0: crimson seastar cpu cores: 0-2 osd.1: crimson seastar cpu cores: 3-5 osd.2: crimson seastar cpu cores: 0-2 osd.3: crimson seastar cpu cores: 3-5 flavor: crimson fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(POOL_APP_NOT_ENABLED\) - reached quota - but it is still running - overall HEALTH_ - \(POOL_FULL\) - \(SMALLER_PGP_NUM\) - \(CACHE_POOL_NO_HIT_SET\) - \(CACHE_POOL_NEAR_FULL\) - \(POOL_APP_NOT_ENABLED\) - \(PG_AVAILABILITY\) - \(PG_DEGRADED\) sha1: 81fedcb3b484c843d22ffbc8d179cb567b7083b5 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: crimson sha1: 81fedcb3b484c843d22ffbc8d179cb567b7083b5 selinux: allowlist: - scontext=system_u:system_r:getty_t:s0 workunit: branch: main env: CRIMSON_COMPAT: '1' sha1: 0325fd3c11a225c0fd9eb38fab88aad1370ad779 owner: scheduled_matan@teuthology priority: 60 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - osd.0 - osd.1 - client.0 - node-exporter.a - - mgr.x - osd.2 - osd.3 - client.1 - prometheus.a - node-exporter.b seed: 6651 sha1: 81fedcb3b484c843d22ffbc8d179cb567b7083b5 sleep_before_teardown: 0 suite: crimson-rados suite_branch: main suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_0325fd3c11a225c0fd9eb38fab88aad1370ad779/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 0325fd3c11a225c0fd9eb38fab88aad1370ad779 targets: smithi027.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLpcwPaHyzmk8hQe88qHeZTFsVJ0Y/iA/xfFISw1dCCxtKNeHjvgrr2HIKX/kdv3qi12ZBBt00FizK4uPoFcr2k= smithi187.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFvXrMa5rbcalzvURNr4bpBSSg2Hb6A1St9wR0SUv7oRgP1gtuEubBZCiD1S1b5rJOf7CrjqqbFJryMDNPVVifY= tasks: - install: null - ceph: conf: mon: mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 paxos service trim min: 10 osd: debug monc: 20 flavor: crimson - workunit: clients: client.0: - rados/test.sh --crimson - rados/test_pool_quota.sh teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-16_07:15:55 tube: smithi user: matan verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-09-16T07:43:02.715 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_0325fd3c11a225c0fd9eb38fab88aad1370ad779/qa; will attempt to use it 2024-09-16T07:43:02.716 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_0325fd3c11a225c0fd9eb38fab88aad1370ad779/qa/tasks 2024-09-16T07:43:02.716 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-16T07:43:02.718 INFO:teuthology.task.internal:Checking packages... 2024-09-16T07:43:02.741 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'crimson' and ceph hash '81fedcb3b484c843d22ffbc8d179cb567b7083b5' 2024-09-16T07:43:02.741 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-16T07:43:02.741 INFO:teuthology.packaging:ref: None 2024-09-16T07:43:02.741 INFO:teuthology.packaging:tag: None 2024-09-16T07:43:02.741 INFO:teuthology.packaging:branch: wip-matanb-crimson-testing-59543 2024-09-16T07:43:02.742 INFO:teuthology.packaging:sha1: 81fedcb3b484c843d22ffbc8d179cb567b7083b5 2024-09-16T07:43:02.742 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&ref=wip-matanb-crimson-testing-59543 2024-09-16T07:43:02.885 INFO:teuthology.task.internal:Found packages for ceph version 19.3.0-4879.g81fedcb3 2024-09-16T07:43:02.887 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-16T07:43:02.893 INFO:teuthology.task.internal:no buildpackages task found 2024-09-16T07:43:02.893 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-16T07:43:02.924 INFO:teuthology.task.internal:Saving configuration 2024-09-16T07:43:02.944 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-16T07:43:02.958 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-16T07:43:02.983 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi027.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/matan-2024-09-16_07:15:55-crimson-rados-wip-matanb-crimson-testing-59543-distro-crimson-smithi/7906886', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-16 07:37:01.421362', 'locked_by': 'scheduled_matan@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLpcwPaHyzmk8hQe88qHeZTFsVJ0Y/iA/xfFISw1dCCxtKNeHjvgrr2HIKX/kdv3qi12ZBBt00FizK4uPoFcr2k='} 2024-09-16T07:43:02.999 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi187.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/matan-2024-09-16_07:15:55-crimson-rados-wip-matanb-crimson-testing-59543-distro-crimson-smithi/7906886', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-16 07:37:01.423922', 'locked_by': 'scheduled_matan@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFvXrMa5rbcalzvURNr4bpBSSg2Hb6A1St9wR0SUv7oRgP1gtuEubBZCiD1S1b5rJOf7CrjqqbFJryMDNPVVifY='} 2024-09-16T07:43:02.999 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-16T07:43:03.005 INFO:teuthology.task.internal:roles: ubuntu@smithi027.front.sepia.ceph.com - ['mon.a', 'osd.0', 'osd.1', 'client.0', 'node-exporter.a'] 2024-09-16T07:43:03.005 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['mgr.x', 'osd.2', 'osd.3', 'client.1', 'prometheus.a', 'node-exporter.b'] 2024-09-16T07:43:03.005 INFO:teuthology.run_tasks:Running task console_log... 2024-09-16T07:43:03.113 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fa6851a3be0>, signals=[15]) 2024-09-16T07:43:03.113 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-16T07:43:03.119 INFO:teuthology.task.internal:Opening connections... 2024-09-16T07:43:03.119 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi027.front.sepia.ceph.com 2024-09-16T07:43:03.121 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T07:43:03.196 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi187.front.sepia.ceph.com 2024-09-16T07:43:03.197 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi187.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T07:43:03.272 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-16T07:43:03.278 DEBUG:teuthology.orchestra.run.smithi027:> uname -m 2024-09-16T07:43:03.297 INFO:teuthology.orchestra.run.smithi027.stdout:x86_64 2024-09-16T07:43:03.297 DEBUG:teuthology.orchestra.run.smithi027:> cat /etc/os-release 2024-09-16T07:43:03.356 INFO:teuthology.orchestra.run.smithi027.stdout:NAME="CentOS Stream" 2024-09-16T07:43:03.356 INFO:teuthology.orchestra.run.smithi027.stdout:VERSION="9" 2024-09-16T07:43:03.356 INFO:teuthology.orchestra.run.smithi027.stdout:ID="centos" 2024-09-16T07:43:03.356 INFO:teuthology.orchestra.run.smithi027.stdout:ID_LIKE="rhel fedora" 2024-09-16T07:43:03.357 INFO:teuthology.orchestra.run.smithi027.stdout:VERSION_ID="9" 2024-09-16T07:43:03.357 INFO:teuthology.orchestra.run.smithi027.stdout:PLATFORM_ID="platform:el9" 2024-09-16T07:43:03.357 INFO:teuthology.orchestra.run.smithi027.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-16T07:43:03.357 INFO:teuthology.orchestra.run.smithi027.stdout:ANSI_COLOR="0;31" 2024-09-16T07:43:03.357 INFO:teuthology.orchestra.run.smithi027.stdout:LOGO="fedora-logo-icon" 2024-09-16T07:43:03.357 INFO:teuthology.orchestra.run.smithi027.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-16T07:43:03.357 INFO:teuthology.orchestra.run.smithi027.stdout:HOME_URL="https://centos.org/" 2024-09-16T07:43:03.357 INFO:teuthology.orchestra.run.smithi027.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-16T07:43:03.357 INFO:teuthology.orchestra.run.smithi027.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-16T07:43:03.357 INFO:teuthology.orchestra.run.smithi027.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-16T07:43:03.358 INFO:teuthology.lock.ops:Updating smithi027.front.sepia.ceph.com on lock server 2024-09-16T07:43:03.380 DEBUG:teuthology.orchestra.run.smithi187:> uname -m 2024-09-16T07:43:03.398 INFO:teuthology.orchestra.run.smithi187.stdout:x86_64 2024-09-16T07:43:03.398 DEBUG:teuthology.orchestra.run.smithi187:> cat /etc/os-release 2024-09-16T07:43:03.456 INFO:teuthology.orchestra.run.smithi187.stdout:NAME="CentOS Stream" 2024-09-16T07:43:03.456 INFO:teuthology.orchestra.run.smithi187.stdout:VERSION="9" 2024-09-16T07:43:03.456 INFO:teuthology.orchestra.run.smithi187.stdout:ID="centos" 2024-09-16T07:43:03.456 INFO:teuthology.orchestra.run.smithi187.stdout:ID_LIKE="rhel fedora" 2024-09-16T07:43:03.456 INFO:teuthology.orchestra.run.smithi187.stdout:VERSION_ID="9" 2024-09-16T07:43:03.456 INFO:teuthology.orchestra.run.smithi187.stdout:PLATFORM_ID="platform:el9" 2024-09-16T07:43:03.456 INFO:teuthology.orchestra.run.smithi187.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-16T07:43:03.456 INFO:teuthology.orchestra.run.smithi187.stdout:ANSI_COLOR="0;31" 2024-09-16T07:43:03.456 INFO:teuthology.orchestra.run.smithi187.stdout:LOGO="fedora-logo-icon" 2024-09-16T07:43:03.457 INFO:teuthology.orchestra.run.smithi187.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-16T07:43:03.457 INFO:teuthology.orchestra.run.smithi187.stdout:HOME_URL="https://centos.org/" 2024-09-16T07:43:03.457 INFO:teuthology.orchestra.run.smithi187.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-16T07:43:03.457 INFO:teuthology.orchestra.run.smithi187.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-16T07:43:03.457 INFO:teuthology.orchestra.run.smithi187.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-16T07:43:03.457 INFO:teuthology.lock.ops:Updating smithi187.front.sepia.ceph.com on lock server 2024-09-16T07:43:03.479 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-16T07:43:03.488 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-16T07:43:03.551 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-16T07:43:03.552 DEBUG:teuthology.orchestra.run.smithi027:> test '!' -e /home/ubuntu/cephtest 2024-09-16T07:43:03.554 DEBUG:teuthology.orchestra.run.smithi187:> test '!' -e /home/ubuntu/cephtest 2024-09-16T07:43:03.571 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-16T07:43:03.582 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-16T07:43:03.582 DEBUG:teuthology.orchestra.run.smithi027:> test -z $(ls -A /var/lib/ceph) 2024-09-16T07:43:03.612 DEBUG:teuthology.orchestra.run.smithi187:> test -z $(ls -A /var/lib/ceph) 2024-09-16T07:43:03.708 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-16T07:43:03.750 INFO:teuthology.run_tasks:Running task kernel... 2024-09-16T07:43:03.765 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-09-16T07:43:03.765 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mgr.x': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-09-16T07:43:03.765 DEBUG:teuthology.orchestra.run.smithi027:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-16T07:43:03.766 DEBUG:teuthology.orchestra.run.smithi187:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-16T07:43:03.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T07:43:03.784 DEBUG:teuthology.orchestra.run.smithi187:> uname -r 2024-09-16T07:43:03.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T07:43:03.785 DEBUG:teuthology.orchestra.run.smithi027:> uname -r 2024-09-16T07:43:03.841 INFO:teuthology.orchestra.run.smithi187.stdout:5.14.0-503.el9.x86_64 2024-09-16T07:43:03.841 INFO:teuthology.task.kernel:Running kernel on smithi187: 5.14.0-503.el9.x86_64 2024-09-16T07:43:03.842 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum install -y kernel 2024-09-16T07:43:03.844 INFO:teuthology.orchestra.run.smithi027.stdout:5.14.0-503.el9.x86_64 2024-09-16T07:43:03.844 INFO:teuthology.task.kernel:Running kernel on smithi027: 5.14.0-503.el9.x86_64 2024-09-16T07:43:03.844 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum install -y kernel 2024-09-16T07:43:07.069 INFO:teuthology.orchestra.run.smithi027.stdout:CentOS Stream 9 - BaseOS 5.6 MB/s | 8.2 MB 00:01 2024-09-16T07:43:07.587 INFO:teuthology.orchestra.run.smithi187.stdout:CentOS Stream 9 - BaseOS 4.2 MB/s | 8.2 MB 00:01 2024-09-16T07:43:10.346 INFO:teuthology.orchestra.run.smithi187.stdout:CentOS Stream 9 - AppStream 12 MB/s | 20 MB 00:01 2024-09-16T07:43:11.546 INFO:teuthology.orchestra.run.smithi027.stdout:CentOS Stream 9 - AppStream 6.1 MB/s | 20 MB 00:03 2024-09-16T07:43:15.730 INFO:teuthology.orchestra.run.smithi187.stdout:CentOS Stream 9 - CRB 4.9 MB/s | 6.5 MB 00:01 2024-09-16T07:43:16.780 INFO:teuthology.orchestra.run.smithi027.stdout:CentOS Stream 9 - CRB 5.0 MB/s | 6.5 MB 00:01 2024-09-16T07:43:18.206 INFO:teuthology.orchestra.run.smithi187.stdout:CentOS Stream 9 - Extras packages 15 kB/s | 19 kB 00:01 2024-09-16T07:43:18.255 INFO:teuthology.orchestra.run.smithi027.stdout:CentOS Stream 9 - Extras packages 88 kB/s | 19 kB 00:00 2024-09-16T07:43:19.451 INFO:teuthology.orchestra.run.smithi187.stdout:Extra Packages for Enterprise Linux 20 MB/s | 23 MB 00:01 2024-09-16T07:43:19.476 INFO:teuthology.orchestra.run.smithi027.stdout:Extra Packages for Enterprise Linux 21 MB/s | 23 MB 00:01 2024-09-16T07:43:24.940 INFO:teuthology.orchestra.run.smithi187.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-09-16T07:43:25.215 INFO:teuthology.orchestra.run.smithi027.stdout:lab-extras 37 kB/s | 1.7 kB 00:00 2024-09-16T07:43:26.955 INFO:teuthology.orchestra.run.smithi187.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-16T07:43:26.955 INFO:teuthology.orchestra.run.smithi187.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-16T07:43:27.050 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2024-09-16T07:43:27.056 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-09-16T07:43:27.056 INFO:teuthology.orchestra.run.smithi187.stdout: Package Architecture Version Repository Size 2024-09-16T07:43:27.056 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-09-16T07:43:27.056 INFO:teuthology.orchestra.run.smithi187.stdout:Installing: 2024-09-16T07:43:27.056 INFO:teuthology.orchestra.run.smithi187.stdout: kernel x86_64 5.14.0-505.el9 baseos 23 k 2024-09-16T07:43:27.057 INFO:teuthology.orchestra.run.smithi187.stdout:Installing dependencies: 2024-09-16T07:43:27.057 INFO:teuthology.orchestra.run.smithi187.stdout: kernel-core x86_64 5.14.0-505.el9 baseos 16 M 2024-09-16T07:43:27.057 INFO:teuthology.orchestra.run.smithi187.stdout: kernel-modules x86_64 5.14.0-505.el9 baseos 35 M 2024-09-16T07:43:27.057 INFO:teuthology.orchestra.run.smithi187.stdout: kernel-modules-core x86_64 5.14.0-505.el9 baseos 29 M 2024-09-16T07:43:27.057 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-16T07:43:27.057 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction Summary 2024-09-16T07:43:27.058 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-09-16T07:43:27.058 INFO:teuthology.orchestra.run.smithi187.stdout:Install 4 Packages 2024-09-16T07:43:27.058 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-16T07:43:27.058 INFO:teuthology.orchestra.run.smithi187.stdout:Total download size: 79 M 2024-09-16T07:43:27.058 INFO:teuthology.orchestra.run.smithi187.stdout:Installed size: 126 M 2024-09-16T07:43:27.058 INFO:teuthology.orchestra.run.smithi187.stdout:Downloading Packages: 2024-09-16T07:43:27.289 INFO:teuthology.orchestra.run.smithi027.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-16T07:43:27.289 INFO:teuthology.orchestra.run.smithi027.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-16T07:43:27.383 INFO:teuthology.orchestra.run.smithi187.stdout:(1/4): kernel-5.14.0-505.el9.x86_64.rpm 329 kB/s | 23 kB 00:00 2024-09-16T07:43:27.387 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-09-16T07:43:27.394 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-09-16T07:43:27.394 INFO:teuthology.orchestra.run.smithi027.stdout: Package Architecture Version Repository Size 2024-09-16T07:43:27.394 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-09-16T07:43:27.394 INFO:teuthology.orchestra.run.smithi027.stdout:Installing: 2024-09-16T07:43:27.394 INFO:teuthology.orchestra.run.smithi027.stdout: kernel x86_64 5.14.0-505.el9 baseos 23 k 2024-09-16T07:43:27.394 INFO:teuthology.orchestra.run.smithi027.stdout:Installing dependencies: 2024-09-16T07:43:27.395 INFO:teuthology.orchestra.run.smithi027.stdout: kernel-core x86_64 5.14.0-505.el9 baseos 16 M 2024-09-16T07:43:27.395 INFO:teuthology.orchestra.run.smithi027.stdout: kernel-modules x86_64 5.14.0-505.el9 baseos 35 M 2024-09-16T07:43:27.395 INFO:teuthology.orchestra.run.smithi027.stdout: kernel-modules-core x86_64 5.14.0-505.el9 baseos 29 M 2024-09-16T07:43:27.395 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-16T07:43:27.395 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2024-09-16T07:43:27.395 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-09-16T07:43:27.395 INFO:teuthology.orchestra.run.smithi027.stdout:Install 4 Packages 2024-09-16T07:43:27.395 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-16T07:43:27.396 INFO:teuthology.orchestra.run.smithi027.stdout:Total download size: 79 M 2024-09-16T07:43:27.396 INFO:teuthology.orchestra.run.smithi027.stdout:Installed size: 126 M 2024-09-16T07:43:27.396 INFO:teuthology.orchestra.run.smithi027.stdout:Downloading Packages: 2024-09-16T07:43:27.542 INFO:teuthology.orchestra.run.smithi027.stdout:(1/4): kernel-5.14.0-505.el9.x86_64.rpm 223 kB/s | 23 kB 00:00 2024-09-16T07:43:27.950 INFO:teuthology.orchestra.run.smithi027.stdout:(2/4): kernel-core-5.14.0-505.el9.x86_64.rpm 31 MB/s | 16 MB 00:00 2024-09-16T07:43:28.359 INFO:teuthology.orchestra.run.smithi027.stdout:(3/4): kernel-modules-core-5.14.0-505.el9.x86_6 35 MB/s | 29 MB 00:00 2024-09-16T07:43:28.668 INFO:teuthology.orchestra.run.smithi027.stdout:(4/4): kernel-modules-5.14.0-505.el9.x86_64.rpm 28 MB/s | 35 MB 00:01 2024-09-16T07:43:28.668 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2024-09-16T07:43:28.668 INFO:teuthology.orchestra.run.smithi027.stdout:Total 62 MB/s | 79 MB 00:01 2024-09-16T07:43:29.383 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction check 2024-09-16T07:43:29.425 INFO:teuthology.orchestra.run.smithi187.stdout:(2/4): kernel-core-5.14.0-505.el9.x86_64.rpm 7.4 MB/s | 16 MB 00:02 2024-09-16T07:43:29.789 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction check succeeded. 2024-09-16T07:43:29.789 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction test 2024-09-16T07:43:30.391 INFO:teuthology.orchestra.run.smithi187.stdout:(3/4): kernel-modules-5.14.0-505.el9.x86_64.rpm 11 MB/s | 35 MB 00:03 2024-09-16T07:43:30.784 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction test succeeded. 2024-09-16T07:43:30.784 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction 2024-09-16T07:43:33.406 INFO:teuthology.orchestra.run.smithi027.stdout: Preparing : 1/1 2024-09-16T07:43:34.089 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : kernel-modules-core-5.14.0-505.el9.x86_64 1/4 2024-09-16T07:43:34.179 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-16T07:43:35.891 INFO:teuthology.orchestra.run.smithi187.stdout:(4/4): kernel-modules-core-5.14.0-505.el9.x86_6 3.4 MB/s | 29 MB 00:08 2024-09-16T07:43:35.892 INFO:teuthology.orchestra.run.smithi187.stdout:-------------------------------------------------------------------------------- 2024-09-16T07:43:35.892 INFO:teuthology.orchestra.run.smithi187.stdout:Total 8.9 MB/s | 79 MB 00:08 2024-09-16T07:43:36.573 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction check 2024-09-16T07:43:36.921 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction check succeeded. 2024-09-16T07:43:36.921 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction test 2024-09-16T07:43:37.113 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-16T07:43:37.239 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-16T07:43:37.895 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction test succeeded. 2024-09-16T07:43:37.896 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction 2024-09-16T07:43:40.357 INFO:teuthology.orchestra.run.smithi187.stdout: Preparing : 1/1 2024-09-16T07:43:41.026 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : kernel-modules-core-5.14.0-505.el9.x86_64 1/4 2024-09-16T07:43:41.099 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-16T07:43:42.606 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-16T07:43:42.648 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : kernel-5.14.0-505.el9.x86_64 4/4 2024-09-16T07:43:43.872 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-16T07:43:44.009 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-16T07:43:47.885 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-16T07:43:48.858 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-16T07:43:48.967 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : kernel-5.14.0-505.el9.x86_64 4/4 2024-09-16T07:43:53.842 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-16T07:44:27.696 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 4/4 2024-09-16T07:44:27.738 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 4/4 2024-09-16T07:44:28.129 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: kernel-5.14.0-505.el9.x86_64 4/4 2024-09-16T07:44:28.129 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : kernel-5.14.0-505.el9.x86_64 1/4 2024-09-16T07:44:28.129 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-16T07:44:28.129 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-16T07:44:28.941 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-16T07:44:28.941 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-16T07:44:28.941 INFO:teuthology.orchestra.run.smithi027.stdout:Installed: 2024-09-16T07:44:28.942 INFO:teuthology.orchestra.run.smithi027.stdout: kernel-5.14.0-505.el9.x86_64 2024-09-16T07:44:28.942 INFO:teuthology.orchestra.run.smithi027.stdout: kernel-core-5.14.0-505.el9.x86_64 2024-09-16T07:44:28.942 INFO:teuthology.orchestra.run.smithi027.stdout: kernel-modules-5.14.0-505.el9.x86_64 2024-09-16T07:44:28.942 INFO:teuthology.orchestra.run.smithi027.stdout: kernel-modules-core-5.14.0-505.el9.x86_64 2024-09-16T07:44:28.942 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-16T07:44:28.942 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-09-16T07:44:29.331 DEBUG:teuthology.orchestra.run.smithi027:> rpm -q kernel | sort -rV | head -n 1 2024-09-16T07:44:29.411 INFO:teuthology.orchestra.run.smithi027.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-16T07:44:29.411 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-16T07:44:29.411 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-505.el9.x86_64 2024-09-16T07:44:29.411 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-16T07:44:29.412 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-16T07:44:29.412 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-09-16T07:44:29.412 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi027.front.sepia.ceph.com, path=None, version=distro) 2024-09-16T07:44:29.412 DEBUG:teuthology.orchestra.run.smithi027:> rpm -q kernel | sort -rV | head -n 1 2024-09-16T07:44:29.472 INFO:teuthology.orchestra.run.smithi027.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-16T07:44:29.472 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-16T07:44:29.473 DEBUG:teuthology.orchestra.run.smithi027:> sudo rpm -qi grub2-tools 2024-09-16T07:44:29.506 INFO:teuthology.orchestra.run.smithi027.stdout:Name : grub2-tools 2024-09-16T07:44:29.506 INFO:teuthology.orchestra.run.smithi027.stdout:Epoch : 1 2024-09-16T07:44:29.506 INFO:teuthology.orchestra.run.smithi027.stdout:Version : 2.06 2024-09-16T07:44:29.506 INFO:teuthology.orchestra.run.smithi027.stdout:Release : 82.el9 2024-09-16T07:44:29.506 INFO:teuthology.orchestra.run.smithi027.stdout:Architecture: x86_64 2024-09-16T07:44:29.506 INFO:teuthology.orchestra.run.smithi027.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-16T07:44:29.506 INFO:teuthology.orchestra.run.smithi027.stdout:Group : Unspecified 2024-09-16T07:44:29.506 INFO:teuthology.orchestra.run.smithi027.stdout:Size : 8274670 2024-09-16T07:44:29.506 INFO:teuthology.orchestra.run.smithi027.stdout:License : GPLv3+ 2024-09-16T07:44:29.506 INFO:teuthology.orchestra.run.smithi027.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-16T07:44:29.506 INFO:teuthology.orchestra.run.smithi027.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-16T07:44:29.507 INFO:teuthology.orchestra.run.smithi027.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-16T07:44:29.507 INFO:teuthology.orchestra.run.smithi027.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-16T07:44:29.507 INFO:teuthology.orchestra.run.smithi027.stdout:Packager : builder@centos.org 2024-09-16T07:44:29.507 INFO:teuthology.orchestra.run.smithi027.stdout:Vendor : CentOS 2024-09-16T07:44:29.507 INFO:teuthology.orchestra.run.smithi027.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-16T07:44:29.507 INFO:teuthology.orchestra.run.smithi027.stdout:Summary : Support tools for GRUB. 2024-09-16T07:44:29.507 INFO:teuthology.orchestra.run.smithi027.stdout:Description : 2024-09-16T07:44:29.507 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-16T07:44:29.507 INFO:teuthology.orchestra.run.smithi027.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-16T07:44:29.507 INFO:teuthology.orchestra.run.smithi027.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-16T07:44:29.508 INFO:teuthology.orchestra.run.smithi027.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-16T07:44:29.508 INFO:teuthology.orchestra.run.smithi027.stdout:hardware devices. 2024-09-16T07:44:29.508 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-16T07:44:29.508 INFO:teuthology.orchestra.run.smithi027.stdout:This subpackage provides tools for support of all platforms. 2024-09-16T07:44:29.509 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-16T07:44:29.509 INFO:teuthology.task.kernel:Updating grub on smithi027 to boot 5.14.0-505.el9.x86_64 2024-09-16T07:44:29.509 DEBUG:teuthology.orchestra.run.smithi027:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-16T07:44:30.197 INFO:teuthology.orchestra.run.smithi027.stderr:Generating grub configuration file ... 2024-09-16T07:44:31.580 INFO:teuthology.orchestra.run.smithi027.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-16T07:44:31.595 INFO:teuthology.orchestra.run.smithi027.stderr:done 2024-09-16T07:44:31.597 DEBUG:teuthology.orchestra.run.smithi027:> mktemp 2024-09-16T07:44:31.614 INFO:teuthology.orchestra.run.smithi027.stdout:/tmp/tmp.GDQk1R4TDB 2024-09-16T07:44:31.614 DEBUG:teuthology.orchestra.run.smithi027:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.GDQk1R4TDB 2024-09-16T07:44:31.682 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod 0666 /tmp/tmp.GDQk1R4TDB 2024-09-16T07:44:31.826 DEBUG:teuthology.orchestra.remote:smithi027:/tmp/tmp.GDQk1R4TDB is 6KB 2024-09-16T07:44:31.876 DEBUG:teuthology.orchestra.run.smithi027:> rm -fr /tmp/tmp.GDQk1R4TDB 2024-09-16T07:44:31.891 DEBUG:teuthology.orchestra.run.smithi027:> sudo /bin/ls /boot/loader/entries || true 2024-09-16T07:44:31.961 INFO:teuthology.orchestra.run.smithi027.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-16T07:44:31.961 INFO:teuthology.orchestra.run.smithi027.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-16T07:44:31.962 INFO:teuthology.orchestra.run.smithi027.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-16T07:44:31.962 INFO:teuthology.orchestra.run.smithi027.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64.conf 2024-09-16T07:44:31.963 DEBUG:teuthology.orchestra.run.smithi027:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64 2024-09-16T07:44:31.980 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 4/4 2024-09-16T07:44:32.019 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 4/4 2024-09-16T07:44:32.124 DEBUG:teuthology.orchestra.run.smithi027:> sudo shutdown -r now 2024-09-16T07:44:32.365 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: kernel-5.14.0-505.el9.x86_64 4/4 2024-09-16T07:44:32.365 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : kernel-5.14.0-505.el9.x86_64 1/4 2024-09-16T07:44:32.365 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-16T07:44:32.365 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-16T07:44:33.243 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-16T07:44:33.243 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-16T07:44:33.243 INFO:teuthology.orchestra.run.smithi187.stdout:Installed: 2024-09-16T07:44:33.243 INFO:teuthology.orchestra.run.smithi187.stdout: kernel-5.14.0-505.el9.x86_64 2024-09-16T07:44:33.243 INFO:teuthology.orchestra.run.smithi187.stdout: kernel-core-5.14.0-505.el9.x86_64 2024-09-16T07:44:33.243 INFO:teuthology.orchestra.run.smithi187.stdout: kernel-modules-5.14.0-505.el9.x86_64 2024-09-16T07:44:33.243 INFO:teuthology.orchestra.run.smithi187.stdout: kernel-modules-core-5.14.0-505.el9.x86_64 2024-09-16T07:44:33.243 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-16T07:44:33.243 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2024-09-16T07:44:33.573 DEBUG:teuthology.orchestra.run.smithi187:> rpm -q kernel | sort -rV | head -n 1 2024-09-16T07:44:33.708 INFO:teuthology.orchestra.run.smithi187.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-16T07:44:33.708 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-16T07:44:33.708 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-505.el9.x86_64 2024-09-16T07:44:33.708 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-16T07:44:33.708 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-16T07:44:33.709 INFO:teuthology.task.kernel:Installing distro kernel on mgr.x... 2024-09-16T07:44:33.709 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi187.front.sepia.ceph.com, path=None, version=distro) 2024-09-16T07:44:33.709 DEBUG:teuthology.orchestra.run.smithi187:> rpm -q kernel | sort -rV | head -n 1 2024-09-16T07:44:33.768 INFO:teuthology.orchestra.run.smithi187.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-16T07:44:33.769 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-16T07:44:33.769 DEBUG:teuthology.orchestra.run.smithi187:> sudo rpm -qi grub2-tools 2024-09-16T07:44:33.801 INFO:teuthology.orchestra.run.smithi187.stdout:Name : grub2-tools 2024-09-16T07:44:33.801 INFO:teuthology.orchestra.run.smithi187.stdout:Epoch : 1 2024-09-16T07:44:33.801 INFO:teuthology.orchestra.run.smithi187.stdout:Version : 2.06 2024-09-16T07:44:33.801 INFO:teuthology.orchestra.run.smithi187.stdout:Release : 82.el9 2024-09-16T07:44:33.801 INFO:teuthology.orchestra.run.smithi187.stdout:Architecture: x86_64 2024-09-16T07:44:33.801 INFO:teuthology.orchestra.run.smithi187.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-16T07:44:33.801 INFO:teuthology.orchestra.run.smithi187.stdout:Group : Unspecified 2024-09-16T07:44:33.801 INFO:teuthology.orchestra.run.smithi187.stdout:Size : 8274670 2024-09-16T07:44:33.801 INFO:teuthology.orchestra.run.smithi187.stdout:License : GPLv3+ 2024-09-16T07:44:33.802 INFO:teuthology.orchestra.run.smithi187.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-16T07:44:33.802 INFO:teuthology.orchestra.run.smithi187.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-16T07:44:33.802 INFO:teuthology.orchestra.run.smithi187.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-16T07:44:33.802 INFO:teuthology.orchestra.run.smithi187.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-16T07:44:33.802 INFO:teuthology.orchestra.run.smithi187.stdout:Packager : builder@centos.org 2024-09-16T07:44:33.802 INFO:teuthology.orchestra.run.smithi187.stdout:Vendor : CentOS 2024-09-16T07:44:33.802 INFO:teuthology.orchestra.run.smithi187.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-16T07:44:33.802 INFO:teuthology.orchestra.run.smithi187.stdout:Summary : Support tools for GRUB. 2024-09-16T07:44:33.802 INFO:teuthology.orchestra.run.smithi187.stdout:Description : 2024-09-16T07:44:33.802 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-16T07:44:33.802 INFO:teuthology.orchestra.run.smithi187.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-16T07:44:33.802 INFO:teuthology.orchestra.run.smithi187.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-16T07:44:33.802 INFO:teuthology.orchestra.run.smithi187.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-16T07:44:33.802 INFO:teuthology.orchestra.run.smithi187.stdout:hardware devices. 2024-09-16T07:44:33.803 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-16T07:44:33.803 INFO:teuthology.orchestra.run.smithi187.stdout:This subpackage provides tools for support of all platforms. 2024-09-16T07:44:33.803 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-16T07:44:33.803 INFO:teuthology.task.kernel:Updating grub on smithi187 to boot 5.14.0-505.el9.x86_64 2024-09-16T07:44:33.804 DEBUG:teuthology.orchestra.run.smithi187:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-16T07:44:34.438 INFO:teuthology.orchestra.run.smithi187.stderr:Generating grub configuration file ... 2024-09-16T07:44:35.784 INFO:teuthology.orchestra.run.smithi187.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-16T07:44:35.815 INFO:teuthology.orchestra.run.smithi187.stderr:done 2024-09-16T07:44:35.817 DEBUG:teuthology.orchestra.run.smithi187:> mktemp 2024-09-16T07:44:35.833 INFO:teuthology.orchestra.run.smithi187.stdout:/tmp/tmp.s3nxsCU4uU 2024-09-16T07:44:35.833 DEBUG:teuthology.orchestra.run.smithi187:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.s3nxsCU4uU 2024-09-16T07:44:35.901 DEBUG:teuthology.orchestra.run.smithi187:> sudo chmod 0666 /tmp/tmp.s3nxsCU4uU 2024-09-16T07:44:36.051 DEBUG:teuthology.orchestra.remote:smithi187:/tmp/tmp.s3nxsCU4uU is 6KB 2024-09-16T07:44:36.099 DEBUG:teuthology.orchestra.run.smithi187:> rm -fr /tmp/tmp.s3nxsCU4uU 2024-09-16T07:44:36.114 DEBUG:teuthology.orchestra.run.smithi187:> sudo /bin/ls /boot/loader/entries || true 2024-09-16T07:44:36.177 INFO:teuthology.orchestra.run.smithi187.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-16T07:44:36.177 INFO:teuthology.orchestra.run.smithi187.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-16T07:44:36.177 INFO:teuthology.orchestra.run.smithi187.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-16T07:44:36.177 INFO:teuthology.orchestra.run.smithi187.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64.conf 2024-09-16T07:44:36.178 DEBUG:teuthology.orchestra.run.smithi187:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64 2024-09-16T07:44:36.311 DEBUG:teuthology.orchestra.run.smithi187:> sudo shutdown -r now 2024-09-16T07:45:02.149 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-09-16T07:45:02.150 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-09-16T07:45:02.151 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T07:45:06.318 INFO:teuthology.task.kernel:Checking client mgr.x for new kernel version... 2024-09-16T07:45:06.318 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2024-09-16T07:45:06.318 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi187.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T07:45:36.829 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.27 2024-09-16T07:45:45.837 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-09-16T07:45:45.838 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T07:45:48.897 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.27 2024-09-16T07:46:00.909 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-09-16T07:46:00.909 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T07:46:03.965 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.27 2024-09-16T07:46:06.324 DEBUG:teuthology.orchestra.remote:timed out 2024-09-16T07:46:15.333 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2024-09-16T07:46:15.334 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi187.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T07:46:16.700 DEBUG:teuthology.orchestra.run.smithi187:> true 2024-09-16T07:46:17.206 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi187.front.sepia.ceph.com' 2024-09-16T07:46:17.206 INFO:teuthology.task.kernel:Checking kernel version of mgr.x, want "5.14.0-505.el9.x86_64"... 2024-09-16T07:46:17.207 DEBUG:teuthology.orchestra.run.smithi187:> uname -r 2024-09-16T07:46:17.221 INFO:teuthology.orchestra.run.smithi187.stdout:5.14.0-505.el9.x86_64 2024-09-16T07:46:17.222 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-505.el9.x86_64 vs 5.14.0-505.el9.x86_64 2024-09-16T07:46:17.222 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-16T07:46:17.222 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-16T07:46:18.223 INFO:teuthology.task.kernel:Enabling kdb on mgr.x... 2024-09-16T07:46:18.223 DEBUG:teuthology.orchestra.run.smithi187:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-16T07:46:18.347 INFO:teuthology.orchestra.run.smithi187.stdout:ttyS1 2024-09-16T07:46:18.372 DEBUG:teuthology.parallel:result is None 2024-09-16T07:46:18.966 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-09-16T07:46:18.966 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T07:46:37.501 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.27 2024-09-16T07:46:55.502 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-09-16T07:46:55.503 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T07:46:55.800 DEBUG:teuthology.orchestra.run.smithi027:> true 2024-09-16T07:46:56.444 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-09-16T07:46:56.444 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-505.el9.x86_64"... 2024-09-16T07:46:56.444 DEBUG:teuthology.orchestra.run.smithi027:> uname -r 2024-09-16T07:46:56.460 INFO:teuthology.orchestra.run.smithi027.stdout:5.14.0-505.el9.x86_64 2024-09-16T07:46:56.460 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-505.el9.x86_64 vs 5.14.0-505.el9.x86_64 2024-09-16T07:46:56.460 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-16T07:46:56.461 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-16T07:46:57.462 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-09-16T07:46:57.462 DEBUG:teuthology.orchestra.run.smithi027:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-16T07:46:57.530 INFO:teuthology.orchestra.run.smithi027.stdout:ttyS1 2024-09-16T07:46:57.557 DEBUG:teuthology.parallel:result is None 2024-09-16T07:46:57.558 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-16T07:46:57.565 INFO:teuthology.task.internal:Creating test directory... 2024-09-16T07:46:57.565 DEBUG:teuthology.orchestra.run.smithi027:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-16T07:46:57.568 DEBUG:teuthology.orchestra.run.smithi187:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-16T07:46:57.589 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-16T07:46:57.595 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-16T07:46:57.630 INFO:teuthology.task.internal:Creating archive directory... 2024-09-16T07:46:57.631 DEBUG:teuthology.orchestra.run.smithi027:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-16T07:46:57.633 DEBUG:teuthology.orchestra.run.smithi187:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-16T07:46:57.672 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-16T07:46:57.679 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-16T07:46:57.679 DEBUG:teuthology.orchestra.run.smithi027:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-16T07:46:57.707 DEBUG:teuthology.orchestra.run.smithi187:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-16T07:46:57.743 INFO:teuthology.orchestra.run.smithi027.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T07:46:57.746 INFO:teuthology.orchestra.run.smithi187.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T07:46:57.755 INFO:teuthology.orchestra.run.smithi027.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T07:46:57.757 INFO:teuthology.orchestra.run.smithi187.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T07:46:57.759 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-16T07:46:57.767 INFO:teuthology.task.internal:Configuring sudo... 2024-09-16T07:46:57.767 DEBUG:teuthology.orchestra.run.smithi027:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-16T07:46:57.800 DEBUG:teuthology.orchestra.run.smithi187:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-16T07:46:57.828 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-16T07:46:57.838 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-16T07:46:57.838 DEBUG:teuthology.orchestra.run.smithi027:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-16T07:46:57.868 DEBUG:teuthology.orchestra.run.smithi187:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-16T07:46:57.885 DEBUG:teuthology.orchestra.run.smithi027:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-16T07:46:57.967 DEBUG:teuthology.orchestra.run.smithi027:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-16T07:46:58.016 DEBUG:teuthology.orchestra.run.smithi027:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-16T07:46:58.103 DEBUG:teuthology.orchestra.run.smithi027:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-16T07:46:58.144 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-09-16T07:46:58.144 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-16T07:46:58.212 DEBUG:teuthology.orchestra.run.smithi187:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-16T07:46:58.256 DEBUG:teuthology.orchestra.run.smithi187:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-16T07:46:58.313 DEBUG:teuthology.orchestra.run.smithi187:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-16T07:46:58.396 DEBUG:teuthology.orchestra.run.smithi187:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-16T07:46:58.440 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-09-16T07:46:58.440 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-16T07:46:58.506 DEBUG:teuthology.orchestra.run.smithi027:> sudo service rsyslog restart 2024-09-16T07:46:58.509 DEBUG:teuthology.orchestra.run.smithi187:> sudo service rsyslog restart 2024-09-16T07:46:58.562 INFO:teuthology.orchestra.run.smithi027.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-16T07:46:58.590 INFO:teuthology.orchestra.run.smithi187.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-16T07:46:59.032 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-16T07:46:59.040 INFO:teuthology.task.internal:Starting timer... 2024-09-16T07:46:59.040 INFO:teuthology.run_tasks:Running task pcp... 2024-09-16T07:46:59.101 INFO:teuthology.run_tasks:Running task selinux... 2024-09-16T07:46:59.134 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:getty_t:s0']} 2024-09-16T07:46:59.134 DEBUG:teuthology.orchestra.run.smithi027:> sudo service auditd rotate 2024-09-16T07:46:59.224 INFO:teuthology.orchestra.run.smithi027.stdout:Rotating logs: 2024-09-16T07:46:59.226 DEBUG:teuthology.orchestra.run.smithi187:> sudo service auditd rotate 2024-09-16T07:46:59.288 INFO:teuthology.orchestra.run.smithi187.stdout:Rotating logs: 2024-09-16T07:46:59.290 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-16T07:46:59.290 DEBUG:teuthology.orchestra.run.smithi027:> /usr/sbin/getenforce 2024-09-16T07:46:59.315 INFO:teuthology.orchestra.run.smithi027.stdout:Permissive 2024-09-16T07:46:59.316 DEBUG:teuthology.orchestra.run.smithi187:> /usr/sbin/getenforce 2024-09-16T07:46:59.339 INFO:teuthology.orchestra.run.smithi187.stdout:Permissive 2024-09-16T07:46:59.339 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi027.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi187.front.sepia.ceph.com': 'permissive'} 2024-09-16T07:46:59.339 DEBUG:teuthology.orchestra.run.smithi027:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-16T07:46:59.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T07:46:59.385 DEBUG:teuthology.orchestra.run.smithi187:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-16T07:46:59.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T07:46:59.413 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-16T07:46:59.413 DEBUG:teuthology.orchestra.run.smithi027:> sudo /usr/sbin/setenforce permissive 2024-09-16T07:46:59.456 DEBUG:teuthology.orchestra.run.smithi187:> sudo /usr/sbin/setenforce permissive 2024-09-16T07:46:59.494 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-16T07:46:59.503 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-16T07:46:59.521 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-16T07:46:59.523 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi027.front.sepia.ceph.com,smithi187.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-16T07:50:46.668 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi027.front.sepia.ceph.com'), Remote(name='ubuntu@smithi187.front.sepia.ceph.com')] 2024-09-16T07:50:46.669 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-09-16T07:50:46.671 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T07:50:46.749 DEBUG:teuthology.orchestra.run.smithi027:> true 2024-09-16T07:50:46.783 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-09-16T07:50:46.784 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2024-09-16T07:50:46.784 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi187.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T07:50:46.859 DEBUG:teuthology.orchestra.run.smithi187:> true 2024-09-16T07:50:46.933 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi187.front.sepia.ceph.com' 2024-09-16T07:50:46.933 INFO:teuthology.run_tasks:Running task clock... 2024-09-16T07:50:46.943 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-16T07:50:46.944 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-16T07:50:46.944 DEBUG:teuthology.orchestra.run.smithi027:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-16T07:50:46.947 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-16T07:50:46.947 DEBUG:teuthology.orchestra.run.smithi187:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-16T07:50:46.975 INFO:teuthology.orchestra.run.smithi027.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-16T07:50:46.989 INFO:teuthology.orchestra.run.smithi027.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-16T07:50:47.003 INFO:teuthology.orchestra.run.smithi187.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-16T07:50:47.015 INFO:teuthology.orchestra.run.smithi027.stderr:sudo: ntpd: command not found 2024-09-16T07:50:47.018 INFO:teuthology.orchestra.run.smithi187.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-16T07:50:47.026 INFO:teuthology.orchestra.run.smithi027.stdout:506 Cannot talk to daemon 2024-09-16T07:50:47.042 INFO:teuthology.orchestra.run.smithi027.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-16T07:50:47.042 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: ntpd: command not found 2024-09-16T07:50:47.054 INFO:teuthology.orchestra.run.smithi187.stdout:506 Cannot talk to daemon 2024-09-16T07:50:47.056 INFO:teuthology.orchestra.run.smithi027.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-16T07:50:47.068 INFO:teuthology.orchestra.run.smithi187.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-16T07:50:47.082 INFO:teuthology.orchestra.run.smithi187.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-16T07:50:47.117 INFO:teuthology.orchestra.run.smithi027.stderr:bash: line 1: ntpq: command not found 2024-09-16T07:50:47.121 INFO:teuthology.orchestra.run.smithi027.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-16T07:50:47.121 INFO:teuthology.orchestra.run.smithi027.stdout:=============================================================================== 2024-09-16T07:50:47.121 INFO:teuthology.orchestra.run.smithi027.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T07:50:47.121 INFO:teuthology.orchestra.run.smithi027.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T07:50:47.121 INFO:teuthology.orchestra.run.smithi027.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T07:50:47.122 INFO:teuthology.orchestra.run.smithi027.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T07:50:47.141 INFO:teuthology.orchestra.run.smithi187.stderr:bash: line 1: ntpq: command not found 2024-09-16T07:50:47.144 INFO:teuthology.orchestra.run.smithi187.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-16T07:50:47.145 INFO:teuthology.orchestra.run.smithi187.stdout:=============================================================================== 2024-09-16T07:50:47.145 INFO:teuthology.orchestra.run.smithi187.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T07:50:47.145 INFO:teuthology.orchestra.run.smithi187.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T07:50:47.145 INFO:teuthology.orchestra.run.smithi187.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T07:50:47.145 INFO:teuthology.orchestra.run.smithi187.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T07:50:47.146 INFO:teuthology.run_tasks:Running task install... 2024-09-16T07:50:47.155 DEBUG:teuthology.task.install:project ceph 2024-09-16T07:50:47.155 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'crimson', 'sha1': '81fedcb3b484c843d22ffbc8d179cb567b7083b5'}} 2024-09-16T07:50:47.156 DEBUG:teuthology.task.install:config {'flavor': 'crimson', 'sha1': '81fedcb3b484c843d22ffbc8d179cb567b7083b5'} 2024-09-16T07:50:47.156 INFO:teuthology.task.install:Using flavor: crimson 2024-09-16T07:50:47.162 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-16T07:50:47.162 INFO:teuthology.task.install:extra packages: [] 2024-09-16T07:50:47.162 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'crimson', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '81fedcb3b484c843d22ffbc8d179cb567b7083b5', 'tag': None, 'wait_for_package': False} 2024-09-16T07:50:47.163 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=81fedcb3b484c843d22ffbc8d179cb567b7083b5 2024-09-16T07:50:47.165 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'crimson', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '81fedcb3b484c843d22ffbc8d179cb567b7083b5', 'tag': None, 'wait_for_package': False} 2024-09-16T07:50:47.166 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=81fedcb3b484c843d22ffbc8d179cb567b7083b5 2024-09-16T07:50:47.307 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-matanb-crimson-testing-59543/81fedcb3b484c843d22ffbc8d179cb567b7083b5/centos/9/flavors/crimson/ 2024-09-16T07:50:47.307 INFO:teuthology.task.install.rpm:Package version is 19.3.0-4879.g81fedcb3 2024-09-16T07:50:47.320 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-matanb-crimson-testing-59543/81fedcb3b484c843d22ffbc8d179cb567b7083b5/centos/9/flavors/crimson/ 2024-09-16T07:50:47.320 INFO:teuthology.task.install.rpm:Package version is 19.3.0-4879.g81fedcb3 2024-09-16T07:50:47.419 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-matanb-crimson-testing-59543/81fedcb3b484c843d22ffbc8d179cb567b7083b5/centos/9/flavors/crimson/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-matanb-crimson-testing-59543/81fedcb3b484c843d22ffbc8d179cb567b7083b5/centos/9/flavors/crimson/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-matanb-crimson-testing-59543/81fedcb3b484c843d22ffbc8d179cb567b7083b5/centos/9/flavors/crimson/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-16T07:50:47.420 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-09-16T07:50:47.420 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-16T07:50:47.432 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-matanb-crimson-testing-59543/81fedcb3b484c843d22ffbc8d179cb567b7083b5/centos/9/flavors/crimson/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-matanb-crimson-testing-59543/81fedcb3b484c843d22ffbc8d179cb567b7083b5/centos/9/flavors/crimson/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-matanb-crimson-testing-59543/81fedcb3b484c843d22ffbc8d179cb567b7083b5/centos/9/flavors/crimson/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-16T07:50:47.432 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-09-16T07:50:47.432 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-16T07:50:47.447 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-16T07:50:47.447 DEBUG:teuthology.orchestra.run.smithi027:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/81fedcb3b484c843d22ffbc8d179cb567b7083b5/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-16T07:50:47.458 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-16T07:50:47.458 DEBUG:teuthology.orchestra.run.smithi187:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/81fedcb3b484c843d22ffbc8d179cb567b7083b5/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-16T07:50:47.514 DEBUG:teuthology.orchestra.run.smithi027:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-16T07:50:47.523 DEBUG:teuthology.orchestra.run.smithi187:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-16T07:50:47.591 DEBUG:teuthology.orchestra.run.smithi027:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-16T07:50:47.599 DEBUG:teuthology.orchestra.run.smithi187:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-16T07:50:47.656 INFO:teuthology.orchestra.run.smithi027.stdout:check_obsoletes = 1 2024-09-16T07:50:47.658 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum clean all 2024-09-16T07:50:47.663 INFO:teuthology.orchestra.run.smithi187.stdout:check_obsoletes = 1 2024-09-16T07:50:47.665 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum clean all 2024-09-16T07:50:47.889 INFO:teuthology.orchestra.run.smithi187.stdout:45 files removed 2024-09-16T07:50:47.898 INFO:teuthology.orchestra.run.smithi027.stdout:45 files removed 2024-09-16T07:50:47.916 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum -y install ceph-radosgw 2024-09-16T07:50:47.926 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install ceph-radosgw 2024-09-16T07:50:48.526 INFO:teuthology.orchestra.run.smithi187.stdout:ceph packages for x86_64 274 kB/s | 88 kB 00:00 2024-09-16T07:50:48.567 INFO:teuthology.orchestra.run.smithi027.stdout:ceph packages for x86_64 258 kB/s | 88 kB 00:00 2024-09-16T07:50:48.818 INFO:teuthology.orchestra.run.smithi187.stdout:ceph noarch packages 97 kB/s | 24 kB 00:00 2024-09-16T07:50:48.859 INFO:teuthology.orchestra.run.smithi027.stdout:ceph noarch packages 98 kB/s | 24 kB 00:00 2024-09-16T07:50:49.102 INFO:teuthology.orchestra.run.smithi187.stdout:ceph source packages 8.4 kB/s | 2.1 kB 00:00 2024-09-16T07:50:49.143 INFO:teuthology.orchestra.run.smithi027.stdout:ceph source packages 8.4 kB/s | 2.1 kB 00:00 2024-09-16T07:50:49.619 INFO:teuthology.orchestra.run.smithi187.stdout:CentOS Stream 9 - BaseOS 17 MB/s | 8.2 MB 00:00 2024-09-16T07:50:50.777 INFO:teuthology.orchestra.run.smithi027.stdout:CentOS Stream 9 - BaseOS 5.2 MB/s | 8.2 MB 00:01 2024-09-16T07:50:55.361 INFO:teuthology.orchestra.run.smithi187.stdout:CentOS Stream 9 - AppStream 4.4 MB/s | 20 MB 00:04 2024-09-16T07:50:59.102 INFO:teuthology.orchestra.run.smithi027.stdout:CentOS Stream 9 - AppStream 2.8 MB/s | 20 MB 00:07 2024-09-16T07:51:01.336 INFO:teuthology.orchestra.run.smithi187.stdout:CentOS Stream 9 - CRB 3.5 MB/s | 6.5 MB 00:01 2024-09-16T07:51:02.979 INFO:teuthology.orchestra.run.smithi187.stdout:CentOS Stream 9 - Extras packages 61 kB/s | 19 kB 00:00 2024-09-16T07:51:03.908 INFO:teuthology.orchestra.run.smithi187.stdout:Extra Packages for Enterprise Linux 28 MB/s | 23 MB 00:00 2024-09-16T07:51:04.578 INFO:teuthology.orchestra.run.smithi027.stdout:CentOS Stream 9 - CRB 4.3 MB/s | 6.5 MB 00:01 2024-09-16T07:51:06.037 INFO:teuthology.orchestra.run.smithi027.stdout:CentOS Stream 9 - Extras packages 97 kB/s | 19 kB 00:00 2024-09-16T07:51:06.892 INFO:teuthology.orchestra.run.smithi027.stdout:Extra Packages for Enterprise Linux 31 MB/s | 23 MB 00:00 2024-09-16T07:51:09.422 INFO:teuthology.orchestra.run.smithi187.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-09-16T07:51:11.781 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2024-09-16T07:51:11.804 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-09-16T07:51:11.804 INFO:teuthology.orchestra.run.smithi187.stdout: Package Arch Version Repository Size 2024-09-16T07:51:11.804 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-09-16T07:51:11.804 INFO:teuthology.orchestra.run.smithi187.stdout:Installing: 2024-09-16T07:51:11.804 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-radosgw x86_64 2:19.3.0-4879.g81fedcb3.el9 ceph 10 M 2024-09-16T07:51:11.805 INFO:teuthology.orchestra.run.smithi187.stdout:Upgrading: 2024-09-16T07:51:11.805 INFO:teuthology.orchestra.run.smithi187.stdout: librados2 x86_64 2:19.3.0-4879.g81fedcb3.el9 ceph 3.5 M 2024-09-16T07:51:11.805 INFO:teuthology.orchestra.run.smithi187.stdout: librbd1 x86_64 2:19.3.0-4879.g81fedcb3.el9 ceph 3.8 M 2024-09-16T07:51:11.805 INFO:teuthology.orchestra.run.smithi187.stdout:Installing dependencies: 2024-09-16T07:51:11.805 INFO:teuthology.orchestra.run.smithi187.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-16T07:51:11.805 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-base x86_64 2:19.3.0-4879.g81fedcb3.el9 ceph 6.1 M 2024-09-16T07:51:11.805 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-common x86_64 2:19.3.0-4879.g81fedcb3.el9 ceph 20 M 2024-09-16T07:51:11.805 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-selinux x86_64 2:19.3.0-4879.g81fedcb3.el9 ceph 25 k 2024-09-16T07:51:11.805 INFO:teuthology.orchestra.run.smithi187.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-16T07:51:11.805 INFO:teuthology.orchestra.run.smithi187.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-16T07:51:11.805 INFO:teuthology.orchestra.run.smithi187.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-16T07:51:11.806 INFO:teuthology.orchestra.run.smithi187.stdout: libcephfs2 x86_64 2:19.3.0-4879.g81fedcb3.el9 ceph 809 k 2024-09-16T07:51:11.806 INFO:teuthology.orchestra.run.smithi187.stdout: libnbd x86_64 1.20.2-2.el9 appstream 171 k 2024-09-16T07:51:11.806 INFO:teuthology.orchestra.run.smithi187.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-16T07:51:11.806 INFO:teuthology.orchestra.run.smithi187.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-16T07:51:11.806 INFO:teuthology.orchestra.run.smithi187.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-16T07:51:11.806 INFO:teuthology.orchestra.run.smithi187.stdout: libradosstriper1 x86_64 2:19.3.0-4879.g81fedcb3.el9 ceph 558 k 2024-09-16T07:51:11.806 INFO:teuthology.orchestra.run.smithi187.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-16T07:51:11.806 INFO:teuthology.orchestra.run.smithi187.stdout: librgw2 x86_64 2:19.3.0-4879.g81fedcb3.el9 ceph 5.4 M 2024-09-16T07:51:11.806 INFO:teuthology.orchestra.run.smithi187.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-16T07:51:11.806 INFO:teuthology.orchestra.run.smithi187.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-16T07:51:11.806 INFO:teuthology.orchestra.run.smithi187.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-16T07:51:11.806 INFO:teuthology.orchestra.run.smithi187.stdout: python3-ceph-argparse x86_64 2:19.3.0-4879.g81fedcb3.el9 ceph 45 k 2024-09-16T07:51:11.806 INFO:teuthology.orchestra.run.smithi187.stdout: python3-ceph-common x86_64 2:19.3.0-4879.g81fedcb3.el9 ceph 143 k 2024-09-16T07:51:11.806 INFO:teuthology.orchestra.run.smithi187.stdout: python3-cephfs x86_64 2:19.3.0-4879.g81fedcb3.el9 ceph 160 k 2024-09-16T07:51:11.807 INFO:teuthology.orchestra.run.smithi187.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-16T07:51:11.807 INFO:teuthology.orchestra.run.smithi187.stdout: python3-rados x86_64 2:19.3.0-4879.g81fedcb3.el9 ceph 315 k 2024-09-16T07:51:11.807 INFO:teuthology.orchestra.run.smithi187.stdout: python3-rbd x86_64 2:19.3.0-4879.g81fedcb3.el9 ceph 302 k 2024-09-16T07:51:11.807 INFO:teuthology.orchestra.run.smithi187.stdout: python3-rgw x86_64 2:19.3.0-4879.g81fedcb3.el9 ceph 99 k 2024-09-16T07:51:11.807 INFO:teuthology.orchestra.run.smithi187.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-16T07:51:11.807 INFO:teuthology.orchestra.run.smithi187.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-16T07:51:11.807 INFO:teuthology.orchestra.run.smithi187.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-16T07:51:11.807 INFO:teuthology.orchestra.run.smithi187.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-16T07:51:11.807 INFO:teuthology.orchestra.run.smithi187.stdout:Installing weak dependencies: 2024-09-16T07:51:11.807 INFO:teuthology.orchestra.run.smithi187.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-16T07:51:11.807 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-16T07:51:11.807 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction Summary 2024-09-16T07:51:11.807 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-09-16T07:51:11.808 INFO:teuthology.orchestra.run.smithi187.stdout:Install 31 Packages 2024-09-16T07:51:11.808 INFO:teuthology.orchestra.run.smithi187.stdout:Upgrade 2 Packages 2024-09-16T07:51:11.808 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-16T07:51:11.808 INFO:teuthology.orchestra.run.smithi187.stdout:Total download size: 60 M 2024-09-16T07:51:11.808 INFO:teuthology.orchestra.run.smithi187.stdout:Downloading Packages: 2024-09-16T07:51:12.481 INFO:teuthology.orchestra.run.smithi027.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-09-16T07:51:12.748 INFO:teuthology.orchestra.run.smithi187.stdout:(1/33): ceph-base-19.3.0-4879.g81fedcb3.el9.x86 8.1 MB/s | 6.1 MB 00:00 2024-09-16T07:51:12.799 INFO:teuthology.orchestra.run.smithi187.stdout:(2/33): ceph-selinux-19.3.0-4879.g81fedcb3.el9. 504 kB/s | 25 kB 00:00 2024-09-16T07:51:12.882 INFO:teuthology.orchestra.run.smithi187.stdout:(3/33): libcephfs2-19.3.0-4879.g81fedcb3.el9.x8 9.5 MB/s | 809 kB 00:00 2024-09-16T07:51:13.024 INFO:teuthology.orchestra.run.smithi187.stdout:(4/33): ceph-radosgw-19.3.0-4879.g81fedcb3.el9. 10 MB/s | 10 MB 00:01 2024-09-16T07:51:13.058 INFO:teuthology.orchestra.run.smithi187.stdout:(5/33): libradosstriper1-19.3.0-4879.g81fedcb3. 3.1 MB/s | 558 kB 00:00 2024-09-16T07:51:13.116 INFO:teuthology.orchestra.run.smithi187.stdout:(6/33): python3-ceph-argparse-19.3.0-4879.g81fe 775 kB/s | 45 kB 00:00 2024-09-16T07:51:13.183 INFO:teuthology.orchestra.run.smithi187.stdout:(7/33): python3-ceph-common-19.3.0-4879.g81fedc 2.1 MB/s | 143 kB 00:00 2024-09-16T07:51:13.272 INFO:teuthology.orchestra.run.smithi187.stdout:(8/33): python3-cephfs-19.3.0-4879.g81fedcb3.el 1.8 MB/s | 160 kB 00:00 2024-09-16T07:51:13.492 INFO:teuthology.orchestra.run.smithi187.stdout:(9/33): ceph-common-19.3.0-4879.g81fedcb3.el9.x 13 MB/s | 20 MB 00:01 2024-09-16T07:51:13.567 INFO:teuthology.orchestra.run.smithi187.stdout:(10/33): librgw2-19.3.0-4879.g81fedcb3.el9.x86_ 9.9 MB/s | 5.4 MB 00:00 2024-09-16T07:51:13.593 INFO:teuthology.orchestra.run.smithi187.stdout:(11/33): python3-rados-19.3.0-4879.g81fedcb3.el 984 kB/s | 315 kB 00:00 2024-09-16T07:51:13.618 INFO:teuthology.orchestra.run.smithi187.stdout:(12/33): python3-rbd-19.3.0-4879.g81fedcb3.el9. 2.4 MB/s | 302 kB 00:00 2024-09-16T07:51:13.643 INFO:teuthology.orchestra.run.smithi187.stdout:(13/33): python3-rgw-19.3.0-4879.g81fedcb3.el9. 1.3 MB/s | 99 kB 00:00 2024-09-16T07:51:13.752 INFO:teuthology.orchestra.run.smithi187.stdout:(14/33): boost-program-options-1.75.0-8.el9.x86 673 kB/s | 107 kB 00:00 2024-09-16T07:51:13.777 INFO:teuthology.orchestra.run.smithi187.stdout:(15/33): libnbd-1.20.2-2.el9.x86_64.rpm 1.0 MB/s | 171 kB 00:00 2024-09-16T07:51:13.794 INFO:teuthology.orchestra.run.smithi187.stdout:(16/33): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.1 MB/s | 45 kB 00:00 2024-09-16T07:51:13.811 INFO:teuthology.orchestra.run.smithi187.stdout:(17/33): libpmemobj-1.12.1-1.el9.x86_64.rpm 957 kB/s | 160 kB 00:00 2024-09-16T07:51:13.853 INFO:teuthology.orchestra.run.smithi187.stdout:(18/33): python3-prettytable-0.7.2-27.el9.noarc 1.0 MB/s | 42 kB 00:00 2024-09-16T07:51:13.903 INFO:teuthology.orchestra.run.smithi187.stdout:(19/33): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.6 MB/s | 292 kB 00:00 2024-09-16T07:51:13.937 INFO:teuthology.orchestra.run.smithi187.stdout:(20/33): qatlib-24.02.0-1.el9.x86_64.rpm 2.6 MB/s | 221 kB 00:00 2024-09-16T07:51:13.962 INFO:teuthology.orchestra.run.smithi187.stdout:(21/33): qatlib-service-24.02.0-1.el9.x86_64.rp 614 kB/s | 36 kB 00:00 2024-09-16T07:51:13.987 INFO:teuthology.orchestra.run.smithi187.stdout:(22/33): librdkafka-1.6.1-102.el9.x86_64.rpm 3.1 MB/s | 662 kB 00:00 2024-09-16T07:51:14.012 INFO:teuthology.orchestra.run.smithi187.stdout:(23/33): qatzip-libs-1.2.0-1.el9.x86_64.rpm 630 kB/s | 47 kB 00:00 2024-09-16T07:51:14.080 INFO:teuthology.orchestra.run.smithi187.stdout:(24/33): libarrow-doc-9.0.0-11.el9.noarch.rpm 502 kB/s | 26 kB 00:00 2024-09-16T07:51:14.113 INFO:teuthology.orchestra.run.smithi187.stdout:(25/33): gperftools-libs-2.9.1-3.el9.x86_64.rpm 2.0 MB/s | 308 kB 00:00 2024-09-16T07:51:14.139 INFO:teuthology.orchestra.run.smithi187.stdout:(26/33): liboath-2.6.7-2.el9.x86_64.rpm 828 kB/s | 49 kB 00:00 2024-09-16T07:51:14.164 INFO:teuthology.orchestra.run.smithi187.stdout:(27/33): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-09-16T07:51:14.197 INFO:teuthology.orchestra.run.smithi187.stdout:(28/33): re2-20211101-20.el9.x86_64.rpm 5.6 MB/s | 191 kB 00:00 2024-09-16T07:51:14.239 INFO:teuthology.orchestra.run.smithi187.stdout:(29/33): parquet-libs-9.0.0-11.el9.x86_64.rpm 8.2 MB/s | 839 kB 00:00 2024-09-16T07:51:14.323 INFO:teuthology.orchestra.run.smithi187.stdout:(30/33): thrift-0.15.0-2.el9.x86_64.rpm 13 MB/s | 1.6 MB 00:00 2024-09-16T07:51:14.432 INFO:teuthology.orchestra.run.smithi187.stdout:(31/33): libarrow-9.0.0-11.el9.x86_64.rpm 9.9 MB/s | 4.4 MB 00:00 2024-09-16T07:51:14.565 INFO:teuthology.orchestra.run.smithi187.stdout:(32/33): librados2-19.3.0-4879.g81fedcb3.el9.x8 11 MB/s | 3.5 MB 00:00 2024-09-16T07:51:14.615 INFO:teuthology.orchestra.run.smithi187.stdout:(33/33): librbd1-19.3.0-4879.g81fedcb3.el9.x86_ 13 MB/s | 3.8 MB 00:00 2024-09-16T07:51:14.620 INFO:teuthology.orchestra.run.smithi187.stdout:-------------------------------------------------------------------------------- 2024-09-16T07:51:14.620 INFO:teuthology.orchestra.run.smithi187.stdout:Total 21 MB/s | 60 MB 00:02 2024-09-16T07:51:14.733 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction check 2024-09-16T07:51:14.771 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction check succeeded. 2024-09-16T07:51:14.771 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction test 2024-09-16T07:51:14.944 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-09-16T07:51:14.960 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-09-16T07:51:14.960 INFO:teuthology.orchestra.run.smithi027.stdout: Package Arch Version Repository Size 2024-09-16T07:51:14.960 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-09-16T07:51:14.960 INFO:teuthology.orchestra.run.smithi027.stdout:Installing: 2024-09-16T07:51:14.960 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-radosgw x86_64 2:19.3.0-4879.g81fedcb3.el9 ceph 10 M 2024-09-16T07:51:14.960 INFO:teuthology.orchestra.run.smithi027.stdout:Upgrading: 2024-09-16T07:51:14.960 INFO:teuthology.orchestra.run.smithi027.stdout: librados2 x86_64 2:19.3.0-4879.g81fedcb3.el9 ceph 3.5 M 2024-09-16T07:51:14.960 INFO:teuthology.orchestra.run.smithi027.stdout: librbd1 x86_64 2:19.3.0-4879.g81fedcb3.el9 ceph 3.8 M 2024-09-16T07:51:14.960 INFO:teuthology.orchestra.run.smithi027.stdout:Installing dependencies: 2024-09-16T07:51:14.961 INFO:teuthology.orchestra.run.smithi027.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-16T07:51:14.961 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-base x86_64 2:19.3.0-4879.g81fedcb3.el9 ceph 6.1 M 2024-09-16T07:51:14.961 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-common x86_64 2:19.3.0-4879.g81fedcb3.el9 ceph 20 M 2024-09-16T07:51:14.961 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-selinux x86_64 2:19.3.0-4879.g81fedcb3.el9 ceph 25 k 2024-09-16T07:51:14.961 INFO:teuthology.orchestra.run.smithi027.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-16T07:51:14.961 INFO:teuthology.orchestra.run.smithi027.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-16T07:51:14.961 INFO:teuthology.orchestra.run.smithi027.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-16T07:51:14.961 INFO:teuthology.orchestra.run.smithi027.stdout: libcephfs2 x86_64 2:19.3.0-4879.g81fedcb3.el9 ceph 809 k 2024-09-16T07:51:14.961 INFO:teuthology.orchestra.run.smithi027.stdout: libnbd x86_64 1.20.2-2.el9 appstream 171 k 2024-09-16T07:51:14.961 INFO:teuthology.orchestra.run.smithi027.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-16T07:51:14.961 INFO:teuthology.orchestra.run.smithi027.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-16T07:51:14.961 INFO:teuthology.orchestra.run.smithi027.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-16T07:51:14.961 INFO:teuthology.orchestra.run.smithi027.stdout: libradosstriper1 x86_64 2:19.3.0-4879.g81fedcb3.el9 ceph 558 k 2024-09-16T07:51:14.962 INFO:teuthology.orchestra.run.smithi027.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-16T07:51:14.962 INFO:teuthology.orchestra.run.smithi027.stdout: librgw2 x86_64 2:19.3.0-4879.g81fedcb3.el9 ceph 5.4 M 2024-09-16T07:51:14.962 INFO:teuthology.orchestra.run.smithi027.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-16T07:51:14.962 INFO:teuthology.orchestra.run.smithi027.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-16T07:51:14.962 INFO:teuthology.orchestra.run.smithi027.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-16T07:51:14.962 INFO:teuthology.orchestra.run.smithi027.stdout: python3-ceph-argparse x86_64 2:19.3.0-4879.g81fedcb3.el9 ceph 45 k 2024-09-16T07:51:14.962 INFO:teuthology.orchestra.run.smithi027.stdout: python3-ceph-common x86_64 2:19.3.0-4879.g81fedcb3.el9 ceph 143 k 2024-09-16T07:51:14.962 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cephfs x86_64 2:19.3.0-4879.g81fedcb3.el9 ceph 160 k 2024-09-16T07:51:14.962 INFO:teuthology.orchestra.run.smithi027.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-16T07:51:14.962 INFO:teuthology.orchestra.run.smithi027.stdout: python3-rados x86_64 2:19.3.0-4879.g81fedcb3.el9 ceph 315 k 2024-09-16T07:51:14.962 INFO:teuthology.orchestra.run.smithi027.stdout: python3-rbd x86_64 2:19.3.0-4879.g81fedcb3.el9 ceph 302 k 2024-09-16T07:51:14.962 INFO:teuthology.orchestra.run.smithi027.stdout: python3-rgw x86_64 2:19.3.0-4879.g81fedcb3.el9 ceph 99 k 2024-09-16T07:51:14.962 INFO:teuthology.orchestra.run.smithi027.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-16T07:51:14.963 INFO:teuthology.orchestra.run.smithi027.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-16T07:51:14.963 INFO:teuthology.orchestra.run.smithi027.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-16T07:51:14.963 INFO:teuthology.orchestra.run.smithi027.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-16T07:51:14.963 INFO:teuthology.orchestra.run.smithi027.stdout:Installing weak dependencies: 2024-09-16T07:51:14.963 INFO:teuthology.orchestra.run.smithi027.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-16T07:51:14.963 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-16T07:51:14.963 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2024-09-16T07:51:14.963 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-09-16T07:51:14.963 INFO:teuthology.orchestra.run.smithi027.stdout:Install 31 Packages 2024-09-16T07:51:14.963 INFO:teuthology.orchestra.run.smithi027.stdout:Upgrade 2 Packages 2024-09-16T07:51:14.963 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-16T07:51:14.964 INFO:teuthology.orchestra.run.smithi027.stdout:Total download size: 60 M 2024-09-16T07:51:14.964 INFO:teuthology.orchestra.run.smithi027.stdout:Downloading Packages: 2024-09-16T07:51:15.424 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction test succeeded. 2024-09-16T07:51:15.424 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction 2024-09-16T07:51:15.982 INFO:teuthology.orchestra.run.smithi027.stdout:(1/33): ceph-base-19.3.0-4879.g81fedcb3.el9.x86 9.6 MB/s | 6.1 MB 00:00 2024-09-16T07:51:16.032 INFO:teuthology.orchestra.run.smithi027.stdout:(2/33): ceph-selinux-19.3.0-4879.g81fedcb3.el9. 503 kB/s | 25 kB 00:00 2024-09-16T07:51:16.125 INFO:teuthology.orchestra.run.smithi027.stdout:(3/33): libcephfs2-19.3.0-4879.g81fedcb3.el9.x8 8.6 MB/s | 809 kB 00:00 2024-09-16T07:51:16.241 INFO:teuthology.orchestra.run.smithi027.stdout:(4/33): ceph-radosgw-19.3.0-4879.g81fedcb3.el9. 12 MB/s | 10 MB 00:00 2024-09-16T07:51:16.283 INFO:teuthology.orchestra.run.smithi027.stdout:(5/33): libradosstriper1-19.3.0-4879.g81fedcb3. 3.4 MB/s | 558 kB 00:00 2024-09-16T07:51:16.350 INFO:teuthology.orchestra.run.smithi027.stdout:(6/33): python3-ceph-argparse-19.3.0-4879.g81fe 679 kB/s | 45 kB 00:00 2024-09-16T07:51:16.425 INFO:teuthology.orchestra.run.smithi027.stdout:(7/33): python3-ceph-common-19.3.0-4879.g81fedc 1.9 MB/s | 143 kB 00:00 2024-09-16T07:51:16.501 INFO:teuthology.orchestra.run.smithi027.stdout:(8/33): python3-cephfs-19.3.0-4879.g81fedcb3.el 2.1 MB/s | 160 kB 00:00 2024-09-16T07:51:16.719 INFO:teuthology.orchestra.run.smithi027.stdout:(9/33): ceph-common-19.3.0-4879.g81fedcb3.el9.x 14 MB/s | 20 MB 00:01 2024-09-16T07:51:16.802 INFO:teuthology.orchestra.run.smithi027.stdout:(10/33): librgw2-19.3.0-4879.g81fedcb3.el9.x86_ 9.6 MB/s | 5.4 MB 00:00 2024-09-16T07:51:16.827 INFO:teuthology.orchestra.run.smithi027.stdout:(11/33): python3-rados-19.3.0-4879.g81fedcb3.el 968 kB/s | 315 kB 00:00 2024-09-16T07:51:16.852 INFO:teuthology.orchestra.run.smithi027.stdout:(12/33): python3-rgw-19.3.0-4879.g81fedcb3.el9. 1.9 MB/s | 99 kB 00:00 2024-09-16T07:51:16.877 INFO:teuthology.orchestra.run.smithi027.stdout:(13/33): python3-rbd-19.3.0-4879.g81fedcb3.el9. 1.9 MB/s | 302 kB 00:00 2024-09-16T07:51:16.928 INFO:teuthology.orchestra.run.smithi027.stdout:(14/33): boost-program-options-1.75.0-8.el9.x86 1.0 MB/s | 107 kB 00:00 2024-09-16T07:51:16.929 INFO:teuthology.orchestra.run.smithi187.stdout: Preparing : 1/1 2024-09-16T07:51:16.986 INFO:teuthology.orchestra.run.smithi027.stdout:(15/33): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.4 MB/s | 160 kB 00:00 2024-09-16T07:51:17.011 INFO:teuthology.orchestra.run.smithi027.stdout:(16/33): librabbitmq-0.11.0-7.el9.x86_64.rpm 541 kB/s | 45 kB 00:00 2024-09-16T07:51:17.037 INFO:teuthology.orchestra.run.smithi027.stdout:(17/33): libnbd-1.20.2-2.el9.x86_64.rpm 924 kB/s | 171 kB 00:00 2024-09-16T07:51:17.079 INFO:teuthology.orchestra.run.smithi027.stdout:(18/33): librdkafka-1.6.1-102.el9.x86_64.rpm 7.0 MB/s | 662 kB 00:00 2024-09-16T07:51:17.104 INFO:teuthology.orchestra.run.smithi027.stdout:(19/33): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.1 MB/s | 292 kB 00:00 2024-09-16T07:51:17.130 INFO:teuthology.orchestra.run.smithi027.stdout:(20/33): python3-prettytable-0.7.2-27.el9.noarc 457 kB/s | 42 kB 00:00 2024-09-16T07:51:17.155 INFO:teuthology.orchestra.run.smithi027.stdout:(21/33): qatlib-24.02.0-1.el9.x86_64.rpm 2.8 MB/s | 221 kB 00:00 2024-09-16T07:51:17.180 INFO:teuthology.orchestra.run.smithi027.stdout:(22/33): qatlib-service-24.02.0-1.el9.x86_64.rp 474 kB/s | 36 kB 00:00 2024-09-16T07:51:17.205 INFO:teuthology.orchestra.run.smithi027.stdout:(23/33): qatzip-libs-1.2.0-1.el9.x86_64.rpm 628 kB/s | 47 kB 00:00 2024-09-16T07:51:17.264 INFO:teuthology.orchestra.run.smithi027.stdout:(24/33): gperftools-libs-2.9.1-3.el9.x86_64.rpm 2.8 MB/s | 308 kB 00:00 2024-09-16T07:51:17.289 INFO:teuthology.orchestra.run.smithi027.stdout:(25/33): liboath-2.6.7-2.el9.x86_64.rpm 2.1 MB/s | 49 kB 00:00 2024-09-16T07:51:17.314 INFO:teuthology.orchestra.run.smithi027.stdout:(26/33): libarrow-doc-9.0.0-11.el9.noarch.rpm 281 kB/s | 26 kB 00:00 2024-09-16T07:51:17.340 INFO:teuthology.orchestra.run.smithi027.stdout:(27/33): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-09-16T07:51:17.373 INFO:teuthology.orchestra.run.smithi027.stdout:(28/33): re2-20211101-20.el9.x86_64.rpm 5.6 MB/s | 191 kB 00:00 2024-09-16T07:51:17.432 INFO:teuthology.orchestra.run.smithi027.stdout:(29/33): parquet-libs-9.0.0-11.el9.x86_64.rpm 7.0 MB/s | 839 kB 00:00 2024-09-16T07:51:17.451 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/35 2024-09-16T07:51:17.474 INFO:teuthology.orchestra.run.smithi027.stdout:(30/33): thrift-0.15.0-2.el9.x86_64.rpm 16 MB/s | 1.6 MB 00:00 2024-09-16T07:51:17.501 INFO:teuthology.orchestra.run.smithi187.stdout: Upgrading : librados2-2:19.3.0-4879.g81fedcb3.el9.x86_64 2/35 2024-09-16T07:51:17.557 INFO:teuthology.orchestra.run.smithi027.stdout:(31/33): libarrow-9.0.0-11.el9.x86_64.rpm 12 MB/s | 4.4 MB 00:00 2024-09-16T07:51:17.592 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: librados2-2:19.3.0-4879.g81fedcb3.el9.x86_64 2/35 2024-09-16T07:51:17.688 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-rados-2:19.3.0-4879.g81fedcb3.el9.x86_64 3/35 2024-09-16T07:51:17.719 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : libcephfs2-2:19.3.0-4879.g81fedcb3.el9.x86_64 4/35 2024-09-16T07:51:17.750 INFO:teuthology.orchestra.run.smithi027.stdout:(32/33): librbd1-19.3.0-4879.g81fedcb3.el9.x86_ 14 MB/s | 3.8 MB 00:00 2024-09-16T07:51:17.782 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: libcephfs2-2:19.3.0-4879.g81fedcb3.el9.x86_64 4/35 2024-09-16T07:51:17.808 INFO:teuthology.orchestra.run.smithi027.stdout:(33/33): librados2-19.3.0-4879.g81fedcb3.el9.x8 9.2 MB/s | 3.5 MB 00:00 2024-09-16T07:51:17.813 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2024-09-16T07:51:17.814 INFO:teuthology.orchestra.run.smithi027.stdout:Total 21 MB/s | 60 MB 00:02 2024-09-16T07:51:17.832 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : liboath-2.6.7-2.el9.x86_64 5/35 2024-09-16T07:51:17.865 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/35 2024-09-16T07:51:17.891 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/35 2024-09-16T07:51:17.915 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 8/35 2024-09-16T07:51:17.928 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction check 2024-09-16T07:51:17.945 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 8/35 2024-09-16T07:51:17.970 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction check succeeded. 2024-09-16T07:51:17.970 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction test 2024-09-16T07:51:18.149 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 9/35 2024-09-16T07:51:18.182 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 9/35 2024-09-16T07:51:18.216 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 10/35 2024-09-16T07:51:18.260 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-ceph-argparse-2:19.3.0-4879.g81fedcb3.el9. 11/35 2024-09-16T07:51:18.289 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-cephfs-2:19.3.0-4879.g81fedcb3.el9.x86_64 12/35 2024-09-16T07:51:18.368 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 13/35 2024-09-16T07:51:18.395 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : libradosstriper1-2:19.3.0-4879.g81fedcb3.el9.x86_6 14/35 2024-09-16T07:51:18.482 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: libradosstriper1-2:19.3.0-4879.g81fedcb3.el9.x86_6 14/35 2024-09-16T07:51:18.527 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : thrift-0.15.0-2.el9.x86_64 15/35 2024-09-16T07:51:18.641 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : re2-1:20211101-20.el9.x86_64 16/35 2024-09-16T07:51:18.672 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 17/35 2024-09-16T07:51:18.673 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction test succeeded. 2024-09-16T07:51:18.674 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction 2024-09-16T07:51:18.720 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 18/35 2024-09-16T07:51:18.752 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 19/35 2024-09-16T07:51:18.781 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 20/35 2024-09-16T07:51:18.817 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 21/35 2024-09-16T07:51:19.188 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : libnbd-1.20.2-2.el9.x86_64 22/35 2024-09-16T07:51:19.235 INFO:teuthology.orchestra.run.smithi187.stdout: Upgrading : librbd1-2:19.3.0-4879.g81fedcb3.el9.x86_64 23/35 2024-09-16T07:51:19.309 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: librbd1-2:19.3.0-4879.g81fedcb3.el9.x86_64 23/35 2024-09-16T07:51:19.340 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-rbd-2:19.3.0-4879.g81fedcb3.el9.x86_64 24/35 2024-09-16T07:51:19.376 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 25/35 2024-09-16T07:51:19.968 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 26/35 2024-09-16T07:51:20.004 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : librgw2-2:19.3.0-4879.g81fedcb3.el9.x86_64 27/35 2024-09-16T07:51:20.075 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: librgw2-2:19.3.0-4879.g81fedcb3.el9.x86_64 27/35 2024-09-16T07:51:20.145 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-rgw-2:19.3.0-4879.g81fedcb3.el9.x86_64 28/35 2024-09-16T07:51:20.181 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-ceph-common-2:19.3.0-4879.g81fedcb3.el9.x8 29/35 2024-09-16T07:51:20.241 INFO:teuthology.orchestra.run.smithi027.stdout: Preparing : 1/1 2024-09-16T07:51:20.790 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/35 2024-09-16T07:51:20.877 INFO:teuthology.orchestra.run.smithi027.stdout: Upgrading : librados2-2:19.3.0-4879.g81fedcb3.el9.x86_64 2/35 2024-09-16T07:51:20.953 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: librados2-2:19.3.0-4879.g81fedcb3.el9.x86_64 2/35 2024-09-16T07:51:21.063 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-rados-2:19.3.0-4879.g81fedcb3.el9.x86_64 3/35 2024-09-16T07:51:21.087 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libcephfs2-2:19.3.0-4879.g81fedcb3.el9.x86_64 4/35 2024-09-16T07:51:21.150 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: libcephfs2-2:19.3.0-4879.g81fedcb3.el9.x86_64 4/35 2024-09-16T07:51:21.202 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : liboath-2.6.7-2.el9.x86_64 5/35 2024-09-16T07:51:21.242 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/35 2024-09-16T07:51:21.267 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/35 2024-09-16T07:51:21.298 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 8/35 2024-09-16T07:51:21.328 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 8/35 2024-09-16T07:51:21.529 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 9/35 2024-09-16T07:51:21.565 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 9/35 2024-09-16T07:51:21.600 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 10/35 2024-09-16T07:51:21.644 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-ceph-argparse-2:19.3.0-4879.g81fedcb3.el9. 11/35 2024-09-16T07:51:21.673 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-cephfs-2:19.3.0-4879.g81fedcb3.el9.x86_64 12/35 2024-09-16T07:51:21.763 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 13/35 2024-09-16T07:51:21.794 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libradosstriper1-2:19.3.0-4879.g81fedcb3.el9.x86_6 14/35 2024-09-16T07:51:21.866 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: libradosstriper1-2:19.3.0-4879.g81fedcb3.el9.x86_6 14/35 2024-09-16T07:51:21.911 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : thrift-0.15.0-2.el9.x86_64 15/35 2024-09-16T07:51:22.038 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : re2-1:20211101-20.el9.x86_64 16/35 2024-09-16T07:51:22.086 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 17/35 2024-09-16T07:51:22.134 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 18/35 2024-09-16T07:51:22.169 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 19/35 2024-09-16T07:51:22.198 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 20/35 2024-09-16T07:51:22.234 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 21/35 2024-09-16T07:51:22.610 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libnbd-1.20.2-2.el9.x86_64 22/35 2024-09-16T07:51:22.659 INFO:teuthology.orchestra.run.smithi027.stdout: Upgrading : librbd1-2:19.3.0-4879.g81fedcb3.el9.x86_64 23/35 2024-09-16T07:51:22.693 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-common-2:19.3.0-4879.g81fedcb3.el9.x86_64 30/35 2024-09-16T07:51:22.736 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-common-2:19.3.0-4879.g81fedcb3.el9.x86_64 30/35 2024-09-16T07:51:22.751 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: librbd1-2:19.3.0-4879.g81fedcb3.el9.x86_64 23/35 2024-09-16T07:51:22.781 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-rbd-2:19.3.0-4879.g81fedcb3.el9.x86_64 24/35 2024-09-16T07:51:22.818 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 25/35 2024-09-16T07:51:23.439 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 26/35 2024-09-16T07:51:23.446 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-common-2:19.3.0-4879.g81fedcb3.el9.x86_64 30/35 2024-09-16T07:51:23.471 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : librgw2-2:19.3.0-4879.g81fedcb3.el9.x86_64 27/35 2024-09-16T07:51:23.481 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-base-2:19.3.0-4879.g81fedcb3.el9.x86_64 31/35 2024-09-16T07:51:23.552 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: librgw2-2:19.3.0-4879.g81fedcb3.el9.x86_64 27/35 2024-09-16T07:51:23.568 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-base-2:19.3.0-4879.g81fedcb3.el9.x86_64 31/35 2024-09-16T07:51:23.568 INFO:teuthology.orchestra.run.smithi187.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-16T07:51:23.568 INFO:teuthology.orchestra.run.smithi187.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-16T07:51:23.568 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-16T07:51:23.599 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-selinux-2:19.3.0-4879.g81fedcb3.el9.x86_64 32/35 2024-09-16T07:51:23.631 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-rgw-2:19.3.0-4879.g81fedcb3.el9.x86_64 28/35 2024-09-16T07:51:23.665 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-ceph-common-2:19.3.0-4879.g81fedcb3.el9.x8 29/35 2024-09-16T07:51:26.290 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-common-2:19.3.0-4879.g81fedcb3.el9.x86_64 30/35 2024-09-16T07:51:26.328 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-common-2:19.3.0-4879.g81fedcb3.el9.x86_64 30/35 2024-09-16T07:51:27.069 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-common-2:19.3.0-4879.g81fedcb3.el9.x86_64 30/35 2024-09-16T07:51:27.106 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-base-2:19.3.0-4879.g81fedcb3.el9.x86_64 31/35 2024-09-16T07:51:27.197 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-base-2:19.3.0-4879.g81fedcb3.el9.x86_64 31/35 2024-09-16T07:51:27.197 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-16T07:51:27.197 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-16T07:51:27.197 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-16T07:51:27.242 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-selinux-2:19.3.0-4879.g81fedcb3.el9.x86_64 32/35 2024-09-16T07:51:34.122 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-selinux-2:19.3.0-4879.g81fedcb3.el9.x86_64 32/35 2024-09-16T07:51:34.122 INFO:teuthology.orchestra.run.smithi187.stdout:skipping the directory /sys 2024-09-16T07:51:34.122 INFO:teuthology.orchestra.run.smithi187.stdout:skipping the directory /proc 2024-09-16T07:51:34.123 INFO:teuthology.orchestra.run.smithi187.stdout:skipping the directory /mnt 2024-09-16T07:51:34.123 INFO:teuthology.orchestra.run.smithi187.stdout:skipping the directory /var/tmp 2024-09-16T07:51:34.123 INFO:teuthology.orchestra.run.smithi187.stdout:skipping the directory /home 2024-09-16T07:51:34.123 INFO:teuthology.orchestra.run.smithi187.stdout:skipping the directory /root 2024-09-16T07:51:34.123 INFO:teuthology.orchestra.run.smithi187.stdout:skipping the directory /tmp 2024-09-16T07:51:34.123 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-16T07:51:35.204 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-radosgw-2:19.3.0-4879.g81fedcb3.el9.x86_64 33/35 2024-09-16T07:51:35.230 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-radosgw-2:19.3.0-4879.g81fedcb3.el9.x86_64 33/35 2024-09-16T07:51:35.231 INFO:teuthology.orchestra.run.smithi187.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T07:51:35.231 INFO:teuthology.orchestra.run.smithi187.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-16T07:51:35.231 INFO:teuthology.orchestra.run.smithi187.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-16T07:51:35.231 INFO:teuthology.orchestra.run.smithi187.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-16T07:51:35.231 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-16T07:51:35.232 INFO:teuthology.orchestra.run.smithi187.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 34/35 2024-09-16T07:51:35.323 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 34/35 2024-09-16T07:51:35.323 INFO:teuthology.orchestra.run.smithi187.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 35/35 2024-09-16T07:51:36.687 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 35/35 2024-09-16T07:51:36.687 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-base-2:19.3.0-4879.g81fedcb3.el9.x86_64 1/35 2024-09-16T07:51:36.687 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-common-2:19.3.0-4879.g81fedcb3.el9.x86_64 2/35 2024-09-16T07:51:36.687 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-radosgw-2:19.3.0-4879.g81fedcb3.el9.x86_64 3/35 2024-09-16T07:51:36.687 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-selinux-2:19.3.0-4879.g81fedcb3.el9.x86_64 4/35 2024-09-16T07:51:36.687 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : libcephfs2-2:19.3.0-4879.g81fedcb3.el9.x86_64 5/35 2024-09-16T07:51:36.688 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : libradosstriper1-2:19.3.0-4879.g81fedcb3.el9.x86_6 6/35 2024-09-16T07:51:36.688 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : librgw2-2:19.3.0-4879.g81fedcb3.el9.x86_64 7/35 2024-09-16T07:51:36.688 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-ceph-argparse-2:19.3.0-4879.g81fedcb3.el9. 8/35 2024-09-16T07:51:36.688 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-ceph-common-2:19.3.0-4879.g81fedcb3.el9.x8 9/35 2024-09-16T07:51:36.688 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-cephfs-2:19.3.0-4879.g81fedcb3.el9.x86_64 10/35 2024-09-16T07:51:36.688 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-rados-2:19.3.0-4879.g81fedcb3.el9.x86_64 11/35 2024-09-16T07:51:36.688 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-rbd-2:19.3.0-4879.g81fedcb3.el9.x86_64 12/35 2024-09-16T07:51:36.688 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-rgw-2:19.3.0-4879.g81fedcb3.el9.x86_64 13/35 2024-09-16T07:51:36.688 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/35 2024-09-16T07:51:36.688 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : libnbd-1.20.2-2.el9.x86_64 15/35 2024-09-16T07:51:36.688 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 16/35 2024-09-16T07:51:36.688 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 17/35 2024-09-16T07:51:36.688 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 18/35 2024-09-16T07:51:36.688 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 19/35 2024-09-16T07:51:36.689 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 20/35 2024-09-16T07:51:36.689 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 21/35 2024-09-16T07:51:36.689 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 22/35 2024-09-16T07:51:36.689 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 23/35 2024-09-16T07:51:36.689 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 24/35 2024-09-16T07:51:36.689 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 25/35 2024-09-16T07:51:36.689 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 26/35 2024-09-16T07:51:36.689 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 27/35 2024-09-16T07:51:36.689 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 28/35 2024-09-16T07:51:36.689 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 29/35 2024-09-16T07:51:36.690 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : re2-1:20211101-20.el9.x86_64 30/35 2024-09-16T07:51:36.690 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 31/35 2024-09-16T07:51:36.690 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : librados2-2:19.3.0-4879.g81fedcb3.el9.x86_64 32/35 2024-09-16T07:51:36.690 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 33/35 2024-09-16T07:51:36.690 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : librbd1-2:19.3.0-4879.g81fedcb3.el9.x86_64 34/35 2024-09-16T07:51:37.711 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 35/35 2024-09-16T07:51:37.711 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-16T07:51:37.711 INFO:teuthology.orchestra.run.smithi187.stdout:Upgraded: 2024-09-16T07:51:37.712 INFO:teuthology.orchestra.run.smithi187.stdout: librados2-2:19.3.0-4879.g81fedcb3.el9.x86_64 2024-09-16T07:51:37.712 INFO:teuthology.orchestra.run.smithi187.stdout: librbd1-2:19.3.0-4879.g81fedcb3.el9.x86_64 2024-09-16T07:51:37.712 INFO:teuthology.orchestra.run.smithi187.stdout:Installed: 2024-09-16T07:51:37.712 INFO:teuthology.orchestra.run.smithi187.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-16T07:51:37.712 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-base-2:19.3.0-4879.g81fedcb3.el9.x86_64 2024-09-16T07:51:37.712 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-common-2:19.3.0-4879.g81fedcb3.el9.x86_64 2024-09-16T07:51:37.712 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-radosgw-2:19.3.0-4879.g81fedcb3.el9.x86_64 2024-09-16T07:51:37.712 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-selinux-2:19.3.0-4879.g81fedcb3.el9.x86_64 2024-09-16T07:51:37.712 INFO:teuthology.orchestra.run.smithi187.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-16T07:51:37.712 INFO:teuthology.orchestra.run.smithi187.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-16T07:51:37.712 INFO:teuthology.orchestra.run.smithi187.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-16T07:51:37.712 INFO:teuthology.orchestra.run.smithi187.stdout: libcephfs2-2:19.3.0-4879.g81fedcb3.el9.x86_64 2024-09-16T07:51:37.712 INFO:teuthology.orchestra.run.smithi187.stdout: libnbd-1.20.2-2.el9.x86_64 2024-09-16T07:51:37.713 INFO:teuthology.orchestra.run.smithi187.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-16T07:51:37.713 INFO:teuthology.orchestra.run.smithi187.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-16T07:51:37.713 INFO:teuthology.orchestra.run.smithi187.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-16T07:51:37.713 INFO:teuthology.orchestra.run.smithi187.stdout: libradosstriper1-2:19.3.0-4879.g81fedcb3.el9.x86_64 2024-09-16T07:51:37.713 INFO:teuthology.orchestra.run.smithi187.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-16T07:51:37.713 INFO:teuthology.orchestra.run.smithi187.stdout: librgw2-2:19.3.0-4879.g81fedcb3.el9.x86_64 2024-09-16T07:51:37.713 INFO:teuthology.orchestra.run.smithi187.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-16T07:51:37.713 INFO:teuthology.orchestra.run.smithi187.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-16T07:51:37.713 INFO:teuthology.orchestra.run.smithi187.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-16T07:51:37.713 INFO:teuthology.orchestra.run.smithi187.stdout: python3-ceph-argparse-2:19.3.0-4879.g81fedcb3.el9.x86_64 2024-09-16T07:51:37.713 INFO:teuthology.orchestra.run.smithi187.stdout: python3-ceph-common-2:19.3.0-4879.g81fedcb3.el9.x86_64 2024-09-16T07:51:37.714 INFO:teuthology.orchestra.run.smithi187.stdout: python3-cephfs-2:19.3.0-4879.g81fedcb3.el9.x86_64 2024-09-16T07:51:37.714 INFO:teuthology.orchestra.run.smithi187.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-16T07:51:37.714 INFO:teuthology.orchestra.run.smithi187.stdout: python3-rados-2:19.3.0-4879.g81fedcb3.el9.x86_64 2024-09-16T07:51:37.714 INFO:teuthology.orchestra.run.smithi187.stdout: python3-rbd-2:19.3.0-4879.g81fedcb3.el9.x86_64 2024-09-16T07:51:37.714 INFO:teuthology.orchestra.run.smithi187.stdout: python3-rgw-2:19.3.0-4879.g81fedcb3.el9.x86_64 2024-09-16T07:51:37.714 INFO:teuthology.orchestra.run.smithi187.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-16T07:51:37.714 INFO:teuthology.orchestra.run.smithi187.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-16T07:51:37.714 INFO:teuthology.orchestra.run.smithi187.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-16T07:51:37.714 INFO:teuthology.orchestra.run.smithi187.stdout: re2-1:20211101-20.el9.x86_64 2024-09-16T07:51:37.714 INFO:teuthology.orchestra.run.smithi187.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-16T07:51:37.714 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-16T07:51:37.714 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2024-09-16T07:51:38.021 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum -y install ceph-test 2024-09-16T07:51:38.073 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-selinux-2:19.3.0-4879.g81fedcb3.el9.x86_64 32/35 2024-09-16T07:51:38.073 INFO:teuthology.orchestra.run.smithi027.stdout:skipping the directory /sys 2024-09-16T07:51:38.073 INFO:teuthology.orchestra.run.smithi027.stdout:skipping the directory /proc 2024-09-16T07:51:38.073 INFO:teuthology.orchestra.run.smithi027.stdout:skipping the directory /mnt 2024-09-16T07:51:38.073 INFO:teuthology.orchestra.run.smithi027.stdout:skipping the directory /var/tmp 2024-09-16T07:51:38.073 INFO:teuthology.orchestra.run.smithi027.stdout:skipping the directory /home 2024-09-16T07:51:38.074 INFO:teuthology.orchestra.run.smithi027.stdout:skipping the directory /root 2024-09-16T07:51:38.074 INFO:teuthology.orchestra.run.smithi027.stdout:skipping the directory /tmp 2024-09-16T07:51:38.074 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-16T07:51:38.588 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:00:29 ago on Mon 16 Sep 2024 07:51:09 AM UTC. 2024-09-16T07:51:38.722 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2024-09-16T07:51:38.723 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-09-16T07:51:38.723 INFO:teuthology.orchestra.run.smithi187.stdout: Package Arch Version Repository Size 2024-09-16T07:51:38.723 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-09-16T07:51:38.724 INFO:teuthology.orchestra.run.smithi187.stdout:Installing: 2024-09-16T07:51:38.724 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-test x86_64 2:19.3.0-4879.g81fedcb3.el9 ceph 85 M 2024-09-16T07:51:38.724 INFO:teuthology.orchestra.run.smithi187.stdout:Installing dependencies: 2024-09-16T07:51:38.724 INFO:teuthology.orchestra.run.smithi187.stdout: libasan8 x86_64 14.2.1-1.2.el9 appstream 513 k 2024-09-16T07:51:38.724 INFO:teuthology.orchestra.run.smithi187.stdout: libcephsqlite x86_64 2:19.3.0-4879.g81fedcb3.el9 ceph 175 k 2024-09-16T07:51:38.724 INFO:teuthology.orchestra.run.smithi187.stdout: libubsan x86_64 11.5.0-2.el9 appstream 183 k 2024-09-16T07:51:38.724 INFO:teuthology.orchestra.run.smithi187.stdout: lksctp-tools x86_64 1.0.19-2.el9 baseos 94 k 2024-09-16T07:51:38.724 INFO:teuthology.orchestra.run.smithi187.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-16T07:51:38.724 INFO:teuthology.orchestra.run.smithi187.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-16T07:51:38.724 INFO:teuthology.orchestra.run.smithi187.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-16T07:51:38.724 INFO:teuthology.orchestra.run.smithi187.stdout: yaml-cpp x86_64 0.6.3-5.el9 epel 122 k 2024-09-16T07:51:38.724 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-16T07:51:38.725 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction Summary 2024-09-16T07:51:38.725 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-09-16T07:51:38.725 INFO:teuthology.orchestra.run.smithi187.stdout:Install 9 Packages 2024-09-16T07:51:38.725 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-16T07:51:38.725 INFO:teuthology.orchestra.run.smithi187.stdout:Total download size: 87 M 2024-09-16T07:51:38.725 INFO:teuthology.orchestra.run.smithi187.stdout:Installed size: 797 M 2024-09-16T07:51:38.725 INFO:teuthology.orchestra.run.smithi187.stdout:Downloading Packages: 2024-09-16T07:51:39.180 INFO:teuthology.orchestra.run.smithi187.stdout:(1/9): libcephsqlite-19.3.0-4879.g81fedcb3.el9. 760 kB/s | 175 kB 00:00 2024-09-16T07:51:39.218 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-radosgw-2:19.3.0-4879.g81fedcb3.el9.x86_64 33/35 2024-09-16T07:51:39.244 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-radosgw-2:19.3.0-4879.g81fedcb3.el9.x86_64 33/35 2024-09-16T07:51:39.244 INFO:teuthology.orchestra.run.smithi027.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T07:51:39.244 INFO:teuthology.orchestra.run.smithi027.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-16T07:51:39.245 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-16T07:51:39.245 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-16T07:51:39.245 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-16T07:51:39.246 INFO:teuthology.orchestra.run.smithi027.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 34/35 2024-09-16T07:51:39.337 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 34/35 2024-09-16T07:51:39.337 INFO:teuthology.orchestra.run.smithi027.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 35/35 2024-09-16T07:51:39.380 INFO:teuthology.orchestra.run.smithi187.stdout:(2/9): libasan8-14.2.1-1.2.el9.x86_64.rpm 2.5 MB/s | 513 kB 00:00 2024-09-16T07:51:39.405 INFO:teuthology.orchestra.run.smithi187.stdout:(3/9): lksctp-tools-1.0.19-2.el9.x86_64.rpm 206 kB/s | 94 kB 00:00 2024-09-16T07:51:39.431 INFO:teuthology.orchestra.run.smithi187.stdout:(4/9): libubsan-11.5.0-2.el9.x86_64.rpm 3.6 MB/s | 183 kB 00:00 2024-09-16T07:51:39.490 INFO:teuthology.orchestra.run.smithi187.stdout:(5/9): socat-1.7.4.1-6.el9.x86_64.rpm 5.1 MB/s | 304 kB 00:00 2024-09-16T07:51:39.540 INFO:teuthology.orchestra.run.smithi187.stdout:(6/9): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 64 kB 00:00 2024-09-16T07:51:39.590 INFO:teuthology.orchestra.run.smithi187.stdout:(7/9): yaml-cpp-0.6.3-5.el9.x86_64.rpm 2.4 MB/s | 122 kB 00:00 2024-09-16T07:51:39.632 INFO:teuthology.orchestra.run.smithi187.stdout:(8/9): protobuf-3.14.0-13.el9.x86_64.rpm 4.5 MB/s | 1.0 MB 00:00 2024-09-16T07:51:40.613 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 35/35 2024-09-16T07:51:40.613 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-base-2:19.3.0-4879.g81fedcb3.el9.x86_64 1/35 2024-09-16T07:51:40.613 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-common-2:19.3.0-4879.g81fedcb3.el9.x86_64 2/35 2024-09-16T07:51:40.613 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-radosgw-2:19.3.0-4879.g81fedcb3.el9.x86_64 3/35 2024-09-16T07:51:40.613 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-selinux-2:19.3.0-4879.g81fedcb3.el9.x86_64 4/35 2024-09-16T07:51:40.613 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libcephfs2-2:19.3.0-4879.g81fedcb3.el9.x86_64 5/35 2024-09-16T07:51:40.613 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libradosstriper1-2:19.3.0-4879.g81fedcb3.el9.x86_6 6/35 2024-09-16T07:51:40.614 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : librgw2-2:19.3.0-4879.g81fedcb3.el9.x86_64 7/35 2024-09-16T07:51:40.614 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-ceph-argparse-2:19.3.0-4879.g81fedcb3.el9. 8/35 2024-09-16T07:51:40.614 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-ceph-common-2:19.3.0-4879.g81fedcb3.el9.x8 9/35 2024-09-16T07:51:40.614 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-cephfs-2:19.3.0-4879.g81fedcb3.el9.x86_64 10/35 2024-09-16T07:51:40.614 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-rados-2:19.3.0-4879.g81fedcb3.el9.x86_64 11/35 2024-09-16T07:51:40.614 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-rbd-2:19.3.0-4879.g81fedcb3.el9.x86_64 12/35 2024-09-16T07:51:40.614 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-rgw-2:19.3.0-4879.g81fedcb3.el9.x86_64 13/35 2024-09-16T07:51:40.614 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/35 2024-09-16T07:51:40.614 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libnbd-1.20.2-2.el9.x86_64 15/35 2024-09-16T07:51:40.614 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 16/35 2024-09-16T07:51:40.614 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 17/35 2024-09-16T07:51:40.614 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 18/35 2024-09-16T07:51:40.614 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 19/35 2024-09-16T07:51:40.615 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 20/35 2024-09-16T07:51:40.615 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 21/35 2024-09-16T07:51:40.615 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 22/35 2024-09-16T07:51:40.615 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 23/35 2024-09-16T07:51:40.615 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 24/35 2024-09-16T07:51:40.615 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 25/35 2024-09-16T07:51:40.616 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 26/35 2024-09-16T07:51:40.616 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 27/35 2024-09-16T07:51:40.616 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 28/35 2024-09-16T07:51:40.616 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 29/35 2024-09-16T07:51:40.616 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : re2-1:20211101-20.el9.x86_64 30/35 2024-09-16T07:51:40.616 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 31/35 2024-09-16T07:51:40.617 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : librados2-2:19.3.0-4879.g81fedcb3.el9.x86_64 32/35 2024-09-16T07:51:40.617 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 33/35 2024-09-16T07:51:40.617 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : librbd1-2:19.3.0-4879.g81fedcb3.el9.x86_64 34/35 2024-09-16T07:51:41.507 INFO:teuthology.orchestra.run.smithi187.stdout:(9/9): ceph-test-19.3.0-4879.g81fedcb3.el9.x86_ 33 MB/s | 85 MB 00:02 2024-09-16T07:51:41.510 INFO:teuthology.orchestra.run.smithi187.stdout:-------------------------------------------------------------------------------- 2024-09-16T07:51:41.510 INFO:teuthology.orchestra.run.smithi187.stdout:Total 31 MB/s | 87 MB 00:02 2024-09-16T07:51:41.580 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 35/35 2024-09-16T07:51:41.580 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-16T07:51:41.580 INFO:teuthology.orchestra.run.smithi027.stdout:Upgraded: 2024-09-16T07:51:41.580 INFO:teuthology.orchestra.run.smithi027.stdout: librados2-2:19.3.0-4879.g81fedcb3.el9.x86_64 2024-09-16T07:51:41.580 INFO:teuthology.orchestra.run.smithi027.stdout: librbd1-2:19.3.0-4879.g81fedcb3.el9.x86_64 2024-09-16T07:51:41.581 INFO:teuthology.orchestra.run.smithi027.stdout:Installed: 2024-09-16T07:51:41.581 INFO:teuthology.orchestra.run.smithi027.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-16T07:51:41.581 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-base-2:19.3.0-4879.g81fedcb3.el9.x86_64 2024-09-16T07:51:41.581 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-common-2:19.3.0-4879.g81fedcb3.el9.x86_64 2024-09-16T07:51:41.581 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-radosgw-2:19.3.0-4879.g81fedcb3.el9.x86_64 2024-09-16T07:51:41.581 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-selinux-2:19.3.0-4879.g81fedcb3.el9.x86_64 2024-09-16T07:51:41.581 INFO:teuthology.orchestra.run.smithi027.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-16T07:51:41.581 INFO:teuthology.orchestra.run.smithi027.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-16T07:51:41.581 INFO:teuthology.orchestra.run.smithi027.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-16T07:51:41.581 INFO:teuthology.orchestra.run.smithi027.stdout: libcephfs2-2:19.3.0-4879.g81fedcb3.el9.x86_64 2024-09-16T07:51:41.581 INFO:teuthology.orchestra.run.smithi027.stdout: libnbd-1.20.2-2.el9.x86_64 2024-09-16T07:51:41.581 INFO:teuthology.orchestra.run.smithi027.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-16T07:51:41.582 INFO:teuthology.orchestra.run.smithi027.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-16T07:51:41.582 INFO:teuthology.orchestra.run.smithi027.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-16T07:51:41.582 INFO:teuthology.orchestra.run.smithi027.stdout: libradosstriper1-2:19.3.0-4879.g81fedcb3.el9.x86_64 2024-09-16T07:51:41.582 INFO:teuthology.orchestra.run.smithi027.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-16T07:51:41.582 INFO:teuthology.orchestra.run.smithi027.stdout: librgw2-2:19.3.0-4879.g81fedcb3.el9.x86_64 2024-09-16T07:51:41.582 INFO:teuthology.orchestra.run.smithi027.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-16T07:51:41.582 INFO:teuthology.orchestra.run.smithi027.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-16T07:51:41.582 INFO:teuthology.orchestra.run.smithi027.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-16T07:51:41.582 INFO:teuthology.orchestra.run.smithi027.stdout: python3-ceph-argparse-2:19.3.0-4879.g81fedcb3.el9.x86_64 2024-09-16T07:51:41.582 INFO:teuthology.orchestra.run.smithi027.stdout: python3-ceph-common-2:19.3.0-4879.g81fedcb3.el9.x86_64 2024-09-16T07:51:41.582 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cephfs-2:19.3.0-4879.g81fedcb3.el9.x86_64 2024-09-16T07:51:41.582 INFO:teuthology.orchestra.run.smithi027.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-16T07:51:41.582 INFO:teuthology.orchestra.run.smithi027.stdout: python3-rados-2:19.3.0-4879.g81fedcb3.el9.x86_64 2024-09-16T07:51:41.582 INFO:teuthology.orchestra.run.smithi027.stdout: python3-rbd-2:19.3.0-4879.g81fedcb3.el9.x86_64 2024-09-16T07:51:41.583 INFO:teuthology.orchestra.run.smithi027.stdout: python3-rgw-2:19.3.0-4879.g81fedcb3.el9.x86_64 2024-09-16T07:51:41.583 INFO:teuthology.orchestra.run.smithi027.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-16T07:51:41.583 INFO:teuthology.orchestra.run.smithi027.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-16T07:51:41.583 INFO:teuthology.orchestra.run.smithi027.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-16T07:51:41.583 INFO:teuthology.orchestra.run.smithi027.stdout: re2-1:20211101-20.el9.x86_64 2024-09-16T07:51:41.583 INFO:teuthology.orchestra.run.smithi027.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-16T07:51:41.583 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-16T07:51:41.583 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-09-16T07:51:41.594 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction check 2024-09-16T07:51:41.617 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction check succeeded. 2024-09-16T07:51:41.617 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction test 2024-09-16T07:51:41.994 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install ceph-test 2024-09-16T07:51:42.230 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction test succeeded. 2024-09-16T07:51:42.230 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction 2024-09-16T07:51:42.580 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:00:30 ago on Mon 16 Sep 2024 07:51:12 AM UTC. 2024-09-16T07:51:42.721 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-09-16T07:51:42.722 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-09-16T07:51:42.722 INFO:teuthology.orchestra.run.smithi027.stdout: Package Arch Version Repository Size 2024-09-16T07:51:42.722 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-09-16T07:51:42.722 INFO:teuthology.orchestra.run.smithi027.stdout:Installing: 2024-09-16T07:51:42.722 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-test x86_64 2:19.3.0-4879.g81fedcb3.el9 ceph 85 M 2024-09-16T07:51:42.723 INFO:teuthology.orchestra.run.smithi027.stdout:Installing dependencies: 2024-09-16T07:51:42.723 INFO:teuthology.orchestra.run.smithi027.stdout: libasan8 x86_64 14.2.1-1.2.el9 appstream 513 k 2024-09-16T07:51:42.723 INFO:teuthology.orchestra.run.smithi027.stdout: libcephsqlite x86_64 2:19.3.0-4879.g81fedcb3.el9 ceph 175 k 2024-09-16T07:51:42.723 INFO:teuthology.orchestra.run.smithi027.stdout: libubsan x86_64 11.5.0-2.el9 appstream 183 k 2024-09-16T07:51:42.723 INFO:teuthology.orchestra.run.smithi027.stdout: lksctp-tools x86_64 1.0.19-2.el9 baseos 94 k 2024-09-16T07:51:42.723 INFO:teuthology.orchestra.run.smithi027.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-16T07:51:42.723 INFO:teuthology.orchestra.run.smithi027.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-16T07:51:42.723 INFO:teuthology.orchestra.run.smithi027.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-16T07:51:42.723 INFO:teuthology.orchestra.run.smithi027.stdout: yaml-cpp x86_64 0.6.3-5.el9 epel 122 k 2024-09-16T07:51:42.723 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-16T07:51:42.723 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2024-09-16T07:51:42.723 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-09-16T07:51:42.724 INFO:teuthology.orchestra.run.smithi027.stdout:Install 9 Packages 2024-09-16T07:51:42.724 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-16T07:51:42.724 INFO:teuthology.orchestra.run.smithi027.stdout:Total download size: 87 M 2024-09-16T07:51:42.724 INFO:teuthology.orchestra.run.smithi027.stdout:Installed size: 797 M 2024-09-16T07:51:42.724 INFO:teuthology.orchestra.run.smithi027.stdout:Downloading Packages: 2024-09-16T07:51:43.127 INFO:teuthology.orchestra.run.smithi187.stdout: Preparing : 1/1 2024-09-16T07:51:43.201 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : yaml-cpp-0.6.3-5.el9.x86_64 1/9 2024-09-16T07:51:43.234 INFO:teuthology.orchestra.run.smithi027.stdout:(1/9): lksctp-tools-1.0.19-2.el9.x86_64.rpm 613 kB/s | 94 kB 00:00 2024-09-16T07:51:43.250 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 2/9 2024-09-16T07:51:43.293 INFO:teuthology.orchestra.run.smithi027.stdout:(2/9): libcephsqlite-19.3.0-4879.g81fedcb3.el9. 824 kB/s | 175 kB 00:00 2024-09-16T07:51:43.294 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 3/9 2024-09-16T07:51:43.329 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 4/9 2024-09-16T07:51:43.369 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : libubsan-11.5.0-2.el9.x86_64 5/9 2024-09-16T07:51:43.393 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : libasan8-14.2.1-1.2.el9.x86_64 6/9 2024-09-16T07:51:43.472 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: libasan8-14.2.1-1.2.el9.x86_64 6/9 2024-09-16T07:51:43.519 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : lksctp-tools-1.0.19-2.el9.x86_64 7/9 2024-09-16T07:51:43.545 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : libcephsqlite-2:19.3.0-4879.g81fedcb3.el9.x86_64 8/9 2024-09-16T07:51:44.010 INFO:teuthology.orchestra.run.smithi027.stdout:(3/9): libubsan-11.5.0-2.el9.x86_64.rpm 255 kB/s | 183 kB 00:00 2024-09-16T07:51:44.060 INFO:teuthology.orchestra.run.smithi027.stdout:(4/9): libasan8-14.2.1-1.2.el9.x86_64.rpm 622 kB/s | 513 kB 00:00 2024-09-16T07:51:45.535 INFO:teuthology.orchestra.run.smithi027.stdout:(5/9): ceph-test-19.3.0-4879.g81fedcb3.el9.x86_ 34 MB/s | 85 MB 00:02 2024-09-16T07:51:45.561 INFO:teuthology.orchestra.run.smithi027.stdout:(6/9): socat-1.7.4.1-6.el9.x86_64.rpm 203 kB/s | 304 kB 00:01 2024-09-16T07:51:45.594 INFO:teuthology.orchestra.run.smithi027.stdout:(7/9): protobuf-3.14.0-13.el9.x86_64.rpm 652 kB/s | 1.0 MB 00:01 2024-09-16T07:51:45.644 INFO:teuthology.orchestra.run.smithi027.stdout:(8/9): yaml-cpp-0.6.3-5.el9.x86_64.rpm 1.4 MB/s | 122 kB 00:00 2024-09-16T07:51:46.153 INFO:teuthology.orchestra.run.smithi027.stdout:(9/9): xmlstarlet-1.6.1-20.el9.x86_64.rpm 103 kB/s | 64 kB 00:00 2024-09-16T07:51:46.156 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2024-09-16T07:51:46.157 INFO:teuthology.orchestra.run.smithi027.stdout:Total 25 MB/s | 87 MB 00:03 2024-09-16T07:51:46.243 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction check 2024-09-16T07:51:46.269 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction check succeeded. 2024-09-16T07:51:46.269 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction test 2024-09-16T07:51:46.933 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction test succeeded. 2024-09-16T07:51:46.934 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction 2024-09-16T07:51:47.847 INFO:teuthology.orchestra.run.smithi027.stdout: Preparing : 1/1 2024-09-16T07:51:47.927 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : yaml-cpp-0.6.3-5.el9.x86_64 1/9 2024-09-16T07:51:47.983 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 2/9 2024-09-16T07:51:48.051 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 3/9 2024-09-16T07:51:48.077 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 4/9 2024-09-16T07:51:48.108 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libubsan-11.5.0-2.el9.x86_64 5/9 2024-09-16T07:51:48.132 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libasan8-14.2.1-1.2.el9.x86_64 6/9 2024-09-16T07:51:48.208 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: libasan8-14.2.1-1.2.el9.x86_64 6/9 2024-09-16T07:51:48.251 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : lksctp-tools-1.0.19-2.el9.x86_64 7/9 2024-09-16T07:51:48.276 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libcephsqlite-2:19.3.0-4879.g81fedcb3.el9.x86_64 8/9 2024-09-16T07:51:54.753 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: libcephsqlite-2:19.3.0-4879.g81fedcb3.el9.x86_64 8/9 2024-09-16T07:51:54.826 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-test-2:19.3.0-4879.g81fedcb3.el9.x86_64 9/9 2024-09-16T07:52:00.425 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: libcephsqlite-2:19.3.0-4879.g81fedcb3.el9.x86_64 8/9 2024-09-16T07:52:00.510 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-test-2:19.3.0-4879.g81fedcb3.el9.x86_64 9/9 2024-09-16T07:52:01.037 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-test-2:19.3.0-4879.g81fedcb3.el9.x86_64 9/9 2024-09-16T07:52:01.038 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-test-2:19.3.0-4879.g81fedcb3.el9.x86_64 1/9 2024-09-16T07:52:01.038 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : libcephsqlite-2:19.3.0-4879.g81fedcb3.el9.x86_64 2/9 2024-09-16T07:52:01.038 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : lksctp-tools-1.0.19-2.el9.x86_64 3/9 2024-09-16T07:52:01.038 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : libasan8-14.2.1-1.2.el9.x86_64 4/9 2024-09-16T07:52:01.038 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : libubsan-11.5.0-2.el9.x86_64 5/9 2024-09-16T07:52:01.038 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 6/9 2024-09-16T07:52:01.038 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 7/9 2024-09-16T07:52:01.038 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 8/9 2024-09-16T07:52:01.416 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : yaml-cpp-0.6.3-5.el9.x86_64 9/9 2024-09-16T07:52:01.416 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-16T07:52:01.416 INFO:teuthology.orchestra.run.smithi187.stdout:Installed: 2024-09-16T07:52:01.416 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-test-2:19.3.0-4879.g81fedcb3.el9.x86_64 2024-09-16T07:52:01.416 INFO:teuthology.orchestra.run.smithi187.stdout: libasan8-14.2.1-1.2.el9.x86_64 2024-09-16T07:52:01.417 INFO:teuthology.orchestra.run.smithi187.stdout: libcephsqlite-2:19.3.0-4879.g81fedcb3.el9.x86_64 2024-09-16T07:52:01.417 INFO:teuthology.orchestra.run.smithi187.stdout: libubsan-11.5.0-2.el9.x86_64 2024-09-16T07:52:01.417 INFO:teuthology.orchestra.run.smithi187.stdout: lksctp-tools-1.0.19-2.el9.x86_64 2024-09-16T07:52:01.417 INFO:teuthology.orchestra.run.smithi187.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-16T07:52:01.417 INFO:teuthology.orchestra.run.smithi187.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-16T07:52:01.417 INFO:teuthology.orchestra.run.smithi187.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-16T07:52:01.417 INFO:teuthology.orchestra.run.smithi187.stdout: yaml-cpp-0.6.3-5.el9.x86_64 2024-09-16T07:52:01.417 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-16T07:52:01.417 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2024-09-16T07:52:01.755 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum -y install ceph 2024-09-16T07:52:02.322 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:00:53 ago on Mon 16 Sep 2024 07:51:09 AM UTC. 2024-09-16T07:52:02.459 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2024-09-16T07:52:02.462 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-09-16T07:52:02.462 INFO:teuthology.orchestra.run.smithi187.stdout: Package Arch Version Repository Size 2024-09-16T07:52:02.462 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-09-16T07:52:02.462 INFO:teuthology.orchestra.run.smithi187.stdout:Installing: 2024-09-16T07:52:02.462 INFO:teuthology.orchestra.run.smithi187.stdout: ceph x86_64 2:19.3.0-4879.g81fedcb3.el9 ceph 6.5 k 2024-09-16T07:52:02.462 INFO:teuthology.orchestra.run.smithi187.stdout:Installing dependencies: 2024-09-16T07:52:02.463 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mds x86_64 2:19.3.0-4879.g81fedcb3.el9 ceph 2.2 M 2024-09-16T07:52:02.463 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mgr x86_64 2:19.3.0-4879.g81fedcb3.el9 ceph 1.6 M 2024-09-16T07:52:02.463 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mgr-modules-core noarch 2:19.3.0-4879.g81fedcb3.el9 ceph-noarch 291 k 2024-09-16T07:52:02.463 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mon x86_64 2:19.3.0-4879.g81fedcb3.el9 ceph 5.1 M 2024-09-16T07:52:02.463 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-osd x86_64 2:19.3.0-4879.g81fedcb3.el9 ceph 70 M 2024-09-16T07:52:02.463 INFO:teuthology.orchestra.run.smithi187.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-16T07:52:02.463 INFO:teuthology.orchestra.run.smithi187.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-16T07:52:02.463 INFO:teuthology.orchestra.run.smithi187.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-16T07:52:02.463 INFO:teuthology.orchestra.run.smithi187.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-16T07:52:02.463 INFO:teuthology.orchestra.run.smithi187.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-16T07:52:02.463 INFO:teuthology.orchestra.run.smithi187.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-16T07:52:02.463 INFO:teuthology.orchestra.run.smithi187.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-16T07:52:02.464 INFO:teuthology.orchestra.run.smithi187.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-16T07:52:02.464 INFO:teuthology.orchestra.run.smithi187.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-16T07:52:02.464 INFO:teuthology.orchestra.run.smithi187.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-16T07:52:02.464 INFO:teuthology.orchestra.run.smithi187.stdout: python3-jaraco-collections 2024-09-16T07:52:02.464 INFO:teuthology.orchestra.run.smithi187.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-09-16T07:52:02.464 INFO:teuthology.orchestra.run.smithi187.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-16T07:52:02.464 INFO:teuthology.orchestra.run.smithi187.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-16T07:52:02.464 INFO:teuthology.orchestra.run.smithi187.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-16T07:52:02.464 INFO:teuthology.orchestra.run.smithi187.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-16T07:52:02.464 INFO:teuthology.orchestra.run.smithi187.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-16T07:52:02.464 INFO:teuthology.orchestra.run.smithi187.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-16T07:52:02.464 INFO:teuthology.orchestra.run.smithi187.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-16T07:52:02.464 INFO:teuthology.orchestra.run.smithi187.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-16T07:52:02.465 INFO:teuthology.orchestra.run.smithi187.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-16T07:52:02.465 INFO:teuthology.orchestra.run.smithi187.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-16T07:52:02.465 INFO:teuthology.orchestra.run.smithi187.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-16T07:52:02.465 INFO:teuthology.orchestra.run.smithi187.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-16T07:52:02.465 INFO:teuthology.orchestra.run.smithi187.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-16T07:52:02.465 INFO:teuthology.orchestra.run.smithi187.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-16T07:52:02.465 INFO:teuthology.orchestra.run.smithi187.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-16T07:52:02.465 INFO:teuthology.orchestra.run.smithi187.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-16T07:52:02.465 INFO:teuthology.orchestra.run.smithi187.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-16T07:52:02.465 INFO:teuthology.orchestra.run.smithi187.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-16T07:52:02.465 INFO:teuthology.orchestra.run.smithi187.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-16T07:52:02.465 INFO:teuthology.orchestra.run.smithi187.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-16T07:52:02.465 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-16T07:52:02.466 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction Summary 2024-09-16T07:52:02.466 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-09-16T07:52:02.466 INFO:teuthology.orchestra.run.smithi187.stdout:Install 37 Packages 2024-09-16T07:52:02.466 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-16T07:52:02.466 INFO:teuthology.orchestra.run.smithi187.stdout:Total download size: 84 M 2024-09-16T07:52:02.466 INFO:teuthology.orchestra.run.smithi187.stdout:Installed size: 897 M 2024-09-16T07:52:02.466 INFO:teuthology.orchestra.run.smithi187.stdout:Downloading Packages: 2024-09-16T07:52:03.070 INFO:teuthology.orchestra.run.smithi187.stdout:(1/37): ceph-19.3.0-4879.g81fedcb3.el9.x86_64.r 30 kB/s | 6.5 kB 00:00 2024-09-16T07:52:03.287 INFO:teuthology.orchestra.run.smithi187.stdout:(2/37): ceph-mds-19.3.0-4879.g81fedcb3.el9.x86_ 5.1 MB/s | 2.2 MB 00:00 2024-09-16T07:52:03.329 INFO:teuthology.orchestra.run.smithi187.stdout:(3/37): ceph-mgr-19.3.0-4879.g81fedcb3.el9.x86_ 3.4 MB/s | 1.6 MB 00:00 2024-09-16T07:52:03.420 INFO:teuthology.orchestra.run.smithi187.stdout:(4/37): ceph-mgr-modules-core-19.3.0-4879.g81fe 3.1 MB/s | 291 kB 00:00 2024-09-16T07:52:03.487 INFO:teuthology.orchestra.run.smithi187.stdout:(5/37): ceph-mon-19.3.0-4879.g81fedcb3.el9.x86_ 12 MB/s | 5.1 MB 00:00 2024-09-16T07:52:03.607 INFO:teuthology.orchestra.run.smithi187.stdout:(6/37): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.3 MB/s | 253 kB 00:00 2024-09-16T07:52:03.649 INFO:teuthology.orchestra.run.smithi187.stdout:(7/37): python3-ply-3.11-14.el9.noarch.rpm 2.5 MB/s | 106 kB 00:00 2024-09-16T07:52:03.701 INFO:teuthology.orchestra.run.smithi187.stdout:(8/37): python3-cryptography-36.0.1-4.el9.x86_6 5.8 MB/s | 1.2 MB 00:00 2024-09-16T07:52:03.725 INFO:teuthology.orchestra.run.smithi187.stdout:(9/37): python3-pycparser-2.20-6.el9.noarch.rpm 1.8 MB/s | 135 kB 00:00 2024-09-16T07:52:03.750 INFO:teuthology.orchestra.run.smithi187.stdout:(10/37): python3-requests-2.25.1-8.el9.noarch.r 2.4 MB/s | 125 kB 00:00 2024-09-16T07:52:03.775 INFO:teuthology.orchestra.run.smithi187.stdout:(11/37): python3-urllib3-1.26.5-6.el9.noarch.rp 4.2 MB/s | 215 kB 00:00 2024-09-16T07:52:03.859 INFO:teuthology.orchestra.run.smithi187.stdout:(12/37): lua-5.4.4-4.el9.x86_64.rpm 1.7 MB/s | 188 kB 00:00 2024-09-16T07:52:03.884 INFO:teuthology.orchestra.run.smithi187.stdout:(13/37): python3-mako-1.1.4-6.el9.noarch.rpm 1.5 MB/s | 172 kB 00:00 2024-09-16T07:52:03.901 INFO:teuthology.orchestra.run.smithi187.stdout:(14/37): python3-markupsafe-1.1.1-12.el9.x86_64 831 kB/s | 35 kB 00:00 2024-09-16T07:52:03.926 INFO:teuthology.orchestra.run.smithi187.stdout:(15/37): python3-packaging-20.9-5.el9.noarch.rp 1.8 MB/s | 77 kB 00:00 2024-09-16T07:52:03.952 INFO:teuthology.orchestra.run.smithi187.stdout:(16/37): python3-pytz-2021.1-5.el9.noarch.rpm 1.0 MB/s | 51 kB 00:00 2024-09-16T07:52:03.977 INFO:teuthology.orchestra.run.smithi187.stdout:(17/37): python3-toml-0.10.2-6.el9.noarch.rpm 831 kB/s | 42 kB 00:00 2024-09-16T07:52:04.027 INFO:teuthology.orchestra.run.smithi187.stdout:(18/37): luarocks-3.9.2-5.el9.noarch.rpm 3.0 MB/s | 151 kB 00:00 2024-09-16T07:52:04.053 INFO:teuthology.orchestra.run.smithi187.stdout:(19/37): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 1.7 MB/s | 43 kB 00:00 2024-09-16T07:52:04.078 INFO:teuthology.orchestra.run.smithi187.stdout:(20/37): lua-devel-5.4.4-4.el9.x86_64.rpm 177 kB/s | 22 kB 00:00 2024-09-16T07:52:04.113 INFO:teuthology.orchestra.run.smithi187.stdout:(21/37): python3-cheroot-8.6.0-4.el9.noarch.rpm 2.9 MB/s | 172 kB 00:00 2024-09-16T07:52:04.137 INFO:teuthology.orchestra.run.smithi187.stdout:(22/37): python3-jaraco-8.2.1-3.el9.noarch.rpm 459 kB/s | 11 kB 00:00 2024-09-16T07:52:04.162 INFO:teuthology.orchestra.run.smithi187.stdout:(23/37): python3-jaraco-classes-3.2.1-5.el9.noa 708 kB/s | 18 kB 00:00 2024-09-16T07:52:04.187 INFO:teuthology.orchestra.run.smithi187.stdout:(24/37): python3-jaraco-collections-3.0.0-8.el9 932 kB/s | 23 kB 00:00 2024-09-16T07:52:04.213 INFO:teuthology.orchestra.run.smithi187.stdout:(25/37): python3-jaraco-functools-3.5.0-2.el9.n 777 kB/s | 19 kB 00:00 2024-09-16T07:52:04.238 INFO:teuthology.orchestra.run.smithi187.stdout:(26/37): python3-cherrypy-18.6.1-2.el9.noarch.r 2.2 MB/s | 358 kB 00:00 2024-09-16T07:52:04.263 INFO:teuthology.orchestra.run.smithi187.stdout:(27/37): python3-jaraco-text-3.2.0-6.el9.noarch 391 kB/s | 20 kB 00:00 2024-09-16T07:52:04.289 INFO:teuthology.orchestra.run.smithi187.stdout:(28/37): python3-logutils-0.3.5-21.el9.noarch.r 1.8 MB/s | 46 kB 00:00 2024-09-16T07:52:04.314 INFO:teuthology.orchestra.run.smithi187.stdout:(29/37): python3-more-itertools-8.12.0-2.el9.no 1.5 MB/s | 79 kB 00:00 2024-09-16T07:52:04.356 INFO:teuthology.orchestra.run.smithi187.stdout:(30/37): python3-portend-3.1.0-2.el9.noarch.rpm 396 kB/s | 16 kB 00:00 2024-09-16T07:52:04.381 INFO:teuthology.orchestra.run.smithi187.stdout:(31/37): python3-pecan-1.4.2-3.el9.noarch.rpm 2.9 MB/s | 272 kB 00:00 2024-09-16T07:52:04.406 INFO:teuthology.orchestra.run.smithi187.stdout:(32/37): python3-tempora-5.0.0-2.el9.noarch.rpm 1.4 MB/s | 36 kB 00:00 2024-09-16T07:52:04.431 INFO:teuthology.orchestra.run.smithi187.stdout:(33/37): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.2 MB/s | 90 kB 00:00 2024-09-16T07:52:04.457 INFO:teuthology.orchestra.run.smithi187.stdout:(34/37): python3-webob-1.8.8-2.el9.noarch.rpm 4.5 MB/s | 230 kB 00:00 2024-09-16T07:52:04.482 INFO:teuthology.orchestra.run.smithi187.stdout:(35/37): python3-zc-lockfile-2.0-10.el9.noarch. 801 kB/s | 20 kB 00:00 2024-09-16T07:52:04.507 INFO:teuthology.orchestra.run.smithi187.stdout:(36/37): python3-werkzeug-2.0.3-3.el9.1.noarch. 5.5 MB/s | 427 kB 00:00 2024-09-16T07:52:05.749 INFO:teuthology.orchestra.run.smithi187.stdout:(37/37): ceph-osd-19.3.0-4879.g81fedcb3.el9.x86 29 MB/s | 70 MB 00:02 2024-09-16T07:52:05.757 INFO:teuthology.orchestra.run.smithi187.stdout:-------------------------------------------------------------------------------- 2024-09-16T07:52:05.757 INFO:teuthology.orchestra.run.smithi187.stdout:Total 26 MB/s | 84 MB 00:03 2024-09-16T07:52:05.901 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction check 2024-09-16T07:52:05.941 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction check succeeded. 2024-09-16T07:52:05.941 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction test 2024-09-16T07:52:06.685 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-test-2:19.3.0-4879.g81fedcb3.el9.x86_64 9/9 2024-09-16T07:52:06.686 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-test-2:19.3.0-4879.g81fedcb3.el9.x86_64 1/9 2024-09-16T07:52:06.686 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libcephsqlite-2:19.3.0-4879.g81fedcb3.el9.x86_64 2/9 2024-09-16T07:52:06.686 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : lksctp-tools-1.0.19-2.el9.x86_64 3/9 2024-09-16T07:52:06.686 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libasan8-14.2.1-1.2.el9.x86_64 4/9 2024-09-16T07:52:06.686 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libubsan-11.5.0-2.el9.x86_64 5/9 2024-09-16T07:52:06.686 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 6/9 2024-09-16T07:52:06.686 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 7/9 2024-09-16T07:52:06.686 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 8/9 2024-09-16T07:52:06.722 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction test succeeded. 2024-09-16T07:52:06.723 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction 2024-09-16T07:52:07.127 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : yaml-cpp-0.6.3-5.el9.x86_64 9/9 2024-09-16T07:52:07.128 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-16T07:52:07.128 INFO:teuthology.orchestra.run.smithi027.stdout:Installed: 2024-09-16T07:52:07.128 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-test-2:19.3.0-4879.g81fedcb3.el9.x86_64 2024-09-16T07:52:07.128 INFO:teuthology.orchestra.run.smithi027.stdout: libasan8-14.2.1-1.2.el9.x86_64 2024-09-16T07:52:07.128 INFO:teuthology.orchestra.run.smithi027.stdout: libcephsqlite-2:19.3.0-4879.g81fedcb3.el9.x86_64 2024-09-16T07:52:07.128 INFO:teuthology.orchestra.run.smithi027.stdout: libubsan-11.5.0-2.el9.x86_64 2024-09-16T07:52:07.128 INFO:teuthology.orchestra.run.smithi027.stdout: lksctp-tools-1.0.19-2.el9.x86_64 2024-09-16T07:52:07.128 INFO:teuthology.orchestra.run.smithi027.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-16T07:52:07.128 INFO:teuthology.orchestra.run.smithi027.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-16T07:52:07.128 INFO:teuthology.orchestra.run.smithi027.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-16T07:52:07.128 INFO:teuthology.orchestra.run.smithi027.stdout: yaml-cpp-0.6.3-5.el9.x86_64 2024-09-16T07:52:07.128 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-16T07:52:07.128 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-09-16T07:52:07.442 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install ceph 2024-09-16T07:52:08.028 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:00:56 ago on Mon 16 Sep 2024 07:51:12 AM UTC. 2024-09-16T07:52:08.177 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-09-16T07:52:08.180 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-09-16T07:52:08.180 INFO:teuthology.orchestra.run.smithi027.stdout: Package Arch Version Repository Size 2024-09-16T07:52:08.180 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-09-16T07:52:08.180 INFO:teuthology.orchestra.run.smithi027.stdout:Installing: 2024-09-16T07:52:08.180 INFO:teuthology.orchestra.run.smithi027.stdout: ceph x86_64 2:19.3.0-4879.g81fedcb3.el9 ceph 6.5 k 2024-09-16T07:52:08.181 INFO:teuthology.orchestra.run.smithi027.stdout:Installing dependencies: 2024-09-16T07:52:08.181 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mds x86_64 2:19.3.0-4879.g81fedcb3.el9 ceph 2.2 M 2024-09-16T07:52:08.181 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr x86_64 2:19.3.0-4879.g81fedcb3.el9 ceph 1.6 M 2024-09-16T07:52:08.181 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-modules-core noarch 2:19.3.0-4879.g81fedcb3.el9 ceph-noarch 291 k 2024-09-16T07:52:08.181 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mon x86_64 2:19.3.0-4879.g81fedcb3.el9 ceph 5.1 M 2024-09-16T07:52:08.181 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-osd x86_64 2:19.3.0-4879.g81fedcb3.el9 ceph 70 M 2024-09-16T07:52:08.181 INFO:teuthology.orchestra.run.smithi027.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-16T07:52:08.181 INFO:teuthology.orchestra.run.smithi027.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-16T07:52:08.181 INFO:teuthology.orchestra.run.smithi027.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-16T07:52:08.181 INFO:teuthology.orchestra.run.smithi027.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-16T07:52:08.181 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-16T07:52:08.181 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-16T07:52:08.182 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-16T07:52:08.182 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-16T07:52:08.182 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-16T07:52:08.182 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-16T07:52:08.182 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jaraco-collections 2024-09-16T07:52:08.182 INFO:teuthology.orchestra.run.smithi027.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-09-16T07:52:08.182 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-16T07:52:08.182 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-16T07:52:08.182 INFO:teuthology.orchestra.run.smithi027.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-16T07:52:08.182 INFO:teuthology.orchestra.run.smithi027.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-16T07:52:08.182 INFO:teuthology.orchestra.run.smithi027.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-16T07:52:08.182 INFO:teuthology.orchestra.run.smithi027.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-16T07:52:08.182 INFO:teuthology.orchestra.run.smithi027.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-16T07:52:08.182 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-16T07:52:08.183 INFO:teuthology.orchestra.run.smithi027.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-16T07:52:08.183 INFO:teuthology.orchestra.run.smithi027.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-16T07:52:08.183 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-16T07:52:08.183 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-16T07:52:08.183 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-16T07:52:08.183 INFO:teuthology.orchestra.run.smithi027.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-16T07:52:08.183 INFO:teuthology.orchestra.run.smithi027.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-16T07:52:08.183 INFO:teuthology.orchestra.run.smithi027.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-16T07:52:08.183 INFO:teuthology.orchestra.run.smithi027.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-16T07:52:08.183 INFO:teuthology.orchestra.run.smithi027.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-16T07:52:08.183 INFO:teuthology.orchestra.run.smithi027.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-16T07:52:08.183 INFO:teuthology.orchestra.run.smithi027.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-16T07:52:08.183 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-16T07:52:08.183 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2024-09-16T07:52:08.184 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-09-16T07:52:08.184 INFO:teuthology.orchestra.run.smithi027.stdout:Install 37 Packages 2024-09-16T07:52:08.184 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-16T07:52:08.184 INFO:teuthology.orchestra.run.smithi027.stdout:Total download size: 84 M 2024-09-16T07:52:08.184 INFO:teuthology.orchestra.run.smithi027.stdout:Installed size: 897 M 2024-09-16T07:52:08.185 INFO:teuthology.orchestra.run.smithi027.stdout:Downloading Packages: 2024-09-16T07:52:08.380 INFO:teuthology.orchestra.run.smithi187.stdout: Preparing : 1/1 2024-09-16T07:52:08.443 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/37 2024-09-16T07:52:08.477 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/37 2024-09-16T07:52:08.504 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/37 2024-09-16T07:52:08.578 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/37 2024-09-16T07:52:08.620 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/37 2024-09-16T07:52:08.652 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/37 2024-09-16T07:52:08.746 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/37 2024-09-16T07:52:08.779 INFO:teuthology.orchestra.run.smithi027.stdout:(1/37): ceph-19.3.0-4879.g81fedcb3.el9.x86_64.r 27 kB/s | 6.5 kB 00:00 2024-09-16T07:52:08.832 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/37 2024-09-16T07:52:08.881 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 9/37 2024-09-16T07:52:08.915 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/37 2024-09-16T07:52:08.945 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/37 2024-09-16T07:52:08.979 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/37 2024-09-16T07:52:09.013 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/37 2024-09-16T07:52:09.053 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/37 2024-09-16T07:52:09.088 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/37 2024-09-16T07:52:09.131 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/37 2024-09-16T07:52:09.163 INFO:teuthology.orchestra.run.smithi027.stdout:(2/37): ceph-mgr-19.3.0-4879.g81fedcb3.el9.x86_ 2.5 MB/s | 1.6 MB 00:00 2024-09-16T07:52:09.180 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/37 2024-09-16T07:52:09.211 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-packaging-20.9-5.el9.noarch 18/37 2024-09-16T07:52:09.286 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 19/37 2024-09-16T07:52:09.330 INFO:teuthology.orchestra.run.smithi027.stdout:(3/37): ceph-mds-19.3.0-4879.g81fedcb3.el9.x86_ 2.8 MB/s | 2.2 MB 00:00 2024-09-16T07:52:09.380 INFO:teuthology.orchestra.run.smithi027.stdout:(4/37): ceph-mgr-modules-core-19.3.0-4879.g81fe 5.7 MB/s | 291 kB 00:00 2024-09-16T07:52:09.427 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 20/37 2024-09-16T07:52:09.504 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 21/37 2024-09-16T07:52:09.547 INFO:teuthology.orchestra.run.smithi027.stdout:(5/37): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.5 MB/s | 253 kB 00:00 2024-09-16T07:52:09.556 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 22/37 2024-09-16T07:52:09.595 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 23/37 2024-09-16T07:52:09.652 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-ply-3.11-14.el9.noarch 24/37 2024-09-16T07:52:09.664 INFO:teuthology.orchestra.run.smithi027.stdout:(6/37): ceph-mon-19.3.0-4879.g81fedcb3.el9.x86_ 5.8 MB/s | 5.1 MB 00:00 2024-09-16T07:52:09.706 INFO:teuthology.orchestra.run.smithi027.stdout:(7/37): python3-cryptography-36.0.1-4.el9.x86_6 7.7 MB/s | 1.2 MB 00:00 2024-09-16T07:52:09.708 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 25/37 2024-09-16T07:52:09.757 INFO:teuthology.orchestra.run.smithi027.stdout:(8/37): python3-pycparser-2.20-6.el9.noarch.rpm 2.7 MB/s | 135 kB 00:00 2024-09-16T07:52:09.807 INFO:teuthology.orchestra.run.smithi027.stdout:(9/37): python3-requests-2.25.1-8.el9.noarch.rp 2.4 MB/s | 125 kB 00:00 2024-09-16T07:52:09.832 INFO:teuthology.orchestra.run.smithi027.stdout:(10/37): python3-ply-3.11-14.el9.noarch.rpm 636 kB/s | 106 kB 00:00 2024-09-16T07:52:09.866 INFO:teuthology.orchestra.run.smithi027.stdout:(11/37): python3-urllib3-1.26.5-6.el9.noarch.rp 3.6 MB/s | 215 kB 00:00 2024-09-16T07:52:09.898 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 26/37 2024-09-16T07:52:09.957 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 27/37 2024-09-16T07:52:09.966 INFO:teuthology.orchestra.run.smithi027.stdout:(12/37): lua-5.4.4-4.el9.x86_64.rpm 1.4 MB/s | 188 kB 00:00 2024-09-16T07:52:09.991 INFO:teuthology.orchestra.run.smithi027.stdout:(13/37): python3-mako-1.1.4-6.el9.noarch.rpm 1.3 MB/s | 172 kB 00:00 2024-09-16T07:52:10.036 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 28/37 2024-09-16T07:52:10.042 INFO:teuthology.orchestra.run.smithi027.stdout:(14/37): python3-markupsafe-1.1.1-12.el9.x86_64 462 kB/s | 35 kB 00:00 2024-09-16T07:52:10.067 INFO:teuthology.orchestra.run.smithi027.stdout:(15/37): python3-packaging-20.9-5.el9.noarch.rp 1.0 MB/s | 77 kB 00:00 2024-09-16T07:52:10.125 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 29/37 2024-09-16T07:52:10.126 INFO:teuthology.orchestra.run.smithi027.stdout:(16/37): python3-pytz-2021.1-5.el9.noarch.rpm 611 kB/s | 51 kB 00:00 2024-09-16T07:52:10.151 INFO:teuthology.orchestra.run.smithi027.stdout:(17/37): python3-toml-0.10.2-6.el9.noarch.rpm 497 kB/s | 42 kB 00:00 2024-09-16T07:52:10.173 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 30/37 2024-09-16T07:52:10.176 INFO:teuthology.orchestra.run.smithi027.stdout:(18/37): lua-devel-5.4.4-4.el9.x86_64.rpm 445 kB/s | 22 kB 00:00 2024-09-16T07:52:10.218 INFO:teuthology.orchestra.run.smithi027.stdout:(19/37): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 1.0 MB/s | 43 kB 00:00 2024-09-16T07:52:10.243 INFO:teuthology.orchestra.run.smithi027.stdout:(20/37): python3-cheroot-8.6.0-4.el9.noarch.rpm 7.2 MB/s | 172 kB 00:00 2024-09-16T07:52:10.269 INFO:teuthology.orchestra.run.smithi027.stdout:(21/37): luarocks-3.9.2-5.el9.noarch.rpm 1.3 MB/s | 151 kB 00:00 2024-09-16T07:52:10.294 INFO:teuthology.orchestra.run.smithi027.stdout:(22/37): python3-jaraco-8.2.1-3.el9.noarch.rpm 425 kB/s | 11 kB 00:00 2024-09-16T07:52:10.305 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 31/37 2024-09-16T07:52:10.319 INFO:teuthology.orchestra.run.smithi027.stdout:(23/37): python3-jaraco-classes-3.2.1-5.el9.noa 708 kB/s | 18 kB 00:00 2024-09-16T07:52:10.345 INFO:teuthology.orchestra.run.smithi027.stdout:(24/37): python3-cherrypy-18.6.1-2.el9.noarch.r 3.5 MB/s | 358 kB 00:00 2024-09-16T07:52:10.370 INFO:teuthology.orchestra.run.smithi027.stdout:(25/37): python3-jaraco-collections-3.0.0-8.el9 459 kB/s | 23 kB 00:00 2024-09-16T07:52:10.395 INFO:teuthology.orchestra.run.smithi027.stdout:(26/37): python3-jaraco-functools-3.5.0-2.el9.n 387 kB/s | 19 kB 00:00 2024-09-16T07:52:10.421 INFO:teuthology.orchestra.run.smithi027.stdout:(27/37): python3-jaraco-text-3.2.0-6.el9.noarch 392 kB/s | 20 kB 00:00 2024-09-16T07:52:10.446 INFO:teuthology.orchestra.run.smithi027.stdout:(28/37): python3-logutils-0.3.5-21.el9.noarch.r 918 kB/s | 46 kB 00:00 2024-09-16T07:52:10.471 INFO:teuthology.orchestra.run.smithi027.stdout:(29/37): python3-more-itertools-8.12.0-2.el9.no 1.5 MB/s | 79 kB 00:00 2024-09-16T07:52:10.504 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-mgr-modules-core-2:19.3.0-4879.g81fedcb3.el9. 32/37 2024-09-16T07:52:10.533 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-mgr-2:19.3.0-4879.g81fedcb3.el9.x86_64 33/37 2024-09-16T07:52:10.559 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-mgr-2:19.3.0-4879.g81fedcb3.el9.x86_64 33/37 2024-09-16T07:52:10.559 INFO:teuthology.orchestra.run.smithi187.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T07:52:10.559 INFO:teuthology.orchestra.run.smithi187.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-16T07:52:10.559 INFO:teuthology.orchestra.run.smithi187.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-16T07:52:10.559 INFO:teuthology.orchestra.run.smithi187.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-16T07:52:10.559 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-16T07:52:10.622 INFO:teuthology.orchestra.run.smithi027.stdout:(30/37): python3-portend-3.1.0-2.el9.noarch.rpm 109 kB/s | 16 kB 00:00 2024-09-16T07:52:10.656 INFO:teuthology.orchestra.run.smithi027.stdout:(31/37): python3-pecan-1.4.2-3.el9.noarch.rpm 1.3 MB/s | 272 kB 00:00 2024-09-16T07:52:10.682 INFO:teuthology.orchestra.run.smithi027.stdout:(32/37): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.5 MB/s | 90 kB 00:00 2024-09-16T07:52:10.707 INFO:teuthology.orchestra.run.smithi027.stdout:(33/37): python3-tempora-5.0.0-2.el9.noarch.rpm 717 kB/s | 36 kB 00:00 2024-09-16T07:52:10.732 INFO:teuthology.orchestra.run.smithi027.stdout:(34/37): python3-webob-1.8.8-2.el9.noarch.rpm 4.5 MB/s | 230 kB 00:00 2024-09-16T07:52:10.758 INFO:teuthology.orchestra.run.smithi027.stdout:(35/37): python3-zc-lockfile-2.0-10.el9.noarch. 800 kB/s | 20 kB 00:00 2024-09-16T07:52:10.783 INFO:teuthology.orchestra.run.smithi027.stdout:(36/37): python3-werkzeug-2.0.3-3.el9.1.noarch. 5.5 MB/s | 427 kB 00:00 2024-09-16T07:52:11.925 INFO:teuthology.orchestra.run.smithi027.stdout:(37/37): ceph-osd-19.3.0-4879.g81fedcb3.el9.x86 26 MB/s | 70 MB 00:02 2024-09-16T07:52:11.929 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2024-09-16T07:52:11.929 INFO:teuthology.orchestra.run.smithi027.stdout:Total 22 MB/s | 84 MB 00:03 2024-09-16T07:52:12.088 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction check 2024-09-16T07:52:12.135 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction check succeeded. 2024-09-16T07:52:12.135 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction test 2024-09-16T07:52:12.961 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction test succeeded. 2024-09-16T07:52:12.961 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction 2024-09-16T07:52:14.673 INFO:teuthology.orchestra.run.smithi027.stdout: Preparing : 1/1 2024-09-16T07:52:14.736 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/37 2024-09-16T07:52:14.769 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/37 2024-09-16T07:52:14.796 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/37 2024-09-16T07:52:14.871 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/37 2024-09-16T07:52:14.912 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/37 2024-09-16T07:52:14.954 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/37 2024-09-16T07:52:15.059 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/37 2024-09-16T07:52:15.144 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/37 2024-09-16T07:52:15.191 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 9/37 2024-09-16T07:52:15.225 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/37 2024-09-16T07:52:15.254 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/37 2024-09-16T07:52:15.297 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/37 2024-09-16T07:52:15.339 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/37 2024-09-16T07:52:15.389 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/37 2024-09-16T07:52:15.440 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/37 2024-09-16T07:52:15.474 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/37 2024-09-16T07:52:15.524 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/37 2024-09-16T07:52:15.563 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-packaging-20.9-5.el9.noarch 18/37 2024-09-16T07:52:15.640 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 19/37 2024-09-16T07:52:15.800 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 20/37 2024-09-16T07:52:15.882 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 21/37 2024-09-16T07:52:15.935 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 22/37 2024-09-16T07:52:15.980 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 23/37 2024-09-16T07:52:16.039 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-ply-3.11-14.el9.noarch 24/37 2024-09-16T07:52:16.102 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 25/37 2024-09-16T07:52:16.306 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 26/37 2024-09-16T07:52:16.359 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 27/37 2024-09-16T07:52:16.440 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 28/37 2024-09-16T07:52:16.538 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 29/37 2024-09-16T07:52:16.580 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 30/37 2024-09-16T07:52:16.720 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 31/37 2024-09-16T07:52:16.928 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-mgr-modules-core-2:19.3.0-4879.g81fedcb3.el9. 32/37 2024-09-16T07:52:16.959 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-mgr-2:19.3.0-4879.g81fedcb3.el9.x86_64 33/37 2024-09-16T07:52:16.987 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-mgr-2:19.3.0-4879.g81fedcb3.el9.x86_64 33/37 2024-09-16T07:52:16.987 INFO:teuthology.orchestra.run.smithi027.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T07:52:16.987 INFO:teuthology.orchestra.run.smithi027.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-16T07:52:16.987 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-16T07:52:16.987 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-16T07:52:16.987 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-16T07:52:21.141 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-osd-2:19.3.0-4879.g81fedcb3.el9.x86_64 34/37 2024-09-16T07:52:21.168 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-osd-2:19.3.0-4879.g81fedcb3.el9.x86_64 34/37 2024-09-16T07:52:21.168 INFO:teuthology.orchestra.run.smithi187.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T07:52:21.168 INFO:teuthology.orchestra.run.smithi187.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-16T07:52:21.169 INFO:teuthology.orchestra.run.smithi187.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-16T07:52:21.169 INFO:teuthology.orchestra.run.smithi187.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-16T07:52:21.169 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-16T07:52:21.717 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-mon-2:19.3.0-4879.g81fedcb3.el9.x86_64 35/37 2024-09-16T07:52:21.740 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-mon-2:19.3.0-4879.g81fedcb3.el9.x86_64 35/37 2024-09-16T07:52:21.740 INFO:teuthology.orchestra.run.smithi187.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T07:52:21.740 INFO:teuthology.orchestra.run.smithi187.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-16T07:52:21.740 INFO:teuthology.orchestra.run.smithi187.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-16T07:52:21.740 INFO:teuthology.orchestra.run.smithi187.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-16T07:52:21.740 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-16T07:52:21.984 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-mds-2:19.3.0-4879.g81fedcb3.el9.x86_64 36/37 2024-09-16T07:52:22.007 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-mds-2:19.3.0-4879.g81fedcb3.el9.x86_64 36/37 2024-09-16T07:52:22.007 INFO:teuthology.orchestra.run.smithi187.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T07:52:22.007 INFO:teuthology.orchestra.run.smithi187.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-16T07:52:22.007 INFO:teuthology.orchestra.run.smithi187.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-16T07:52:22.008 INFO:teuthology.orchestra.run.smithi187.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-16T07:52:22.008 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-16T07:52:22.072 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-2:19.3.0-4879.g81fedcb3.el9.x86_64 37/37 2024-09-16T07:52:28.201 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-osd-2:19.3.0-4879.g81fedcb3.el9.x86_64 34/37 2024-09-16T07:52:28.230 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-osd-2:19.3.0-4879.g81fedcb3.el9.x86_64 34/37 2024-09-16T07:52:28.230 INFO:teuthology.orchestra.run.smithi027.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T07:52:28.230 INFO:teuthology.orchestra.run.smithi027.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-16T07:52:28.230 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-16T07:52:28.230 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-16T07:52:28.230 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-16T07:52:28.740 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-2:19.3.0-4879.g81fedcb3.el9.x86_64 37/37 2024-09-16T07:52:28.740 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-2:19.3.0-4879.g81fedcb3.el9.x86_64 1/37 2024-09-16T07:52:28.740 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-mds-2:19.3.0-4879.g81fedcb3.el9.x86_64 2/37 2024-09-16T07:52:28.741 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-mgr-2:19.3.0-4879.g81fedcb3.el9.x86_64 3/37 2024-09-16T07:52:28.741 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-mon-2:19.3.0-4879.g81fedcb3.el9.x86_64 4/37 2024-09-16T07:52:28.741 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-osd-2:19.3.0-4879.g81fedcb3.el9.x86_64 5/37 2024-09-16T07:52:28.741 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-mgr-modules-core-2:19.3.0-4879.g81fedcb3.el9. 6/37 2024-09-16T07:52:28.741 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/37 2024-09-16T07:52:28.741 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/37 2024-09-16T07:52:28.741 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/37 2024-09-16T07:52:28.741 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/37 2024-09-16T07:52:28.741 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/37 2024-09-16T07:52:28.741 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/37 2024-09-16T07:52:28.741 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/37 2024-09-16T07:52:28.741 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/37 2024-09-16T07:52:28.741 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/37 2024-09-16T07:52:28.742 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 16/37 2024-09-16T07:52:28.742 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 17/37 2024-09-16T07:52:28.742 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 18/37 2024-09-16T07:52:28.742 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 19/37 2024-09-16T07:52:28.742 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 20/37 2024-09-16T07:52:28.742 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 21/37 2024-09-16T07:52:28.742 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 22/37 2024-09-16T07:52:28.742 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 23/37 2024-09-16T07:52:28.742 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 24/37 2024-09-16T07:52:28.742 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 25/37 2024-09-16T07:52:28.743 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 26/37 2024-09-16T07:52:28.743 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 27/37 2024-09-16T07:52:28.743 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 28/37 2024-09-16T07:52:28.743 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 29/37 2024-09-16T07:52:28.743 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 30/37 2024-09-16T07:52:28.743 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 31/37 2024-09-16T07:52:28.743 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 32/37 2024-09-16T07:52:28.743 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 33/37 2024-09-16T07:52:28.743 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 34/37 2024-09-16T07:52:28.743 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 35/37 2024-09-16T07:52:28.743 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 36/37 2024-09-16T07:52:28.830 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-mon-2:19.3.0-4879.g81fedcb3.el9.x86_64 35/37 2024-09-16T07:52:28.857 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-mon-2:19.3.0-4879.g81fedcb3.el9.x86_64 35/37 2024-09-16T07:52:28.857 INFO:teuthology.orchestra.run.smithi027.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T07:52:28.857 INFO:teuthology.orchestra.run.smithi027.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-16T07:52:28.857 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-16T07:52:28.857 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-16T07:52:28.857 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-16T07:52:29.140 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-mds-2:19.3.0-4879.g81fedcb3.el9.x86_64 36/37 2024-09-16T07:52:29.165 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-mds-2:19.3.0-4879.g81fedcb3.el9.x86_64 36/37 2024-09-16T07:52:29.165 INFO:teuthology.orchestra.run.smithi027.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T07:52:29.165 INFO:teuthology.orchestra.run.smithi027.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-16T07:52:29.165 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-16T07:52:29.165 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-16T07:52:29.165 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-16T07:52:29.230 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-2:19.3.0-4879.g81fedcb3.el9.x86_64 37/37 2024-09-16T07:52:29.830 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 37/37 2024-09-16T07:52:29.831 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-16T07:52:29.831 INFO:teuthology.orchestra.run.smithi187.stdout:Installed: 2024-09-16T07:52:29.831 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-2:19.3.0-4879.g81fedcb3.el9.x86_64 2024-09-16T07:52:29.831 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mds-2:19.3.0-4879.g81fedcb3.el9.x86_64 2024-09-16T07:52:29.831 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mgr-2:19.3.0-4879.g81fedcb3.el9.x86_64 2024-09-16T07:52:29.831 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mgr-modules-core-2:19.3.0-4879.g81fedcb3.el9.noarch 2024-09-16T07:52:29.831 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mon-2:19.3.0-4879.g81fedcb3.el9.x86_64 2024-09-16T07:52:29.831 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-osd-2:19.3.0-4879.g81fedcb3.el9.x86_64 2024-09-16T07:52:29.831 INFO:teuthology.orchestra.run.smithi187.stdout: lua-5.4.4-4.el9.x86_64 2024-09-16T07:52:29.831 INFO:teuthology.orchestra.run.smithi187.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-16T07:52:29.831 INFO:teuthology.orchestra.run.smithi187.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-16T07:52:29.832 INFO:teuthology.orchestra.run.smithi187.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-16T07:52:29.832 INFO:teuthology.orchestra.run.smithi187.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-16T07:52:29.832 INFO:teuthology.orchestra.run.smithi187.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-16T07:52:29.832 INFO:teuthology.orchestra.run.smithi187.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-16T07:52:29.832 INFO:teuthology.orchestra.run.smithi187.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-16T07:52:29.832 INFO:teuthology.orchestra.run.smithi187.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-16T07:52:29.832 INFO:teuthology.orchestra.run.smithi187.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-16T07:52:29.832 INFO:teuthology.orchestra.run.smithi187.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-16T07:52:29.832 INFO:teuthology.orchestra.run.smithi187.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-16T07:52:29.832 INFO:teuthology.orchestra.run.smithi187.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-16T07:52:29.832 INFO:teuthology.orchestra.run.smithi187.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-16T07:52:29.832 INFO:teuthology.orchestra.run.smithi187.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-16T07:52:29.832 INFO:teuthology.orchestra.run.smithi187.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-16T07:52:29.833 INFO:teuthology.orchestra.run.smithi187.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-16T07:52:29.833 INFO:teuthology.orchestra.run.smithi187.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-16T07:52:29.833 INFO:teuthology.orchestra.run.smithi187.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-16T07:52:29.833 INFO:teuthology.orchestra.run.smithi187.stdout: python3-ply-3.11-14.el9.noarch 2024-09-16T07:52:29.833 INFO:teuthology.orchestra.run.smithi187.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-16T07:52:29.833 INFO:teuthology.orchestra.run.smithi187.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-16T07:52:29.833 INFO:teuthology.orchestra.run.smithi187.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-16T07:52:29.833 INFO:teuthology.orchestra.run.smithi187.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-16T07:52:29.833 INFO:teuthology.orchestra.run.smithi187.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-16T07:52:29.833 INFO:teuthology.orchestra.run.smithi187.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-16T07:52:29.833 INFO:teuthology.orchestra.run.smithi187.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-16T07:52:29.833 INFO:teuthology.orchestra.run.smithi187.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-16T07:52:29.833 INFO:teuthology.orchestra.run.smithi187.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-16T07:52:29.834 INFO:teuthology.orchestra.run.smithi187.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-16T07:52:29.834 INFO:teuthology.orchestra.run.smithi187.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-16T07:52:29.834 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-16T07:52:29.834 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2024-09-16T07:52:30.117 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum -y install ceph-base 2024-09-16T07:52:30.707 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:01:21 ago on Mon 16 Sep 2024 07:51:09 AM UTC. 2024-09-16T07:52:30.802 INFO:teuthology.orchestra.run.smithi187.stdout:Package ceph-base-2:19.3.0-4879.g81fedcb3.el9.x86_64 is already installed. 2024-09-16T07:52:30.846 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2024-09-16T07:52:30.847 INFO:teuthology.orchestra.run.smithi187.stdout:Nothing to do. 2024-09-16T07:52:30.847 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2024-09-16T07:52:30.920 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum -y install cephadm 2024-09-16T07:52:31.493 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:01:22 ago on Mon 16 Sep 2024 07:51:09 AM UTC. 2024-09-16T07:52:31.625 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2024-09-16T07:52:31.626 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-09-16T07:52:31.626 INFO:teuthology.orchestra.run.smithi187.stdout: Package Arch Version Repository Size 2024-09-16T07:52:31.626 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-09-16T07:52:31.626 INFO:teuthology.orchestra.run.smithi187.stdout:Installing: 2024-09-16T07:52:31.626 INFO:teuthology.orchestra.run.smithi187.stdout: cephadm noarch 2:19.3.0-4879.g81fedcb3.el9 ceph-noarch 785 k 2024-09-16T07:52:31.626 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-16T07:52:31.626 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction Summary 2024-09-16T07:52:31.627 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-09-16T07:52:31.627 INFO:teuthology.orchestra.run.smithi187.stdout:Install 1 Package 2024-09-16T07:52:31.627 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-16T07:52:31.627 INFO:teuthology.orchestra.run.smithi187.stdout:Total download size: 785 k 2024-09-16T07:52:31.627 INFO:teuthology.orchestra.run.smithi187.stdout:Installed size: 791 k 2024-09-16T07:52:31.627 INFO:teuthology.orchestra.run.smithi187.stdout:Downloading Packages: 2024-09-16T07:52:31.940 INFO:teuthology.orchestra.run.smithi187.stdout:cephadm-19.3.0-4879.g81fedcb3.el9.noarch.rpm 2.4 MB/s | 785 kB 00:00 2024-09-16T07:52:31.941 INFO:teuthology.orchestra.run.smithi187.stdout:-------------------------------------------------------------------------------- 2024-09-16T07:52:31.941 INFO:teuthology.orchestra.run.smithi187.stdout:Total 2.4 MB/s | 785 kB 00:00 2024-09-16T07:52:31.942 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction check 2024-09-16T07:52:31.947 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction check succeeded. 2024-09-16T07:52:31.947 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction test 2024-09-16T07:52:31.953 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction test succeeded. 2024-09-16T07:52:31.953 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction 2024-09-16T07:52:32.023 INFO:teuthology.orchestra.run.smithi187.stdout: Preparing : 1/1 2024-09-16T07:52:32.502 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: cephadm-2:19.3.0-4879.g81fedcb3.el9.noarch 1/1 2024-09-16T07:52:32.595 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : cephadm-2:19.3.0-4879.g81fedcb3.el9.noarch 1/1 2024-09-16T07:52:33.000 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: cephadm-2:19.3.0-4879.g81fedcb3.el9.noarch 1/1 2024-09-16T07:52:33.258 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : cephadm-2:19.3.0-4879.g81fedcb3.el9.noarch 1/1 2024-09-16T07:52:33.258 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-16T07:52:33.258 INFO:teuthology.orchestra.run.smithi187.stdout:Installed: 2024-09-16T07:52:33.258 INFO:teuthology.orchestra.run.smithi187.stdout: cephadm-2:19.3.0-4879.g81fedcb3.el9.noarch 2024-09-16T07:52:33.258 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-16T07:52:33.258 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2024-09-16T07:52:33.462 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum -y install ceph-immutable-object-cache 2024-09-16T07:52:34.043 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:01:25 ago on Mon 16 Sep 2024 07:51:09 AM UTC. 2024-09-16T07:52:34.179 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2024-09-16T07:52:34.179 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-09-16T07:52:34.180 INFO:teuthology.orchestra.run.smithi187.stdout: Package Arch Version Repo Size 2024-09-16T07:52:34.180 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-09-16T07:52:34.180 INFO:teuthology.orchestra.run.smithi187.stdout:Installing: 2024-09-16T07:52:34.180 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-immutable-object-cache x86_64 2:19.3.0-4879.g81fedcb3.el9 ceph 158 k 2024-09-16T07:52:34.180 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-16T07:52:34.180 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction Summary 2024-09-16T07:52:34.180 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-09-16T07:52:34.180 INFO:teuthology.orchestra.run.smithi187.stdout:Install 1 Package 2024-09-16T07:52:34.180 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-16T07:52:34.180 INFO:teuthology.orchestra.run.smithi187.stdout:Total download size: 158 k 2024-09-16T07:52:34.180 INFO:teuthology.orchestra.run.smithi187.stdout:Installed size: 644 k 2024-09-16T07:52:34.180 INFO:teuthology.orchestra.run.smithi187.stdout:Downloading Packages: 2024-09-16T07:52:34.397 INFO:teuthology.orchestra.run.smithi187.stdout:ceph-immutable-object-cache-19.3.0-4879.g81fedc 730 kB/s | 158 kB 00:00 2024-09-16T07:52:34.397 INFO:teuthology.orchestra.run.smithi187.stdout:-------------------------------------------------------------------------------- 2024-09-16T07:52:34.398 INFO:teuthology.orchestra.run.smithi187.stdout:Total 724 kB/s | 158 kB 00:00 2024-09-16T07:52:34.398 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction check 2024-09-16T07:52:34.408 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction check succeeded. 2024-09-16T07:52:34.408 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction test 2024-09-16T07:52:34.466 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction test succeeded. 2024-09-16T07:52:34.466 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction 2024-09-16T07:52:34.627 INFO:teuthology.orchestra.run.smithi187.stdout: Preparing : 1/1 2024-09-16T07:52:34.715 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-immutable-object-cache-2:19.3.0-4879.g81fedcb3. 1/1 2024-09-16T07:52:34.740 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.3.0-4879.g81fedcb3. 1/1 2024-09-16T07:52:34.740 INFO:teuthology.orchestra.run.smithi187.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T07:52:34.741 INFO:teuthology.orchestra.run.smithi187.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-16T07:52:34.741 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-16T07:52:35.657 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-immutable-object-cache-2:19.3.0-4879.g81fedcb3. 1/1 2024-09-16T07:52:35.658 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-16T07:52:35.658 INFO:teuthology.orchestra.run.smithi187.stdout:Installed: 2024-09-16T07:52:35.658 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-immutable-object-cache-2:19.3.0-4879.g81fedcb3.el9.x86_64 2024-09-16T07:52:35.658 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-16T07:52:35.658 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2024-09-16T07:52:35.867 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum -y install ceph-mgr 2024-09-16T07:52:35.925 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-2:19.3.0-4879.g81fedcb3.el9.x86_64 37/37 2024-09-16T07:52:35.925 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-2:19.3.0-4879.g81fedcb3.el9.x86_64 1/37 2024-09-16T07:52:35.925 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-mds-2:19.3.0-4879.g81fedcb3.el9.x86_64 2/37 2024-09-16T07:52:35.925 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-mgr-2:19.3.0-4879.g81fedcb3.el9.x86_64 3/37 2024-09-16T07:52:35.925 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-mon-2:19.3.0-4879.g81fedcb3.el9.x86_64 4/37 2024-09-16T07:52:35.925 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-osd-2:19.3.0-4879.g81fedcb3.el9.x86_64 5/37 2024-09-16T07:52:35.925 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-mgr-modules-core-2:19.3.0-4879.g81fedcb3.el9. 6/37 2024-09-16T07:52:35.926 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/37 2024-09-16T07:52:35.926 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/37 2024-09-16T07:52:35.926 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/37 2024-09-16T07:52:35.926 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/37 2024-09-16T07:52:35.926 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/37 2024-09-16T07:52:35.926 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/37 2024-09-16T07:52:35.926 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/37 2024-09-16T07:52:35.926 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/37 2024-09-16T07:52:35.927 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/37 2024-09-16T07:52:35.927 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 16/37 2024-09-16T07:52:35.927 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 17/37 2024-09-16T07:52:35.927 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 18/37 2024-09-16T07:52:35.927 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 19/37 2024-09-16T07:52:35.927 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 20/37 2024-09-16T07:52:35.927 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 21/37 2024-09-16T07:52:35.927 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 22/37 2024-09-16T07:52:35.927 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 23/37 2024-09-16T07:52:35.927 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 24/37 2024-09-16T07:52:35.927 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 25/37 2024-09-16T07:52:35.927 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 26/37 2024-09-16T07:52:35.927 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 27/37 2024-09-16T07:52:35.928 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 28/37 2024-09-16T07:52:35.928 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 29/37 2024-09-16T07:52:35.928 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 30/37 2024-09-16T07:52:35.928 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 31/37 2024-09-16T07:52:35.928 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 32/37 2024-09-16T07:52:35.928 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 33/37 2024-09-16T07:52:35.928 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 34/37 2024-09-16T07:52:35.928 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 35/37 2024-09-16T07:52:35.929 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 36/37 2024-09-16T07:52:36.433 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:01:27 ago on Mon 16 Sep 2024 07:51:09 AM UTC. 2024-09-16T07:52:36.525 INFO:teuthology.orchestra.run.smithi187.stdout:Package ceph-mgr-2:19.3.0-4879.g81fedcb3.el9.x86_64 is already installed. 2024-09-16T07:52:36.566 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2024-09-16T07:52:36.567 INFO:teuthology.orchestra.run.smithi187.stdout:Nothing to do. 2024-09-16T07:52:36.567 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2024-09-16T07:52:36.639 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum -y install ceph-mgr-dashboard 2024-09-16T07:52:37.086 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 37/37 2024-09-16T07:52:37.086 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-16T07:52:37.086 INFO:teuthology.orchestra.run.smithi027.stdout:Installed: 2024-09-16T07:52:37.087 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-2:19.3.0-4879.g81fedcb3.el9.x86_64 2024-09-16T07:52:37.087 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mds-2:19.3.0-4879.g81fedcb3.el9.x86_64 2024-09-16T07:52:37.087 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-2:19.3.0-4879.g81fedcb3.el9.x86_64 2024-09-16T07:52:37.087 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-modules-core-2:19.3.0-4879.g81fedcb3.el9.noarch 2024-09-16T07:52:37.087 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mon-2:19.3.0-4879.g81fedcb3.el9.x86_64 2024-09-16T07:52:37.087 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-osd-2:19.3.0-4879.g81fedcb3.el9.x86_64 2024-09-16T07:52:37.087 INFO:teuthology.orchestra.run.smithi027.stdout: lua-5.4.4-4.el9.x86_64 2024-09-16T07:52:37.087 INFO:teuthology.orchestra.run.smithi027.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-16T07:52:37.087 INFO:teuthology.orchestra.run.smithi027.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-16T07:52:37.087 INFO:teuthology.orchestra.run.smithi027.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-16T07:52:37.087 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-16T07:52:37.087 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-16T07:52:37.088 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-16T07:52:37.088 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-16T07:52:37.088 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-16T07:52:37.088 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-16T07:52:37.088 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-16T07:52:37.088 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-16T07:52:37.088 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-16T07:52:37.088 INFO:teuthology.orchestra.run.smithi027.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-16T07:52:37.088 INFO:teuthology.orchestra.run.smithi027.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-16T07:52:37.088 INFO:teuthology.orchestra.run.smithi027.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-16T07:52:37.088 INFO:teuthology.orchestra.run.smithi027.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-16T07:52:37.088 INFO:teuthology.orchestra.run.smithi027.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-16T07:52:37.089 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-16T07:52:37.089 INFO:teuthology.orchestra.run.smithi027.stdout: python3-ply-3.11-14.el9.noarch 2024-09-16T07:52:37.089 INFO:teuthology.orchestra.run.smithi027.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-16T07:52:37.089 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-16T07:52:37.089 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-16T07:52:37.089 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-16T07:52:37.089 INFO:teuthology.orchestra.run.smithi027.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-16T07:52:37.089 INFO:teuthology.orchestra.run.smithi027.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-16T07:52:37.089 INFO:teuthology.orchestra.run.smithi027.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-16T07:52:37.089 INFO:teuthology.orchestra.run.smithi027.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-16T07:52:37.089 INFO:teuthology.orchestra.run.smithi027.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-16T07:52:37.089 INFO:teuthology.orchestra.run.smithi027.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-16T07:52:37.089 INFO:teuthology.orchestra.run.smithi027.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-16T07:52:37.090 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-16T07:52:37.090 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-09-16T07:52:37.205 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:01:28 ago on Mon 16 Sep 2024 07:51:09 AM UTC. 2024-09-16T07:52:37.331 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install ceph-base 2024-09-16T07:52:37.340 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2024-09-16T07:52:37.341 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-09-16T07:52:37.341 INFO:teuthology.orchestra.run.smithi187.stdout: Package Arch Version Repository Size 2024-09-16T07:52:37.341 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-09-16T07:52:37.342 INFO:teuthology.orchestra.run.smithi187.stdout:Installing: 2024-09-16T07:52:37.342 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mgr-dashboard noarch 2:19.3.0-4879.g81fedcb3.el9 ceph-noarch 14 M 2024-09-16T07:52:37.342 INFO:teuthology.orchestra.run.smithi187.stdout:Installing dependencies: 2024-09-16T07:52:37.342 INFO:teuthology.orchestra.run.smithi187.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-16T07:52:37.342 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-grafana-dashboards noarch 2:19.3.0-4879.g81fedcb3.el9 ceph-noarch 34 k 2024-09-16T07:52:37.342 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-prometheus-alerts noarch 2:19.3.0-4879.g81fedcb3.el9 ceph-noarch 16 k 2024-09-16T07:52:37.342 INFO:teuthology.orchestra.run.smithi187.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-16T07:52:37.342 INFO:teuthology.orchestra.run.smithi187.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-16T07:52:37.342 INFO:teuthology.orchestra.run.smithi187.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-16T07:52:37.342 INFO:teuthology.orchestra.run.smithi187.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-16T07:52:37.342 INFO:teuthology.orchestra.run.smithi187.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-16T07:52:37.342 INFO:teuthology.orchestra.run.smithi187.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-16T07:52:37.343 INFO:teuthology.orchestra.run.smithi187.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-16T07:52:37.343 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-16T07:52:37.343 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction Summary 2024-09-16T07:52:37.343 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-09-16T07:52:37.343 INFO:teuthology.orchestra.run.smithi187.stdout:Install 11 Packages 2024-09-16T07:52:37.343 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-16T07:52:37.344 INFO:teuthology.orchestra.run.smithi187.stdout:Total download size: 18 M 2024-09-16T07:52:37.344 INFO:teuthology.orchestra.run.smithi187.stdout:Installed size: 155 M 2024-09-16T07:52:37.344 INFO:teuthology.orchestra.run.smithi187.stdout:Downloading Packages: 2024-09-16T07:52:37.919 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:01:25 ago on Mon 16 Sep 2024 07:51:12 AM UTC. 2024-09-16T07:52:38.014 INFO:teuthology.orchestra.run.smithi027.stdout:Package ceph-base-2:19.3.0-4879.g81fedcb3.el9.x86_64 is already installed. 2024-09-16T07:52:38.058 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-09-16T07:52:38.059 INFO:teuthology.orchestra.run.smithi027.stdout:Nothing to do. 2024-09-16T07:52:38.059 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-09-16T07:52:38.145 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install cephadm 2024-09-16T07:52:38.185 INFO:teuthology.orchestra.run.smithi187.stdout:(1/11): ceph-grafana-dashboards-19.3.0-4879.g81 228 kB/s | 34 kB 00:00 2024-09-16T07:52:38.219 INFO:teuthology.orchestra.run.smithi187.stdout:(2/11): ceph-prometheus-alerts-19.3.0-4879.g81f 89 kB/s | 16 kB 00:00 2024-09-16T07:52:38.336 INFO:teuthology.orchestra.run.smithi187.stdout:(3/11): python3-protobuf-3.14.0-13.el9.noarch.r 1.7 MB/s | 269 kB 00:00 2024-09-16T07:52:38.386 INFO:teuthology.orchestra.run.smithi187.stdout:(4/11): abseil-cpp-20211102.0-3.el9.x86_64.rpm 11 MB/s | 548 kB 00:00 2024-09-16T07:52:38.428 INFO:teuthology.orchestra.run.smithi187.stdout:(5/11): protobuf-compiler-3.14.0-13.el9.x86_64. 4.0 MB/s | 863 kB 00:00 2024-09-16T07:52:38.453 INFO:teuthology.orchestra.run.smithi187.stdout:(6/11): grpc-data-1.46.7-10.el9.noarch.rpm 291 kB/s | 19 kB 00:00 2024-09-16T07:52:38.487 INFO:teuthology.orchestra.run.smithi187.stdout:(7/11): python3-grpcio-tools-1.46.7-10.el9.x86_ 5.4 MB/s | 144 kB 00:00 2024-09-16T07:52:38.512 INFO:teuthology.orchestra.run.smithi187.stdout:(8/11): python3-repoze-lru-0.7-16.el9.noarch.rp 1.3 MB/s | 31 kB 00:00 2024-09-16T07:52:38.546 INFO:teuthology.orchestra.run.smithi187.stdout:(9/11): python3-routes-2.5.1-5.el9.noarch.rpm 5.6 MB/s | 188 kB 00:00 2024-09-16T07:52:38.621 INFO:teuthology.orchestra.run.smithi187.stdout:(10/11): python3-grpcio-1.46.7-10.el9.x86_64.rp 11 MB/s | 2.0 MB 00:00 2024-09-16T07:52:38.735 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:01:26 ago on Mon 16 Sep 2024 07:51:12 AM UTC. 2024-09-16T07:52:38.854 INFO:teuthology.orchestra.run.smithi187.stdout:(11/11): ceph-mgr-dashboard-19.3.0-4879.g81fedc 17 MB/s | 14 MB 00:00 2024-09-16T07:52:38.859 INFO:teuthology.orchestra.run.smithi187.stdout:-------------------------------------------------------------------------------- 2024-09-16T07:52:38.859 INFO:teuthology.orchestra.run.smithi187.stdout:Total 12 MB/s | 18 MB 00:01 2024-09-16T07:52:38.872 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-09-16T07:52:38.872 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-09-16T07:52:38.872 INFO:teuthology.orchestra.run.smithi027.stdout: Package Arch Version Repository Size 2024-09-16T07:52:38.873 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-09-16T07:52:38.873 INFO:teuthology.orchestra.run.smithi027.stdout:Installing: 2024-09-16T07:52:38.873 INFO:teuthology.orchestra.run.smithi027.stdout: cephadm noarch 2:19.3.0-4879.g81fedcb3.el9 ceph-noarch 785 k 2024-09-16T07:52:38.873 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-16T07:52:38.873 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2024-09-16T07:52:38.873 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-09-16T07:52:38.873 INFO:teuthology.orchestra.run.smithi027.stdout:Install 1 Package 2024-09-16T07:52:38.873 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-16T07:52:38.874 INFO:teuthology.orchestra.run.smithi027.stdout:Total download size: 785 k 2024-09-16T07:52:38.874 INFO:teuthology.orchestra.run.smithi027.stdout:Installed size: 791 k 2024-09-16T07:52:38.874 INFO:teuthology.orchestra.run.smithi027.stdout:Downloading Packages: 2024-09-16T07:52:38.889 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction check 2024-09-16T07:52:38.908 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction check succeeded. 2024-09-16T07:52:38.908 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction test 2024-09-16T07:52:39.163 INFO:teuthology.orchestra.run.smithi027.stdout:cephadm-19.3.0-4879.g81fedcb3.el9.noarch.rpm 2.7 MB/s | 785 kB 00:00 2024-09-16T07:52:39.164 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2024-09-16T07:52:39.164 INFO:teuthology.orchestra.run.smithi027.stdout:Total 2.6 MB/s | 785 kB 00:00 2024-09-16T07:52:39.164 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction check 2024-09-16T07:52:39.166 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction test succeeded. 2024-09-16T07:52:39.166 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction 2024-09-16T07:52:39.170 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction check succeeded. 2024-09-16T07:52:39.170 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction test 2024-09-16T07:52:39.176 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction test succeeded. 2024-09-16T07:52:39.176 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction 2024-09-16T07:52:39.282 INFO:teuthology.orchestra.run.smithi027.stdout: Preparing : 1/1 2024-09-16T07:52:39.716 INFO:teuthology.orchestra.run.smithi187.stdout: Preparing : 1/1 2024-09-16T07:52:39.759 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: cephadm-2:19.3.0-4879.g81fedcb3.el9.noarch 1/1 2024-09-16T07:52:39.802 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 1/11 2024-09-16T07:52:39.833 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 2/11 2024-09-16T07:52:39.852 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : cephadm-2:19.3.0-4879.g81fedcb3.el9.noarch 1/1 2024-09-16T07:52:39.921 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 3/11 2024-09-16T07:52:40.059 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 4/11 2024-09-16T07:52:40.111 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 5/11 2024-09-16T07:52:40.195 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 6/11 2024-09-16T07:52:40.195 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: cephadm-2:19.3.0-4879.g81fedcb3.el9.noarch 1/1 2024-09-16T07:52:40.246 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 7/11 2024-09-16T07:52:40.275 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 8/11 2024-09-16T07:52:40.307 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-prometheus-alerts-2:19.3.0-4879.g81fedcb3.el9 9/11 2024-09-16T07:52:40.455 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : cephadm-2:19.3.0-4879.g81fedcb3.el9.noarch 1/1 2024-09-16T07:52:40.455 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-16T07:52:40.456 INFO:teuthology.orchestra.run.smithi027.stdout:Installed: 2024-09-16T07:52:40.456 INFO:teuthology.orchestra.run.smithi027.stdout: cephadm-2:19.3.0-4879.g81fedcb3.el9.noarch 2024-09-16T07:52:40.456 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-16T07:52:40.456 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-09-16T07:52:40.695 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install ceph-immutable-object-cache 2024-09-16T07:52:41.288 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:01:29 ago on Mon 16 Sep 2024 07:51:12 AM UTC. 2024-09-16T07:52:41.425 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-09-16T07:52:41.426 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-09-16T07:52:41.426 INFO:teuthology.orchestra.run.smithi027.stdout: Package Arch Version Repo Size 2024-09-16T07:52:41.426 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-09-16T07:52:41.426 INFO:teuthology.orchestra.run.smithi027.stdout:Installing: 2024-09-16T07:52:41.427 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-immutable-object-cache x86_64 2:19.3.0-4879.g81fedcb3.el9 ceph 158 k 2024-09-16T07:52:41.427 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-16T07:52:41.427 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2024-09-16T07:52:41.427 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-09-16T07:52:41.427 INFO:teuthology.orchestra.run.smithi027.stdout:Install 1 Package 2024-09-16T07:52:41.427 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-16T07:52:41.427 INFO:teuthology.orchestra.run.smithi027.stdout:Total download size: 158 k 2024-09-16T07:52:41.427 INFO:teuthology.orchestra.run.smithi027.stdout:Installed size: 644 k 2024-09-16T07:52:41.427 INFO:teuthology.orchestra.run.smithi027.stdout:Downloading Packages: 2024-09-16T07:52:41.635 INFO:teuthology.orchestra.run.smithi027.stdout:ceph-immutable-object-cache-19.3.0-4879.g81fedc 760 kB/s | 158 kB 00:00 2024-09-16T07:52:41.636 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2024-09-16T07:52:41.636 INFO:teuthology.orchestra.run.smithi027.stdout:Total 755 kB/s | 158 kB 00:00 2024-09-16T07:52:41.636 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction check 2024-09-16T07:52:41.647 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction check succeeded. 2024-09-16T07:52:41.647 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction test 2024-09-16T07:52:41.707 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction test succeeded. 2024-09-16T07:52:41.708 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction 2024-09-16T07:52:41.857 INFO:teuthology.orchestra.run.smithi027.stdout: Preparing : 1/1 2024-09-16T07:52:41.911 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-immutable-object-cache-2:19.3.0-4879.g81fedcb3. 1/1 2024-09-16T07:52:41.938 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.3.0-4879.g81fedcb3. 1/1 2024-09-16T07:52:41.938 INFO:teuthology.orchestra.run.smithi027.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T07:52:41.938 INFO:teuthology.orchestra.run.smithi027.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-16T07:52:41.938 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-16T07:52:42.875 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-immutable-object-cache-2:19.3.0-4879.g81fedcb3. 1/1 2024-09-16T07:52:42.875 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-16T07:52:42.875 INFO:teuthology.orchestra.run.smithi027.stdout:Installed: 2024-09-16T07:52:42.875 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-immutable-object-cache-2:19.3.0-4879.g81fedcb3.el9.x86_64 2024-09-16T07:52:42.875 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-16T07:52:42.875 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-09-16T07:52:43.099 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install ceph-mgr 2024-09-16T07:52:43.550 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-grafana-dashboards-2:19.3.0-4879.g81fedcb3.el 10/11 2024-09-16T07:52:43.603 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-mgr-dashboard-2:19.3.0-4879.g81fedcb3.el9.noa 11/11 2024-09-16T07:52:43.688 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:01:31 ago on Mon 16 Sep 2024 07:51:12 AM UTC. 2024-09-16T07:52:43.783 INFO:teuthology.orchestra.run.smithi027.stdout:Package ceph-mgr-2:19.3.0-4879.g81fedcb3.el9.x86_64 is already installed. 2024-09-16T07:52:43.828 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-09-16T07:52:43.829 INFO:teuthology.orchestra.run.smithi027.stdout:Nothing to do. 2024-09-16T07:52:43.829 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-09-16T07:52:43.907 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install ceph-mgr-dashboard 2024-09-16T07:52:44.489 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:01:32 ago on Mon 16 Sep 2024 07:51:12 AM UTC. 2024-09-16T07:52:44.639 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-09-16T07:52:44.639 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-09-16T07:52:44.639 INFO:teuthology.orchestra.run.smithi027.stdout: Package Arch Version Repository Size 2024-09-16T07:52:44.640 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-09-16T07:52:44.640 INFO:teuthology.orchestra.run.smithi027.stdout:Installing: 2024-09-16T07:52:44.640 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-dashboard noarch 2:19.3.0-4879.g81fedcb3.el9 ceph-noarch 14 M 2024-09-16T07:52:44.640 INFO:teuthology.orchestra.run.smithi027.stdout:Installing dependencies: 2024-09-16T07:52:44.640 INFO:teuthology.orchestra.run.smithi027.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-16T07:52:44.640 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-grafana-dashboards noarch 2:19.3.0-4879.g81fedcb3.el9 ceph-noarch 34 k 2024-09-16T07:52:44.640 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-prometheus-alerts noarch 2:19.3.0-4879.g81fedcb3.el9 ceph-noarch 16 k 2024-09-16T07:52:44.640 INFO:teuthology.orchestra.run.smithi027.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-16T07:52:44.640 INFO:teuthology.orchestra.run.smithi027.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-16T07:52:44.640 INFO:teuthology.orchestra.run.smithi027.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-16T07:52:44.640 INFO:teuthology.orchestra.run.smithi027.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-16T07:52:44.640 INFO:teuthology.orchestra.run.smithi027.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-16T07:52:44.641 INFO:teuthology.orchestra.run.smithi027.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-16T07:52:44.641 INFO:teuthology.orchestra.run.smithi027.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-16T07:52:44.641 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-16T07:52:44.641 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2024-09-16T07:52:44.641 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-09-16T07:52:44.641 INFO:teuthology.orchestra.run.smithi027.stdout:Install 11 Packages 2024-09-16T07:52:44.641 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-16T07:52:44.641 INFO:teuthology.orchestra.run.smithi027.stdout:Total download size: 18 M 2024-09-16T07:52:44.641 INFO:teuthology.orchestra.run.smithi027.stdout:Installed size: 155 M 2024-09-16T07:52:44.641 INFO:teuthology.orchestra.run.smithi027.stdout:Downloading Packages: 2024-09-16T07:52:44.915 INFO:teuthology.orchestra.run.smithi027.stdout:(1/11): ceph-grafana-dashboards-19.3.0-4879.g81 218 kB/s | 34 kB 00:00 2024-09-16T07:52:44.931 INFO:teuthology.orchestra.run.smithi027.stdout:(2/11): ceph-prometheus-alerts-19.3.0-4879.g81f 95 kB/s | 16 kB 00:00 2024-09-16T07:52:45.140 INFO:teuthology.orchestra.run.smithi027.stdout:(3/11): python3-protobuf-3.14.0-13.el9.noarch.r 1.2 MB/s | 269 kB 00:00 2024-09-16T07:52:45.174 INFO:teuthology.orchestra.run.smithi027.stdout:(4/11): protobuf-compiler-3.14.0-13.el9.x86_64. 3.5 MB/s | 863 kB 00:00 2024-09-16T07:52:45.224 INFO:teuthology.orchestra.run.smithi027.stdout:(5/11): grpc-data-1.46.7-10.el9.noarch.rpm 390 kB/s | 19 kB 00:00 2024-09-16T07:52:45.249 INFO:teuthology.orchestra.run.smithi027.stdout:(6/11): abseil-cpp-20211102.0-3.el9.x86_64.rpm 4.9 MB/s | 548 kB 00:00 2024-09-16T07:52:45.283 INFO:teuthology.orchestra.run.smithi027.stdout:(7/11): python3-grpcio-tools-1.46.7-10.el9.x86_ 4.2 MB/s | 144 kB 00:00 2024-09-16T07:52:45.308 INFO:teuthology.orchestra.run.smithi027.stdout:(8/11): python3-repoze-lru-0.7-16.el9.noarch.rp 1.2 MB/s | 31 kB 00:00 2024-09-16T07:52:45.342 INFO:teuthology.orchestra.run.smithi027.stdout:(9/11): python3-routes-2.5.1-5.el9.noarch.rpm 5.5 MB/s | 188 kB 00:00 2024-09-16T07:52:45.501 INFO:teuthology.orchestra.run.smithi027.stdout:(10/11): ceph-mgr-dashboard-19.3.0-4879.g81fedc 19 MB/s | 14 MB 00:00 2024-09-16T07:52:45.542 INFO:teuthology.orchestra.run.smithi027.stdout:(11/11): python3-grpcio-1.46.7-10.el9.x86_64.rp 6.4 MB/s | 2.0 MB 00:00 2024-09-16T07:52:45.546 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2024-09-16T07:52:45.546 INFO:teuthology.orchestra.run.smithi027.stdout:Total 20 MB/s | 18 MB 00:00 2024-09-16T07:52:45.577 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction check 2024-09-16T07:52:45.597 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction check succeeded. 2024-09-16T07:52:45.597 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction test 2024-09-16T07:52:45.661 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.3.0-4879.g81fedcb3.el9.noa 11/11 2024-09-16T07:52:45.661 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-grafana-dashboards-2:19.3.0-4879.g81fedcb3.el 1/11 2024-09-16T07:52:45.661 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-mgr-dashboard-2:19.3.0-4879.g81fedcb3.el9.noa 2/11 2024-09-16T07:52:45.661 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-prometheus-alerts-2:19.3.0-4879.g81fedcb3.el9 3/11 2024-09-16T07:52:45.661 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 4/11 2024-09-16T07:52:45.661 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 5/11 2024-09-16T07:52:45.661 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 6/11 2024-09-16T07:52:45.661 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 7/11 2024-09-16T07:52:45.661 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 8/11 2024-09-16T07:52:45.661 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/11 2024-09-16T07:52:45.661 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 10/11 2024-09-16T07:52:45.880 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction test succeeded. 2024-09-16T07:52:45.881 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction 2024-09-16T07:52:45.972 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 11/11 2024-09-16T07:52:45.972 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-16T07:52:45.973 INFO:teuthology.orchestra.run.smithi187.stdout:Installed: 2024-09-16T07:52:45.973 INFO:teuthology.orchestra.run.smithi187.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-16T07:52:45.973 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-grafana-dashboards-2:19.3.0-4879.g81fedcb3.el9.noarch 2024-09-16T07:52:45.973 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mgr-dashboard-2:19.3.0-4879.g81fedcb3.el9.noarch 2024-09-16T07:52:45.973 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-prometheus-alerts-2:19.3.0-4879.g81fedcb3.el9.noarch 2024-09-16T07:52:45.973 INFO:teuthology.orchestra.run.smithi187.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-16T07:52:45.973 INFO:teuthology.orchestra.run.smithi187.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-16T07:52:45.973 INFO:teuthology.orchestra.run.smithi187.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-16T07:52:45.973 INFO:teuthology.orchestra.run.smithi187.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-16T07:52:45.973 INFO:teuthology.orchestra.run.smithi187.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-16T07:52:45.973 INFO:teuthology.orchestra.run.smithi187.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-16T07:52:45.973 INFO:teuthology.orchestra.run.smithi187.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-16T07:52:45.973 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-16T07:52:45.973 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2024-09-16T07:52:46.212 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-16T07:52:46.442 INFO:teuthology.orchestra.run.smithi027.stdout: Preparing : 1/1 2024-09-16T07:52:46.525 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 1/11 2024-09-16T07:52:46.562 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 2/11 2024-09-16T07:52:46.659 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 3/11 2024-09-16T07:52:46.766 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:01:37 ago on Mon 16 Sep 2024 07:51:09 AM UTC. 2024-09-16T07:52:46.795 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 4/11 2024-09-16T07:52:46.858 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 5/11 2024-09-16T07:52:46.900 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2024-09-16T07:52:46.901 INFO:teuthology.orchestra.run.smithi187.stdout:======================================================================================= 2024-09-16T07:52:46.901 INFO:teuthology.orchestra.run.smithi187.stdout: Package Arch Version Repository Size 2024-09-16T07:52:46.901 INFO:teuthology.orchestra.run.smithi187.stdout:======================================================================================= 2024-09-16T07:52:46.901 INFO:teuthology.orchestra.run.smithi187.stdout:Installing: 2024-09-16T07:52:46.901 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mgr-diskprediction-local noarch 2:19.3.0-4879.g81fedcb3.el9 ceph-noarch 7.4 M 2024-09-16T07:52:46.901 INFO:teuthology.orchestra.run.smithi187.stdout:Installing dependencies: 2024-09-16T07:52:46.901 INFO:teuthology.orchestra.run.smithi187.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-16T07:52:46.902 INFO:teuthology.orchestra.run.smithi187.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-16T07:52:46.902 INFO:teuthology.orchestra.run.smithi187.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-16T07:52:46.902 INFO:teuthology.orchestra.run.smithi187.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-16T07:52:46.902 INFO:teuthology.orchestra.run.smithi187.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-16T07:52:46.902 INFO:teuthology.orchestra.run.smithi187.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-16T07:52:46.902 INFO:teuthology.orchestra.run.smithi187.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-16T07:52:46.902 INFO:teuthology.orchestra.run.smithi187.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-16T07:52:46.902 INFO:teuthology.orchestra.run.smithi187.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-16T07:52:46.902 INFO:teuthology.orchestra.run.smithi187.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-16T07:52:46.902 INFO:teuthology.orchestra.run.smithi187.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-16T07:52:46.902 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-16T07:52:46.902 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction Summary 2024-09-16T07:52:46.902 INFO:teuthology.orchestra.run.smithi187.stdout:======================================================================================= 2024-09-16T07:52:46.903 INFO:teuthology.orchestra.run.smithi187.stdout:Install 12 Packages 2024-09-16T07:52:46.903 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-16T07:52:46.903 INFO:teuthology.orchestra.run.smithi187.stdout:Total download size: 38 M 2024-09-16T07:52:46.903 INFO:teuthology.orchestra.run.smithi187.stdout:Installed size: 212 M 2024-09-16T07:52:46.903 INFO:teuthology.orchestra.run.smithi187.stdout:Downloading Packages: 2024-09-16T07:52:46.944 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 6/11 2024-09-16T07:52:47.002 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 7/11 2024-09-16T07:52:47.030 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 8/11 2024-09-16T07:52:47.063 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-prometheus-alerts-2:19.3.0-4879.g81fedcb3.el9 9/11 2024-09-16T07:52:47.260 INFO:teuthology.orchestra.run.smithi187.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 1.1 MB/s | 191 kB 00:00 2024-09-16T07:52:47.444 INFO:teuthology.orchestra.run.smithi187.stdout:(2/12): libgfortran-11.5.0-2.el9.x86_64.rpm 2.2 MB/s | 801 kB 00:00 2024-09-16T07:52:47.544 INFO:teuthology.orchestra.run.smithi187.stdout:(3/12): ceph-mgr-diskprediction-local-19.3.0-48 16 MB/s | 7.4 MB 00:00 2024-09-16T07:52:47.561 INFO:teuthology.orchestra.run.smithi187.stdout:(4/12): flexiblas-3.0.4-8.el9.x86_64.rpm 109 kB/s | 33 kB 00:00 2024-09-16T07:52:47.761 INFO:teuthology.orchestra.run.smithi187.stdout:(5/12): openblas-0.3.26-2.el9.x86_64.rpm 192 kB/s | 38 kB 00:00 2024-09-16T07:52:47.828 INFO:teuthology.orchestra.run.smithi187.stdout:(6/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 64 kB/s | 18 kB 00:00 2024-09-16T07:52:48.286 INFO:teuthology.orchestra.run.smithi187.stdout:(7/12): python3-devel-3.9.19-8.el9.x86_64.rpm 534 kB/s | 245 kB 00:00 2024-09-16T07:52:48.604 INFO:teuthology.orchestra.run.smithi187.stdout:(8/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 2.6 MB/s | 3.0 MB 00:01 2024-09-16T07:52:48.720 INFO:teuthology.orchestra.run.smithi187.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.1 MB/s | 250 kB 00:00 2024-09-16T07:52:48.854 INFO:teuthology.orchestra.run.smithi187.stdout:(10/12): openblas-openmp-0.3.26-2.el9.x86_64.rp 4.5 MB/s | 4.9 MB 00:01 2024-09-16T07:52:49.187 INFO:teuthology.orchestra.run.smithi187.stdout:(11/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 5.7 MB/s | 5.1 MB 00:00 2024-09-16T07:52:50.430 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-grafana-dashboards-2:19.3.0-4879.g81fedcb3.el 10/11 2024-09-16T07:52:50.492 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-mgr-dashboard-2:19.3.0-4879.g81fedcb3.el9.noa 11/11 2024-09-16T07:52:52.295 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.3.0-4879.g81fedcb3.el9.noa 11/11 2024-09-16T07:52:52.295 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-grafana-dashboards-2:19.3.0-4879.g81fedcb3.el 1/11 2024-09-16T07:52:52.295 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-mgr-dashboard-2:19.3.0-4879.g81fedcb3.el9.noa 2/11 2024-09-16T07:52:52.295 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-prometheus-alerts-2:19.3.0-4879.g81fedcb3.el9 3/11 2024-09-16T07:52:52.295 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 4/11 2024-09-16T07:52:52.295 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 5/11 2024-09-16T07:52:52.295 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 6/11 2024-09-16T07:52:52.295 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 7/11 2024-09-16T07:52:52.296 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 8/11 2024-09-16T07:52:52.296 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/11 2024-09-16T07:52:52.296 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 10/11 2024-09-16T07:52:52.618 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 11/11 2024-09-16T07:52:52.618 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-16T07:52:52.618 INFO:teuthology.orchestra.run.smithi027.stdout:Installed: 2024-09-16T07:52:52.618 INFO:teuthology.orchestra.run.smithi027.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-16T07:52:52.618 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-grafana-dashboards-2:19.3.0-4879.g81fedcb3.el9.noarch 2024-09-16T07:52:52.619 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-dashboard-2:19.3.0-4879.g81fedcb3.el9.noarch 2024-09-16T07:52:52.619 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-prometheus-alerts-2:19.3.0-4879.g81fedcb3.el9.noarch 2024-09-16T07:52:52.619 INFO:teuthology.orchestra.run.smithi027.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-16T07:52:52.619 INFO:teuthology.orchestra.run.smithi027.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-16T07:52:52.619 INFO:teuthology.orchestra.run.smithi027.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-16T07:52:52.619 INFO:teuthology.orchestra.run.smithi027.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-16T07:52:52.619 INFO:teuthology.orchestra.run.smithi027.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-16T07:52:52.619 INFO:teuthology.orchestra.run.smithi027.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-16T07:52:52.619 INFO:teuthology.orchestra.run.smithi027.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-16T07:52:52.619 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-16T07:52:52.619 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-09-16T07:52:52.865 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-16T07:52:53.455 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:01:41 ago on Mon 16 Sep 2024 07:51:12 AM UTC. 2024-09-16T07:52:53.595 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-09-16T07:52:53.596 INFO:teuthology.orchestra.run.smithi027.stdout:======================================================================================= 2024-09-16T07:52:53.596 INFO:teuthology.orchestra.run.smithi027.stdout: Package Arch Version Repository Size 2024-09-16T07:52:53.596 INFO:teuthology.orchestra.run.smithi027.stdout:======================================================================================= 2024-09-16T07:52:53.596 INFO:teuthology.orchestra.run.smithi027.stdout:Installing: 2024-09-16T07:52:53.596 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-diskprediction-local noarch 2:19.3.0-4879.g81fedcb3.el9 ceph-noarch 7.4 M 2024-09-16T07:52:53.596 INFO:teuthology.orchestra.run.smithi027.stdout:Installing dependencies: 2024-09-16T07:52:53.596 INFO:teuthology.orchestra.run.smithi027.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-16T07:52:53.596 INFO:teuthology.orchestra.run.smithi027.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-16T07:52:53.596 INFO:teuthology.orchestra.run.smithi027.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-16T07:52:53.596 INFO:teuthology.orchestra.run.smithi027.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-16T07:52:53.597 INFO:teuthology.orchestra.run.smithi027.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-16T07:52:53.597 INFO:teuthology.orchestra.run.smithi027.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-16T07:52:53.597 INFO:teuthology.orchestra.run.smithi027.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-16T07:52:53.597 INFO:teuthology.orchestra.run.smithi027.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-16T07:52:53.597 INFO:teuthology.orchestra.run.smithi027.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-16T07:52:53.597 INFO:teuthology.orchestra.run.smithi027.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-16T07:52:53.597 INFO:teuthology.orchestra.run.smithi027.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-16T07:52:53.597 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-16T07:52:53.597 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2024-09-16T07:52:53.597 INFO:teuthology.orchestra.run.smithi027.stdout:======================================================================================= 2024-09-16T07:52:53.597 INFO:teuthology.orchestra.run.smithi027.stdout:Install 12 Packages 2024-09-16T07:52:53.597 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-16T07:52:53.598 INFO:teuthology.orchestra.run.smithi027.stdout:Total download size: 38 M 2024-09-16T07:52:53.598 INFO:teuthology.orchestra.run.smithi027.stdout:Installed size: 212 M 2024-09-16T07:52:53.598 INFO:teuthology.orchestra.run.smithi027.stdout:Downloading Packages: 2024-09-16T07:52:54.349 INFO:teuthology.orchestra.run.smithi027.stdout:(1/12): ceph-mgr-diskprediction-local-19.3.0-48 16 MB/s | 7.4 MB 00:00 2024-09-16T07:52:54.440 INFO:teuthology.orchestra.run.smithi027.stdout:(2/12): libquadmath-11.5.0-2.el9.x86_64.rpm 349 kB/s | 191 kB 00:00 2024-09-16T07:52:54.466 INFO:teuthology.orchestra.run.smithi027.stdout:(3/12): flexiblas-3.0.4-8.el9.x86_64.rpm 280 kB/s | 33 kB 00:00 2024-09-16T07:52:54.516 INFO:teuthology.orchestra.run.smithi027.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 363 kB/s | 18 kB 00:00 2024-09-16T07:52:54.566 INFO:teuthology.orchestra.run.smithi027.stdout:(5/12): openblas-0.3.26-2.el9.x86_64.rpm 769 kB/s | 38 kB 00:00 2024-09-16T07:52:54.808 INFO:teuthology.orchestra.run.smithi027.stdout:(6/12): libgfortran-11.5.0-2.el9.x86_64.rpm 875 kB/s | 801 kB 00:00 2024-09-16T07:52:55.000 INFO:teuthology.orchestra.run.smithi027.stdout:(7/12): python3-devel-3.9.19-8.el9.x86_64.rpm 1.2 MB/s | 245 kB 00:00 2024-09-16T07:52:55.150 INFO:teuthology.orchestra.run.smithi027.stdout:(8/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 4.2 MB/s | 3.0 MB 00:00 2024-09-16T07:52:55.217 INFO:teuthology.orchestra.run.smithi027.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 3.7 MB/s | 250 kB 00:00 2024-09-16T07:52:55.326 INFO:teuthology.orchestra.run.smithi027.stdout:(10/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 16 MB/s | 5.1 MB 00:00 2024-09-16T07:52:55.393 INFO:teuthology.orchestra.run.smithi027.stdout:(11/12): openblas-openmp-0.3.26-2.el9.x86_64.rp 6.0 MB/s | 4.9 MB 00:00 2024-09-16T07:52:55.760 INFO:teuthology.orchestra.run.smithi027.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 30 MB/s | 16 MB 00:00 2024-09-16T07:52:55.760 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2024-09-16T07:52:55.760 INFO:teuthology.orchestra.run.smithi027.stdout:Total 18 MB/s | 38 MB 00:02 2024-09-16T07:52:56.125 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction check 2024-09-16T07:52:56.152 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction check succeeded. 2024-09-16T07:52:56.152 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction test 2024-09-16T07:52:56.620 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction test succeeded. 2024-09-16T07:52:56.620 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction 2024-09-16T07:52:57.533 INFO:teuthology.orchestra.run.smithi027.stdout: Preparing : 1/1 2024-09-16T07:52:57.601 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-16T07:52:57.642 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-16T07:52:57.754 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-16T07:52:57.793 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-16T07:52:57.985 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-16T07:52:58.016 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-16T07:52:58.104 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-16T07:52:58.798 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-16T07:52:58.934 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-16T07:53:00.233 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-16T07:53:01.322 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-16T07:53:01.349 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-mgr-diskprediction-local-2:19.3.0-4879.g81fed 12/12 2024-09-16T07:53:03.949 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.3.0-4879.g81fed 12/12 2024-09-16T07:53:03.949 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.3.0-4879.g81fed 1/12 2024-09-16T07:53:03.949 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-16T07:53:03.949 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-16T07:53:03.949 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-16T07:53:03.949 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-16T07:53:03.949 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-16T07:53:03.949 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-16T07:53:03.949 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-16T07:53:03.949 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-16T07:53:03.950 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-16T07:53:03.950 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-16T07:53:04.431 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-16T07:53:04.431 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-16T07:53:04.432 INFO:teuthology.orchestra.run.smithi027.stdout:Installed: 2024-09-16T07:53:04.432 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-diskprediction-local-2:19.3.0-4879.g81fedcb3.el9.noarch 2024-09-16T07:53:04.432 INFO:teuthology.orchestra.run.smithi027.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-16T07:53:04.432 INFO:teuthology.orchestra.run.smithi027.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-16T07:53:04.432 INFO:teuthology.orchestra.run.smithi027.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-16T07:53:04.432 INFO:teuthology.orchestra.run.smithi027.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-16T07:53:04.432 INFO:teuthology.orchestra.run.smithi027.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-16T07:53:04.432 INFO:teuthology.orchestra.run.smithi027.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-16T07:53:04.432 INFO:teuthology.orchestra.run.smithi027.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-16T07:53:04.432 INFO:teuthology.orchestra.run.smithi027.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-16T07:53:04.432 INFO:teuthology.orchestra.run.smithi027.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-16T07:53:04.432 INFO:teuthology.orchestra.run.smithi027.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-16T07:53:04.433 INFO:teuthology.orchestra.run.smithi027.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-16T07:53:04.433 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-16T07:53:04.433 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-09-16T07:53:04.828 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install ceph-mgr-rook 2024-09-16T07:53:05.424 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:01:53 ago on Mon 16 Sep 2024 07:51:12 AM UTC. 2024-09-16T07:53:05.565 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-09-16T07:53:05.566 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-09-16T07:53:05.566 INFO:teuthology.orchestra.run.smithi027.stdout: Package Arch Version Repository Size 2024-09-16T07:53:05.566 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-09-16T07:53:05.567 INFO:teuthology.orchestra.run.smithi027.stdout:Installing: 2024-09-16T07:53:05.567 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-rook noarch 2:19.3.0-4879.g81fedcb3.el9 ceph-noarch 49 k 2024-09-16T07:53:05.567 INFO:teuthology.orchestra.run.smithi027.stdout:Installing dependencies: 2024-09-16T07:53:05.567 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-16T07:53:05.567 INFO:teuthology.orchestra.run.smithi027.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-16T07:53:05.567 INFO:teuthology.orchestra.run.smithi027.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-16T07:53:05.567 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-16T07:53:05.567 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-16T07:53:05.567 INFO:teuthology.orchestra.run.smithi027.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-16T07:53:05.567 INFO:teuthology.orchestra.run.smithi027.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-16T07:53:05.567 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-16T07:53:05.567 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-16T07:53:05.568 INFO:teuthology.orchestra.run.smithi027.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-16T07:53:05.568 INFO:teuthology.orchestra.run.smithi027.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-16T07:53:05.568 INFO:teuthology.orchestra.run.smithi027.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-16T07:53:05.568 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-16T07:53:05.568 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2024-09-16T07:53:05.568 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-09-16T07:53:05.568 INFO:teuthology.orchestra.run.smithi027.stdout:Install 13 Packages 2024-09-16T07:53:05.568 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-16T07:53:05.569 INFO:teuthology.orchestra.run.smithi027.stdout:Total download size: 2.2 M 2024-09-16T07:53:05.569 INFO:teuthology.orchestra.run.smithi027.stdout:Installed size: 26 M 2024-09-16T07:53:05.569 INFO:teuthology.orchestra.run.smithi027.stdout:Downloading Packages: 2024-09-16T07:53:05.734 INFO:teuthology.orchestra.run.smithi027.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 253 kB/s | 26 kB 00:00 2024-09-16T07:53:05.761 INFO:teuthology.orchestra.run.smithi027.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 152 kB/s | 19 kB 00:00 2024-09-16T07:53:05.852 INFO:teuthology.orchestra.run.smithi027.stdout:(3/13): ceph-mgr-rook-19.3.0-4879.g81fedcb3.el9 225 kB/s | 49 kB 00:00 2024-09-16T07:53:05.877 INFO:teuthology.orchestra.run.smithi027.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.3 MB/s | 159 kB 00:00 2024-09-16T07:53:05.902 INFO:teuthology.orchestra.run.smithi027.stdout:(5/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.5 MB/s | 222 kB 00:00 2024-09-16T07:53:05.927 INFO:teuthology.orchestra.run.smithi027.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 1.0 MB/s | 54 kB 00:00 2024-09-16T07:53:05.969 INFO:teuthology.orchestra.run.smithi027.stdout:(7/13): python3-cachetools-4.2.4-1.el9.noarch.r 479 kB/s | 32 kB 00:00 2024-09-16T07:53:06.003 INFO:teuthology.orchestra.run.smithi027.stdout:(8/13): python3-certifi-2023.05.07-4.el9.noarch 206 kB/s | 14 kB 00:00 2024-09-16T07:53:06.029 INFO:teuthology.orchestra.run.smithi027.stdout:(9/13): python3-google-auth-2.34.0-1.el9.noarch 3.8 MB/s | 220 kB 00:00 2024-09-16T07:53:06.054 INFO:teuthology.orchestra.run.smithi027.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 2.3 MB/s | 59 kB 00:00 2024-09-16T07:53:06.079 INFO:teuthology.orchestra.run.smithi027.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 3.5 MB/s | 90 kB 00:00 2024-09-16T07:53:06.104 INFO:teuthology.orchestra.run.smithi027.stdout:(12/13): python3-pyasn1-modules-0.4.8-6.el9.noa 1.1 MB/s | 279 kB 00:00 2024-09-16T07:53:06.163 INFO:teuthology.orchestra.run.smithi027.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 6.4 MB/s | 1.0 MB 00:00 2024-09-16T07:53:06.164 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2024-09-16T07:53:06.165 INFO:teuthology.orchestra.run.smithi027.stdout:Total 3.7 MB/s | 2.2 MB 00:00 2024-09-16T07:53:06.230 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction check 2024-09-16T07:53:06.246 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction check succeeded. 2024-09-16T07:53:06.246 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction test 2024-09-16T07:53:06.341 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction test succeeded. 2024-09-16T07:53:06.342 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction 2024-09-16T07:53:06.989 INFO:teuthology.orchestra.run.smithi027.stdout: Preparing : 1/1 2024-09-16T07:53:07.179 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-16T07:53:07.233 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-16T07:53:07.297 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-16T07:53:07.341 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-16T07:53:07.382 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-16T07:53:07.489 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-16T07:53:07.601 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-16T07:53:07.660 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-16T07:53:08.426 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-16T07:53:08.473 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-16T07:53:08.506 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-16T07:53:08.555 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-16T07:53:08.584 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-mgr-rook-2:19.3.0-4879.g81fedcb3.el9.noarch 13/13 2024-09-16T07:53:09.370 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-mgr-rook-2:19.3.0-4879.g81fedcb3.el9.noarch 13/13 2024-09-16T07:53:09.370 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-mgr-rook-2:19.3.0-4879.g81fedcb3.el9.noarch 1/13 2024-09-16T07:53:09.370 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-16T07:53:09.370 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-16T07:53:09.370 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-16T07:53:09.370 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-16T07:53:09.370 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-16T07:53:09.370 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-16T07:53:09.371 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-16T07:53:09.371 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-16T07:53:09.371 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-16T07:53:09.371 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-16T07:53:09.371 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-16T07:53:09.705 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-16T07:53:09.705 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-16T07:53:09.705 INFO:teuthology.orchestra.run.smithi027.stdout:Installed: 2024-09-16T07:53:09.705 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-rook-2:19.3.0-4879.g81fedcb3.el9.noarch 2024-09-16T07:53:09.706 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-16T07:53:09.706 INFO:teuthology.orchestra.run.smithi027.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-16T07:53:09.706 INFO:teuthology.orchestra.run.smithi027.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-16T07:53:09.706 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-16T07:53:09.706 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-16T07:53:09.706 INFO:teuthology.orchestra.run.smithi027.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-16T07:53:09.706 INFO:teuthology.orchestra.run.smithi027.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-16T07:53:09.706 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-16T07:53:09.706 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-16T07:53:09.706 INFO:teuthology.orchestra.run.smithi027.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-16T07:53:09.706 INFO:teuthology.orchestra.run.smithi027.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-16T07:53:09.706 INFO:teuthology.orchestra.run.smithi027.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-16T07:53:09.706 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-16T07:53:09.707 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-09-16T07:53:09.885 INFO:teuthology.orchestra.run.smithi187.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 786 kB/s | 16 MB 00:21 2024-09-16T07:53:09.885 INFO:teuthology.orchestra.run.smithi187.stdout:-------------------------------------------------------------------------------- 2024-09-16T07:53:09.886 INFO:teuthology.orchestra.run.smithi187.stdout:Total 1.7 MB/s | 38 MB 00:22 2024-09-16T07:53:09.962 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install ceph-mgr-cephadm 2024-09-16T07:53:10.237 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction check 2024-09-16T07:53:10.262 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction check succeeded. 2024-09-16T07:53:10.262 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction test 2024-09-16T07:53:10.555 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:01:58 ago on Mon 16 Sep 2024 07:51:12 AM UTC. 2024-09-16T07:53:10.697 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-09-16T07:53:10.698 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-09-16T07:53:10.698 INFO:teuthology.orchestra.run.smithi027.stdout: Package Arch Version Repository Size 2024-09-16T07:53:10.698 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-09-16T07:53:10.698 INFO:teuthology.orchestra.run.smithi027.stdout:Installing: 2024-09-16T07:53:10.698 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-cephadm noarch 2:19.3.0-4879.g81fedcb3.el9 ceph-noarch 154 k 2024-09-16T07:53:10.698 INFO:teuthology.orchestra.run.smithi027.stdout:Installing dependencies: 2024-09-16T07:53:10.698 INFO:teuthology.orchestra.run.smithi027.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-16T07:53:10.698 INFO:teuthology.orchestra.run.smithi027.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-16T07:53:10.698 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-16T07:53:10.698 INFO:teuthology.orchestra.run.smithi027.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-16T07:53:10.698 INFO:teuthology.orchestra.run.smithi027.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-16T07:53:10.699 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-16T07:53:10.699 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2024-09-16T07:53:10.699 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-09-16T07:53:10.699 INFO:teuthology.orchestra.run.smithi027.stdout:Install 6 Packages 2024-09-16T07:53:10.699 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-16T07:53:10.700 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction test succeeded. 2024-09-16T07:53:10.700 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction 2024-09-16T07:53:10.700 INFO:teuthology.orchestra.run.smithi027.stdout:Total download size: 7.0 M 2024-09-16T07:53:10.700 INFO:teuthology.orchestra.run.smithi027.stdout:Installed size: 33 M 2024-09-16T07:53:10.700 INFO:teuthology.orchestra.run.smithi027.stdout:Downloading Packages: 2024-09-16T07:53:11.048 INFO:teuthology.orchestra.run.smithi027.stdout:(1/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.7 MB/s | 248 kB 00:00 2024-09-16T07:53:11.132 INFO:teuthology.orchestra.run.smithi027.stdout:(2/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 6.4 MB/s | 548 kB 00:00 2024-09-16T07:53:11.157 INFO:teuthology.orchestra.run.smithi027.stdout:(3/6): ceph-mgr-cephadm-19.3.0-4879.g81fedcb3.e 618 kB/s | 154 kB 00:00 2024-09-16T07:53:11.182 INFO:teuthology.orchestra.run.smithi027.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 1.1 MB/s | 58 kB 00:00 2024-09-16T07:53:11.274 INFO:teuthology.orchestra.run.smithi027.stdout:(5/6): python3-babel-2.9.1-2.el9.noarch.rpm 16 MB/s | 6.0 MB 00:00 2024-09-16T07:53:11.299 INFO:teuthology.orchestra.run.smithi027.stdout:(6/6): python3-typing-extensions-4.12.2-2.el9.n 524 kB/s | 74 kB 00:00 2024-09-16T07:53:11.302 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2024-09-16T07:53:11.302 INFO:teuthology.orchestra.run.smithi027.stdout:Total 12 MB/s | 7.0 MB 00:00 2024-09-16T07:53:11.376 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction check 2024-09-16T07:53:11.387 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction check succeeded. 2024-09-16T07:53:11.388 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction test 2024-09-16T07:53:11.451 INFO:teuthology.orchestra.run.smithi187.stdout: Preparing : 1/1 2024-09-16T07:53:11.494 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction test succeeded. 2024-09-16T07:53:11.494 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction 2024-09-16T07:53:11.524 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-16T07:53:11.566 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-16T07:53:11.657 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-16T07:53:11.691 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-16T07:53:11.812 INFO:teuthology.orchestra.run.smithi027.stdout: Preparing : 1/1 2024-09-16T07:53:11.876 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-16T07:53:11.916 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-16T07:53:11.990 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-16T07:53:12.000 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-16T07:53:12.044 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-16T07:53:12.608 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-16T07:53:12.661 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-16T07:53:12.700 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-16T07:53:12.775 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-16T07:53:12.787 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-16T07:53:12.805 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-mgr-cephadm-2:19.3.0-4879.g81fedcb3.el9.noarch 6/6 2024-09-16T07:53:13.771 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.3.0-4879.g81fedcb3.el9.noarch 6/6 2024-09-16T07:53:13.772 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-mgr-cephadm-2:19.3.0-4879.g81fedcb3.el9.noarch 1/6 2024-09-16T07:53:13.772 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-16T07:53:13.772 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-16T07:53:13.772 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-16T07:53:13.772 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-16T07:53:14.021 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-16T07:53:14.055 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-16T07:53:14.056 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-16T07:53:14.056 INFO:teuthology.orchestra.run.smithi027.stdout:Installed: 2024-09-16T07:53:14.056 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-cephadm-2:19.3.0-4879.g81fedcb3.el9.noarch 2024-09-16T07:53:14.056 INFO:teuthology.orchestra.run.smithi027.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-16T07:53:14.056 INFO:teuthology.orchestra.run.smithi027.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-16T07:53:14.056 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-16T07:53:14.056 INFO:teuthology.orchestra.run.smithi027.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-16T07:53:14.056 INFO:teuthology.orchestra.run.smithi027.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-16T07:53:14.056 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-16T07:53:14.056 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-09-16T07:53:14.307 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install ceph-fuse 2024-09-16T07:53:14.899 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:02:02 ago on Mon 16 Sep 2024 07:51:12 AM UTC. 2024-09-16T07:53:15.039 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-09-16T07:53:15.040 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-09-16T07:53:15.040 INFO:teuthology.orchestra.run.smithi027.stdout: Package Arch Version Repository Size 2024-09-16T07:53:15.040 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-09-16T07:53:15.040 INFO:teuthology.orchestra.run.smithi027.stdout:Installing: 2024-09-16T07:53:15.040 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-fuse x86_64 2:19.3.0-4879.g81fedcb3.el9 ceph 884 k 2024-09-16T07:53:15.040 INFO:teuthology.orchestra.run.smithi027.stdout:Installing dependencies: 2024-09-16T07:53:15.040 INFO:teuthology.orchestra.run.smithi027.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-16T07:53:15.040 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-16T07:53:15.041 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2024-09-16T07:53:15.041 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-09-16T07:53:15.041 INFO:teuthology.orchestra.run.smithi027.stdout:Install 2 Packages 2024-09-16T07:53:15.041 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-16T07:53:15.041 INFO:teuthology.orchestra.run.smithi027.stdout:Total download size: 964 k 2024-09-16T07:53:15.041 INFO:teuthology.orchestra.run.smithi027.stdout:Installed size: 4.1 M 2024-09-16T07:53:15.041 INFO:teuthology.orchestra.run.smithi027.stdout:Downloading Packages: 2024-09-16T07:53:15.061 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-16T07:53:15.090 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-mgr-diskprediction-local-2:19.3.0-4879.g81fed 12/12 2024-09-16T07:53:15.244 INFO:teuthology.orchestra.run.smithi027.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 510 kB/s | 80 kB 00:00 2024-09-16T07:53:15.361 INFO:teuthology.orchestra.run.smithi027.stdout:(2/2): ceph-fuse-19.3.0-4879.g81fedcb3.el9.x86_ 3.2 MB/s | 884 kB 00:00 2024-09-16T07:53:15.361 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2024-09-16T07:53:15.362 INFO:teuthology.orchestra.run.smithi027.stdout:Total 2.9 MB/s | 964 kB 00:00 2024-09-16T07:53:15.373 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction check 2024-09-16T07:53:15.398 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction check succeeded. 2024-09-16T07:53:15.398 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction test 2024-09-16T07:53:15.484 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction test succeeded. 2024-09-16T07:53:15.484 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction 2024-09-16T07:53:15.679 INFO:teuthology.orchestra.run.smithi027.stdout: Preparing : 1/1 2024-09-16T07:53:15.836 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-16T07:53:15.905 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-fuse-2:19.3.0-4879.g81fedcb3.el9.x86_64 2/2 2024-09-16T07:53:16.605 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-fuse-2:19.3.0-4879.g81fedcb3.el9.x86_64 2/2 2024-09-16T07:53:16.605 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-fuse-2:19.3.0-4879.g81fedcb3.el9.x86_64 1/2 2024-09-16T07:53:16.880 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-16T07:53:16.880 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-16T07:53:16.880 INFO:teuthology.orchestra.run.smithi027.stdout:Installed: 2024-09-16T07:53:16.880 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-fuse-2:19.3.0-4879.g81fedcb3.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-16T07:53:16.880 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-16T07:53:16.880 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-09-16T07:53:17.176 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install ceph-volume 2024-09-16T07:53:17.772 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:02:05 ago on Mon 16 Sep 2024 07:51:12 AM UTC. 2024-09-16T07:53:17.910 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-09-16T07:53:17.911 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-09-16T07:53:17.911 INFO:teuthology.orchestra.run.smithi027.stdout: Package Arch Version Repository Size 2024-09-16T07:53:17.911 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-09-16T07:53:17.911 INFO:teuthology.orchestra.run.smithi027.stdout:Installing: 2024-09-16T07:53:17.911 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-volume noarch 2:19.3.0-4879.g81fedcb3.el9 ceph-noarch 285 k 2024-09-16T07:53:17.911 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-16T07:53:17.911 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2024-09-16T07:53:17.911 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-09-16T07:53:17.911 INFO:teuthology.orchestra.run.smithi027.stdout:Install 1 Package 2024-09-16T07:53:17.912 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-16T07:53:17.912 INFO:teuthology.orchestra.run.smithi027.stdout:Total download size: 285 k 2024-09-16T07:53:17.912 INFO:teuthology.orchestra.run.smithi027.stdout:Installed size: 1.3 M 2024-09-16T07:53:17.912 INFO:teuthology.orchestra.run.smithi027.stdout:Downloading Packages: 2024-09-16T07:53:18.157 INFO:teuthology.orchestra.run.smithi027.stdout:ceph-volume-19.3.0-4879.g81fedcb3.el9.noarch.rp 1.1 MB/s | 285 kB 00:00 2024-09-16T07:53:18.158 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2024-09-16T07:53:18.158 INFO:teuthology.orchestra.run.smithi027.stdout:Total 1.1 MB/s | 285 kB 00:00 2024-09-16T07:53:18.158 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction check 2024-09-16T07:53:18.167 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction check succeeded. 2024-09-16T07:53:18.167 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction test 2024-09-16T07:53:18.303 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.3.0-4879.g81fed 12/12 2024-09-16T07:53:18.303 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.3.0-4879.g81fed 1/12 2024-09-16T07:53:18.303 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-16T07:53:18.303 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-16T07:53:18.304 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-16T07:53:18.304 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-16T07:53:18.304 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-16T07:53:18.304 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-16T07:53:18.304 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-16T07:53:18.304 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-16T07:53:18.304 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-16T07:53:18.304 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-16T07:53:18.310 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction test succeeded. 2024-09-16T07:53:18.310 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction 2024-09-16T07:53:18.544 INFO:teuthology.orchestra.run.smithi027.stdout: Preparing : 1/1 2024-09-16T07:53:18.617 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-volume-2:19.3.0-4879.g81fedcb3.el9.noarch 1/1 2024-09-16T07:53:18.634 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-volume-2:19.3.0-4879.g81fedcb3.el9.noarch 1/1 2024-09-16T07:53:18.635 INFO:teuthology.orchestra.run.smithi027.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T07:53:18.635 INFO:teuthology.orchestra.run.smithi027.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-16T07:53:18.635 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-16T07:53:18.744 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-16T07:53:18.744 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-16T07:53:18.744 INFO:teuthology.orchestra.run.smithi187.stdout:Installed: 2024-09-16T07:53:18.745 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mgr-diskprediction-local-2:19.3.0-4879.g81fedcb3.el9.noarch 2024-09-16T07:53:18.745 INFO:teuthology.orchestra.run.smithi187.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-16T07:53:18.745 INFO:teuthology.orchestra.run.smithi187.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-16T07:53:18.745 INFO:teuthology.orchestra.run.smithi187.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-16T07:53:18.745 INFO:teuthology.orchestra.run.smithi187.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-16T07:53:18.745 INFO:teuthology.orchestra.run.smithi187.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-16T07:53:18.745 INFO:teuthology.orchestra.run.smithi187.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-16T07:53:18.745 INFO:teuthology.orchestra.run.smithi187.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-16T07:53:18.745 INFO:teuthology.orchestra.run.smithi187.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-16T07:53:18.745 INFO:teuthology.orchestra.run.smithi187.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-16T07:53:18.745 INFO:teuthology.orchestra.run.smithi187.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-16T07:53:18.745 INFO:teuthology.orchestra.run.smithi187.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-16T07:53:18.745 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-16T07:53:18.746 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2024-09-16T07:53:19.127 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum -y install ceph-mgr-rook 2024-09-16T07:53:19.588 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-volume-2:19.3.0-4879.g81fedcb3.el9.noarch 1/1 2024-09-16T07:53:19.588 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-16T07:53:19.588 INFO:teuthology.orchestra.run.smithi027.stdout:Installed: 2024-09-16T07:53:19.588 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-volume-2:19.3.0-4879.g81fedcb3.el9.noarch 2024-09-16T07:53:19.588 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-16T07:53:19.588 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-09-16T07:53:19.708 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:02:10 ago on Mon 16 Sep 2024 07:51:09 AM UTC. 2024-09-16T07:53:19.842 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2024-09-16T07:53:19.844 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-09-16T07:53:19.844 INFO:teuthology.orchestra.run.smithi187.stdout: Package Arch Version Repository Size 2024-09-16T07:53:19.844 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-09-16T07:53:19.844 INFO:teuthology.orchestra.run.smithi187.stdout:Installing: 2024-09-16T07:53:19.844 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mgr-rook noarch 2:19.3.0-4879.g81fedcb3.el9 ceph-noarch 49 k 2024-09-16T07:53:19.844 INFO:teuthology.orchestra.run.smithi187.stdout:Installing dependencies: 2024-09-16T07:53:19.844 INFO:teuthology.orchestra.run.smithi187.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-16T07:53:19.844 INFO:teuthology.orchestra.run.smithi187.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-16T07:53:19.844 INFO:teuthology.orchestra.run.smithi187.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-16T07:53:19.844 INFO:teuthology.orchestra.run.smithi187.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-16T07:53:19.844 INFO:teuthology.orchestra.run.smithi187.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-16T07:53:19.844 INFO:teuthology.orchestra.run.smithi187.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-16T07:53:19.845 INFO:teuthology.orchestra.run.smithi187.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-16T07:53:19.845 INFO:teuthology.orchestra.run.smithi187.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-16T07:53:19.845 INFO:teuthology.orchestra.run.smithi187.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-16T07:53:19.845 INFO:teuthology.orchestra.run.smithi187.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-16T07:53:19.845 INFO:teuthology.orchestra.run.smithi187.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-16T07:53:19.845 INFO:teuthology.orchestra.run.smithi187.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-16T07:53:19.845 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-16T07:53:19.845 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction Summary 2024-09-16T07:53:19.845 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-09-16T07:53:19.845 INFO:teuthology.orchestra.run.smithi187.stdout:Install 13 Packages 2024-09-16T07:53:19.845 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-16T07:53:19.846 INFO:teuthology.orchestra.run.smithi187.stdout:Total download size: 2.2 M 2024-09-16T07:53:19.846 INFO:teuthology.orchestra.run.smithi187.stdout:Installed size: 26 M 2024-09-16T07:53:19.846 INFO:teuthology.orchestra.run.smithi187.stdout:Downloading Packages: 2024-09-16T07:53:19.887 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install librados-devel 2024-09-16T07:53:20.116 INFO:teuthology.orchestra.run.smithi187.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 302 kB/s | 26 kB 00:00 2024-09-16T07:53:20.142 INFO:teuthology.orchestra.run.smithi187.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 174 kB/s | 19 kB 00:00 2024-09-16T07:53:20.195 INFO:teuthology.orchestra.run.smithi187.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.9 MB/s | 222 kB 00:00 2024-09-16T07:53:20.225 INFO:teuthology.orchestra.run.smithi187.stdout:(4/13): ceph-mgr-rook-19.3.0-4879.g81fedcb3.el9 254 kB/s | 49 kB 00:00 2024-09-16T07:53:20.250 INFO:teuthology.orchestra.run.smithi187.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.4 MB/s | 159 kB 00:00 2024-09-16T07:53:20.275 INFO:teuthology.orchestra.run.smithi187.stdout:(6/13): python3-pyasn1-modules-0.4.8-6.el9.noar 3.2 MB/s | 279 kB 00:00 2024-09-16T07:53:20.317 INFO:teuthology.orchestra.run.smithi187.stdout:(7/13): python3-cachetools-4.2.4-1.el9.noarch.r 482 kB/s | 32 kB 00:00 2024-09-16T07:53:20.342 INFO:teuthology.orchestra.run.smithi187.stdout:(8/13): python3-certifi-2023.05.07-4.el9.noarch 234 kB/s | 14 kB 00:00 2024-09-16T07:53:20.368 INFO:teuthology.orchestra.run.smithi187.stdout:(9/13): python3-requests-oauthlib-1.3.0-12.el9. 375 kB/s | 54 kB 00:00 2024-09-16T07:53:20.401 INFO:teuthology.orchestra.run.smithi187.stdout:(10/13): python3-google-auth-2.34.0-1.el9.noarc 2.6 MB/s | 220 kB 00:00 2024-09-16T07:53:20.435 INFO:teuthology.orchestra.run.smithi187.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 2.8 MB/s | 90 kB 00:00 2024-09-16T07:53:20.460 INFO:teuthology.orchestra.run.smithi187.stdout:(12/13): python3-rsa-4.9-2.el9.noarch.rpm 643 kB/s | 59 kB 00:00 2024-09-16T07:53:20.483 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:02:08 ago on Mon 16 Sep 2024 07:51:12 AM UTC. 2024-09-16T07:53:20.502 INFO:teuthology.orchestra.run.smithi187.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 6.4 MB/s | 1.0 MB 00:00 2024-09-16T07:53:20.505 INFO:teuthology.orchestra.run.smithi187.stdout:-------------------------------------------------------------------------------- 2024-09-16T07:53:20.505 INFO:teuthology.orchestra.run.smithi187.stdout:Total 3.4 MB/s | 2.2 MB 00:00 2024-09-16T07:53:20.566 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction check 2024-09-16T07:53:20.581 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction check succeeded. 2024-09-16T07:53:20.581 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction test 2024-09-16T07:53:20.625 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-09-16T07:53:20.625 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-09-16T07:53:20.626 INFO:teuthology.orchestra.run.smithi027.stdout: Package Arch Version Repo Size 2024-09-16T07:53:20.626 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-09-16T07:53:20.626 INFO:teuthology.orchestra.run.smithi027.stdout:Installing: 2024-09-16T07:53:20.626 INFO:teuthology.orchestra.run.smithi027.stdout: librados-devel x86_64 2:19.3.0-4879.g81fedcb3.el9 ceph 125 k 2024-09-16T07:53:20.626 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-16T07:53:20.626 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2024-09-16T07:53:20.626 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-09-16T07:53:20.626 INFO:teuthology.orchestra.run.smithi027.stdout:Install 1 Package 2024-09-16T07:53:20.626 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-16T07:53:20.627 INFO:teuthology.orchestra.run.smithi027.stdout:Total download size: 125 k 2024-09-16T07:53:20.627 INFO:teuthology.orchestra.run.smithi027.stdout:Installed size: 445 k 2024-09-16T07:53:20.627 INFO:teuthology.orchestra.run.smithi027.stdout:Downloading Packages: 2024-09-16T07:53:20.670 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction test succeeded. 2024-09-16T07:53:20.672 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction 2024-09-16T07:53:20.887 INFO:teuthology.orchestra.run.smithi027.stdout:librados-devel-19.3.0-4879.g81fedcb3.el9.x86_64 479 kB/s | 125 kB 00:00 2024-09-16T07:53:20.888 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2024-09-16T07:53:20.888 INFO:teuthology.orchestra.run.smithi027.stdout:Total 477 kB/s | 125 kB 00:00 2024-09-16T07:53:20.888 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction check 2024-09-16T07:53:20.894 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction check succeeded. 2024-09-16T07:53:20.894 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction test 2024-09-16T07:53:20.962 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction test succeeded. 2024-09-16T07:53:20.962 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction 2024-09-16T07:53:21.131 INFO:teuthology.orchestra.run.smithi027.stdout: Preparing : 1/1 2024-09-16T07:53:21.171 INFO:teuthology.orchestra.run.smithi187.stdout: Preparing : 1/1 2024-09-16T07:53:21.246 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : librados-devel-2:19.3.0-4879.g81fedcb3.el9.x86_64 1/1 2024-09-16T07:53:21.332 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-16T07:53:21.386 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-16T07:53:21.442 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-16T07:53:21.479 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-16T07:53:21.521 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-16T07:53:21.622 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-16T07:53:21.724 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-16T07:53:21.733 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: librados-devel-2:19.3.0-4879.g81fedcb3.el9.x86_64 1/1 2024-09-16T07:53:21.780 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-16T07:53:21.997 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : librados-devel-2:19.3.0-4879.g81fedcb3.el9.x86_64 1/1 2024-09-16T07:53:21.997 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-16T07:53:21.997 INFO:teuthology.orchestra.run.smithi027.stdout:Installed: 2024-09-16T07:53:21.997 INFO:teuthology.orchestra.run.smithi027.stdout: librados-devel-2:19.3.0-4879.g81fedcb3.el9.x86_64 2024-09-16T07:53:21.997 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-16T07:53:21.997 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-09-16T07:53:22.258 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install libcephfs2 2024-09-16T07:53:22.499 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-16T07:53:22.553 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-16T07:53:22.594 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-16T07:53:22.642 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-16T07:53:22.664 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-mgr-rook-2:19.3.0-4879.g81fedcb3.el9.noarch 13/13 2024-09-16T07:53:22.848 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:02:10 ago on Mon 16 Sep 2024 07:51:12 AM UTC. 2024-09-16T07:53:22.941 INFO:teuthology.orchestra.run.smithi027.stdout:Package libcephfs2-2:19.3.0-4879.g81fedcb3.el9.x86_64 is already installed. 2024-09-16T07:53:22.986 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-09-16T07:53:22.988 INFO:teuthology.orchestra.run.smithi027.stdout:Nothing to do. 2024-09-16T07:53:22.988 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-09-16T07:53:23.067 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install libcephfs-devel 2024-09-16T07:53:23.576 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-mgr-rook-2:19.3.0-4879.g81fedcb3.el9.noarch 13/13 2024-09-16T07:53:23.576 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-mgr-rook-2:19.3.0-4879.g81fedcb3.el9.noarch 1/13 2024-09-16T07:53:23.577 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-16T07:53:23.577 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-16T07:53:23.577 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-16T07:53:23.577 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-16T07:53:23.577 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-16T07:53:23.578 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-16T07:53:23.578 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-16T07:53:23.578 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-16T07:53:23.578 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-16T07:53:23.578 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-16T07:53:23.578 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-16T07:53:23.660 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:02:11 ago on Mon 16 Sep 2024 07:51:12 AM UTC. 2024-09-16T07:53:23.801 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-09-16T07:53:23.802 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-09-16T07:53:23.802 INFO:teuthology.orchestra.run.smithi027.stdout: Package Arch Version Repo Size 2024-09-16T07:53:23.802 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-09-16T07:53:23.802 INFO:teuthology.orchestra.run.smithi027.stdout:Installing: 2024-09-16T07:53:23.803 INFO:teuthology.orchestra.run.smithi027.stdout: libcephfs-devel x86_64 2:19.3.0-4879.g81fedcb3.el9 ceph 32 k 2024-09-16T07:53:23.803 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-16T07:53:23.803 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2024-09-16T07:53:23.803 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-09-16T07:53:23.803 INFO:teuthology.orchestra.run.smithi027.stdout:Install 1 Package 2024-09-16T07:53:23.803 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-16T07:53:23.804 INFO:teuthology.orchestra.run.smithi027.stdout:Total download size: 32 k 2024-09-16T07:53:23.804 INFO:teuthology.orchestra.run.smithi027.stdout:Installed size: 142 k 2024-09-16T07:53:23.804 INFO:teuthology.orchestra.run.smithi027.stdout:Downloading Packages: 2024-09-16T07:53:23.908 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-16T07:53:23.908 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-16T07:53:23.908 INFO:teuthology.orchestra.run.smithi187.stdout:Installed: 2024-09-16T07:53:23.908 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mgr-rook-2:19.3.0-4879.g81fedcb3.el9.noarch 2024-09-16T07:53:23.908 INFO:teuthology.orchestra.run.smithi187.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-16T07:53:23.908 INFO:teuthology.orchestra.run.smithi187.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-16T07:53:23.908 INFO:teuthology.orchestra.run.smithi187.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-16T07:53:23.908 INFO:teuthology.orchestra.run.smithi187.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-16T07:53:23.909 INFO:teuthology.orchestra.run.smithi187.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-16T07:53:23.909 INFO:teuthology.orchestra.run.smithi187.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-16T07:53:23.909 INFO:teuthology.orchestra.run.smithi187.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-16T07:53:23.909 INFO:teuthology.orchestra.run.smithi187.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-16T07:53:23.909 INFO:teuthology.orchestra.run.smithi187.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-16T07:53:23.909 INFO:teuthology.orchestra.run.smithi187.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-16T07:53:23.909 INFO:teuthology.orchestra.run.smithi187.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-16T07:53:23.909 INFO:teuthology.orchestra.run.smithi187.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-16T07:53:23.909 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-16T07:53:23.909 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2024-09-16T07:53:24.099 INFO:teuthology.orchestra.run.smithi027.stdout:libcephfs-devel-19.3.0-4879.g81fedcb3.el9.x86_6 107 kB/s | 32 kB 00:00 2024-09-16T07:53:24.100 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2024-09-16T07:53:24.100 INFO:teuthology.orchestra.run.smithi027.stdout:Total 106 kB/s | 32 kB 00:00 2024-09-16T07:53:24.100 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction check 2024-09-16T07:53:24.104 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction check succeeded. 2024-09-16T07:53:24.104 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction test 2024-09-16T07:53:24.122 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction test succeeded. 2024-09-16T07:53:24.122 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction 2024-09-16T07:53:24.168 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum -y install ceph-mgr-cephadm 2024-09-16T07:53:24.205 INFO:teuthology.orchestra.run.smithi027.stdout: Preparing : 1/1 2024-09-16T07:53:24.324 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libcephfs-devel-2:19.3.0-4879.g81fedcb3.el9.x86_64 1/1 2024-09-16T07:53:24.689 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: libcephfs-devel-2:19.3.0-4879.g81fedcb3.el9.x86_64 1/1 2024-09-16T07:53:24.737 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:02:15 ago on Mon 16 Sep 2024 07:51:09 AM UTC. 2024-09-16T07:53:24.872 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2024-09-16T07:53:24.872 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-09-16T07:53:24.873 INFO:teuthology.orchestra.run.smithi187.stdout: Package Arch Version Repository Size 2024-09-16T07:53:24.873 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-09-16T07:53:24.873 INFO:teuthology.orchestra.run.smithi187.stdout:Installing: 2024-09-16T07:53:24.873 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mgr-cephadm noarch 2:19.3.0-4879.g81fedcb3.el9 ceph-noarch 154 k 2024-09-16T07:53:24.873 INFO:teuthology.orchestra.run.smithi187.stdout:Installing dependencies: 2024-09-16T07:53:24.873 INFO:teuthology.orchestra.run.smithi187.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-16T07:53:24.873 INFO:teuthology.orchestra.run.smithi187.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-16T07:53:24.873 INFO:teuthology.orchestra.run.smithi187.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-16T07:53:24.874 INFO:teuthology.orchestra.run.smithi187.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-16T07:53:24.874 INFO:teuthology.orchestra.run.smithi187.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-16T07:53:24.874 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-16T07:53:24.874 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction Summary 2024-09-16T07:53:24.874 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-09-16T07:53:24.874 INFO:teuthology.orchestra.run.smithi187.stdout:Install 6 Packages 2024-09-16T07:53:24.874 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-16T07:53:24.875 INFO:teuthology.orchestra.run.smithi187.stdout:Total download size: 7.0 M 2024-09-16T07:53:24.875 INFO:teuthology.orchestra.run.smithi187.stdout:Installed size: 33 M 2024-09-16T07:53:24.875 INFO:teuthology.orchestra.run.smithi187.stdout:Downloading Packages: 2024-09-16T07:53:24.921 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libcephfs-devel-2:19.3.0-4879.g81fedcb3.el9.x86_64 1/1 2024-09-16T07:53:24.921 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-16T07:53:24.921 INFO:teuthology.orchestra.run.smithi027.stdout:Installed: 2024-09-16T07:53:24.921 INFO:teuthology.orchestra.run.smithi027.stdout: libcephfs-devel-2:19.3.0-4879.g81fedcb3.el9.x86_64 2024-09-16T07:53:24.921 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-16T07:53:24.921 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-09-16T07:53:25.126 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install librados2 2024-09-16T07:53:25.303 INFO:teuthology.orchestra.run.smithi187.stdout:(1/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.7 MB/s | 248 kB 00:00 2024-09-16T07:53:25.387 INFO:teuthology.orchestra.run.smithi187.stdout:(2/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 6.4 MB/s | 548 kB 00:00 2024-09-16T07:53:25.420 INFO:teuthology.orchestra.run.smithi187.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 1.7 MB/s | 58 kB 00:00 2024-09-16T07:53:25.504 INFO:teuthology.orchestra.run.smithi187.stdout:(4/6): python3-babel-2.9.1-2.el9.noarch.rpm 17 MB/s | 6.0 MB 00:00 2024-09-16T07:53:25.529 INFO:teuthology.orchestra.run.smithi187.stdout:(5/6): python3-typing-extensions-4.12.2-2.el9.n 686 kB/s | 74 kB 00:00 2024-09-16T07:53:25.588 INFO:teuthology.orchestra.run.smithi187.stdout:(6/6): ceph-mgr-cephadm-19.3.0-4879.g81fedcb3.e 359 kB/s | 154 kB 00:00 2024-09-16T07:53:25.589 INFO:teuthology.orchestra.run.smithi187.stdout:-------------------------------------------------------------------------------- 2024-09-16T07:53:25.590 INFO:teuthology.orchestra.run.smithi187.stdout:Total 9.8 MB/s | 7.0 MB 00:00 2024-09-16T07:53:25.660 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction check 2024-09-16T07:53:25.671 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction check succeeded. 2024-09-16T07:53:25.671 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction test 2024-09-16T07:53:25.719 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:02:13 ago on Mon 16 Sep 2024 07:51:12 AM UTC. 2024-09-16T07:53:25.772 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction test succeeded. 2024-09-16T07:53:25.772 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction 2024-09-16T07:53:25.813 INFO:teuthology.orchestra.run.smithi027.stdout:Package librados2-2:19.3.0-4879.g81fedcb3.el9.x86_64 is already installed. 2024-09-16T07:53:25.858 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-09-16T07:53:25.859 INFO:teuthology.orchestra.run.smithi027.stdout:Nothing to do. 2024-09-16T07:53:25.859 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-09-16T07:53:25.938 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install librbd1 2024-09-16T07:53:26.076 INFO:teuthology.orchestra.run.smithi187.stdout: Preparing : 1/1 2024-09-16T07:53:26.270 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-16T07:53:26.314 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-16T07:53:26.528 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:02:14 ago on Mon 16 Sep 2024 07:51:12 AM UTC. 2024-09-16T07:53:26.622 INFO:teuthology.orchestra.run.smithi027.stdout:Package librbd1-2:19.3.0-4879.g81fedcb3.el9.x86_64 is already installed. 2024-09-16T07:53:26.668 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-09-16T07:53:26.669 INFO:teuthology.orchestra.run.smithi027.stdout:Nothing to do. 2024-09-16T07:53:26.669 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-09-16T07:53:26.733 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install python3-rados 2024-09-16T07:53:26.842 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-16T07:53:26.925 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-16T07:53:27.002 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-16T07:53:27.035 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-mgr-cephadm-2:19.3.0-4879.g81fedcb3.el9.noarch 6/6 2024-09-16T07:53:27.321 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:02:15 ago on Mon 16 Sep 2024 07:51:12 AM UTC. 2024-09-16T07:53:27.414 INFO:teuthology.orchestra.run.smithi027.stdout:Package python3-rados-2:19.3.0-4879.g81fedcb3.el9.x86_64 is already installed. 2024-09-16T07:53:27.459 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-09-16T07:53:27.460 INFO:teuthology.orchestra.run.smithi027.stdout:Nothing to do. 2024-09-16T07:53:27.460 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-09-16T07:53:27.543 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install python3-rgw 2024-09-16T07:53:27.984 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.3.0-4879.g81fedcb3.el9.noarch 6/6 2024-09-16T07:53:27.984 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-mgr-cephadm-2:19.3.0-4879.g81fedcb3.el9.noarch 1/6 2024-09-16T07:53:27.985 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-16T07:53:27.985 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-16T07:53:27.985 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-16T07:53:27.985 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-16T07:53:28.136 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:02:16 ago on Mon 16 Sep 2024 07:51:12 AM UTC. 2024-09-16T07:53:28.229 INFO:teuthology.orchestra.run.smithi027.stdout:Package python3-rgw-2:19.3.0-4879.g81fedcb3.el9.x86_64 is already installed. 2024-09-16T07:53:28.256 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-16T07:53:28.256 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-16T07:53:28.256 INFO:teuthology.orchestra.run.smithi187.stdout:Installed: 2024-09-16T07:53:28.256 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mgr-cephadm-2:19.3.0-4879.g81fedcb3.el9.noarch 2024-09-16T07:53:28.256 INFO:teuthology.orchestra.run.smithi187.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-16T07:53:28.257 INFO:teuthology.orchestra.run.smithi187.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-16T07:53:28.257 INFO:teuthology.orchestra.run.smithi187.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-16T07:53:28.257 INFO:teuthology.orchestra.run.smithi187.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-16T07:53:28.257 INFO:teuthology.orchestra.run.smithi187.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-16T07:53:28.257 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-16T07:53:28.257 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2024-09-16T07:53:28.275 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-09-16T07:53:28.277 INFO:teuthology.orchestra.run.smithi027.stdout:Nothing to do. 2024-09-16T07:53:28.277 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-09-16T07:53:28.350 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install python3-cephfs 2024-09-16T07:53:28.521 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum -y install ceph-fuse 2024-09-16T07:53:28.944 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:02:16 ago on Mon 16 Sep 2024 07:51:12 AM UTC. 2024-09-16T07:53:29.037 INFO:teuthology.orchestra.run.smithi027.stdout:Package python3-cephfs-2:19.3.0-4879.g81fedcb3.el9.x86_64 is already installed. 2024-09-16T07:53:29.083 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-09-16T07:53:29.084 INFO:teuthology.orchestra.run.smithi027.stdout:Nothing to do. 2024-09-16T07:53:29.085 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-09-16T07:53:29.088 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:02:20 ago on Mon 16 Sep 2024 07:51:09 AM UTC. 2024-09-16T07:53:29.161 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install python3-rbd 2024-09-16T07:53:29.221 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2024-09-16T07:53:29.221 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-09-16T07:53:29.221 INFO:teuthology.orchestra.run.smithi187.stdout: Package Arch Version Repository Size 2024-09-16T07:53:29.221 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-09-16T07:53:29.221 INFO:teuthology.orchestra.run.smithi187.stdout:Installing: 2024-09-16T07:53:29.221 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-fuse x86_64 2:19.3.0-4879.g81fedcb3.el9 ceph 884 k 2024-09-16T07:53:29.221 INFO:teuthology.orchestra.run.smithi187.stdout:Installing dependencies: 2024-09-16T07:53:29.222 INFO:teuthology.orchestra.run.smithi187.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-16T07:53:29.222 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-16T07:53:29.222 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction Summary 2024-09-16T07:53:29.222 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-09-16T07:53:29.222 INFO:teuthology.orchestra.run.smithi187.stdout:Install 2 Packages 2024-09-16T07:53:29.222 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-16T07:53:29.222 INFO:teuthology.orchestra.run.smithi187.stdout:Total download size: 964 k 2024-09-16T07:53:29.222 INFO:teuthology.orchestra.run.smithi187.stdout:Installed size: 4.1 M 2024-09-16T07:53:29.223 INFO:teuthology.orchestra.run.smithi187.stdout:Downloading Packages: 2024-09-16T07:53:29.748 INFO:teuthology.orchestra.run.smithi187.stdout:(1/2): ceph-fuse-19.3.0-4879.g81fedcb3.el9.x86_ 3.2 MB/s | 884 kB 00:00 2024-09-16T07:53:29.755 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:02:17 ago on Mon 16 Sep 2024 07:51:12 AM UTC. 2024-09-16T07:53:29.847 INFO:teuthology.orchestra.run.smithi027.stdout:Package python3-rbd-2:19.3.0-4879.g81fedcb3.el9.x86_64 is already installed. 2024-09-16T07:53:29.893 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-09-16T07:53:29.894 INFO:teuthology.orchestra.run.smithi027.stdout:Nothing to do. 2024-09-16T07:53:29.895 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-09-16T07:53:29.969 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install rbd-fuse 2024-09-16T07:53:30.107 INFO:teuthology.orchestra.run.smithi187.stdout:(2/2): fuse-2.9.9-16.el9.x86_64.rpm 127 kB/s | 80 kB 00:00 2024-09-16T07:53:30.107 INFO:teuthology.orchestra.run.smithi187.stdout:-------------------------------------------------------------------------------- 2024-09-16T07:53:30.108 INFO:teuthology.orchestra.run.smithi187.stdout:Total 1.1 MB/s | 964 kB 00:00 2024-09-16T07:53:30.120 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction check 2024-09-16T07:53:30.144 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction check succeeded. 2024-09-16T07:53:30.145 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction test 2024-09-16T07:53:30.226 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction test succeeded. 2024-09-16T07:53:30.227 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction 2024-09-16T07:53:30.410 INFO:teuthology.orchestra.run.smithi187.stdout: Preparing : 1/1 2024-09-16T07:53:30.552 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-16T07:53:30.562 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:02:18 ago on Mon 16 Sep 2024 07:51:12 AM UTC. 2024-09-16T07:53:30.622 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-fuse-2:19.3.0-4879.g81fedcb3.el9.x86_64 2/2 2024-09-16T07:53:30.702 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-09-16T07:53:30.703 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-09-16T07:53:30.703 INFO:teuthology.orchestra.run.smithi027.stdout: Package Architecture Version Repository Size 2024-09-16T07:53:30.703 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-09-16T07:53:30.703 INFO:teuthology.orchestra.run.smithi027.stdout:Installing: 2024-09-16T07:53:30.704 INFO:teuthology.orchestra.run.smithi027.stdout: rbd-fuse x86_64 2:19.3.0-4879.g81fedcb3.el9 ceph 91 k 2024-09-16T07:53:30.704 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-16T07:53:30.704 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2024-09-16T07:53:30.704 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-09-16T07:53:30.704 INFO:teuthology.orchestra.run.smithi027.stdout:Install 1 Package 2024-09-16T07:53:30.704 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-16T07:53:30.704 INFO:teuthology.orchestra.run.smithi027.stdout:Total download size: 91 k 2024-09-16T07:53:30.704 INFO:teuthology.orchestra.run.smithi027.stdout:Installed size: 321 k 2024-09-16T07:53:30.704 INFO:teuthology.orchestra.run.smithi027.stdout:Downloading Packages: 2024-09-16T07:53:30.902 INFO:teuthology.orchestra.run.smithi027.stdout:rbd-fuse-19.3.0-4879.g81fedcb3.el9.x86_64.rpm 459 kB/s | 91 kB 00:00 2024-09-16T07:53:30.903 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2024-09-16T07:53:30.903 INFO:teuthology.orchestra.run.smithi027.stdout:Total 456 kB/s | 91 kB 00:00 2024-09-16T07:53:30.903 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction check 2024-09-16T07:53:30.913 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction check succeeded. 2024-09-16T07:53:30.913 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction test 2024-09-16T07:53:30.980 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction test succeeded. 2024-09-16T07:53:30.981 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction 2024-09-16T07:53:31.137 INFO:teuthology.orchestra.run.smithi027.stdout: Preparing : 1/1 2024-09-16T07:53:31.228 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : rbd-fuse-2:19.3.0-4879.g81fedcb3.el9.x86_64 1/1 2024-09-16T07:53:31.330 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-fuse-2:19.3.0-4879.g81fedcb3.el9.x86_64 2/2 2024-09-16T07:53:31.330 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-fuse-2:19.3.0-4879.g81fedcb3.el9.x86_64 1/2 2024-09-16T07:53:31.607 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-16T07:53:31.607 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-16T07:53:31.607 INFO:teuthology.orchestra.run.smithi187.stdout:Installed: 2024-09-16T07:53:31.607 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-fuse-2:19.3.0-4879.g81fedcb3.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-16T07:53:31.608 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-16T07:53:31.608 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2024-09-16T07:53:31.638 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: rbd-fuse-2:19.3.0-4879.g81fedcb3.el9.x86_64 1/1 2024-09-16T07:53:31.911 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum -y install ceph-volume 2024-09-16T07:53:31.912 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : rbd-fuse-2:19.3.0-4879.g81fedcb3.el9.x86_64 1/1 2024-09-16T07:53:31.912 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-16T07:53:31.912 INFO:teuthology.orchestra.run.smithi027.stdout:Installed: 2024-09-16T07:53:31.912 INFO:teuthology.orchestra.run.smithi027.stdout: rbd-fuse-2:19.3.0-4879.g81fedcb3.el9.x86_64 2024-09-16T07:53:31.913 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-16T07:53:31.913 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-09-16T07:53:32.202 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install rbd-mirror 2024-09-16T07:53:32.496 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:02:23 ago on Mon 16 Sep 2024 07:51:09 AM UTC. 2024-09-16T07:53:32.633 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2024-09-16T07:53:32.634 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-09-16T07:53:32.634 INFO:teuthology.orchestra.run.smithi187.stdout: Package Arch Version Repository Size 2024-09-16T07:53:32.634 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-09-16T07:53:32.634 INFO:teuthology.orchestra.run.smithi187.stdout:Installing: 2024-09-16T07:53:32.634 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-volume noarch 2:19.3.0-4879.g81fedcb3.el9 ceph-noarch 285 k 2024-09-16T07:53:32.635 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-16T07:53:32.635 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction Summary 2024-09-16T07:53:32.635 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-09-16T07:53:32.635 INFO:teuthology.orchestra.run.smithi187.stdout:Install 1 Package 2024-09-16T07:53:32.635 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-16T07:53:32.635 INFO:teuthology.orchestra.run.smithi187.stdout:Total download size: 285 k 2024-09-16T07:53:32.635 INFO:teuthology.orchestra.run.smithi187.stdout:Installed size: 1.3 M 2024-09-16T07:53:32.635 INFO:teuthology.orchestra.run.smithi187.stdout:Downloading Packages: 2024-09-16T07:53:32.818 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:02:20 ago on Mon 16 Sep 2024 07:51:12 AM UTC. 2024-09-16T07:53:32.877 INFO:teuthology.orchestra.run.smithi187.stdout:ceph-volume-19.3.0-4879.g81fedcb3.el9.noarch.rp 1.2 MB/s | 285 kB 00:00 2024-09-16T07:53:32.878 INFO:teuthology.orchestra.run.smithi187.stdout:-------------------------------------------------------------------------------- 2024-09-16T07:53:32.878 INFO:teuthology.orchestra.run.smithi187.stdout:Total 1.1 MB/s | 285 kB 00:00 2024-09-16T07:53:32.878 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction check 2024-09-16T07:53:32.887 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction check succeeded. 2024-09-16T07:53:32.887 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction test 2024-09-16T07:53:32.922 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction test succeeded. 2024-09-16T07:53:32.923 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction 2024-09-16T07:53:32.962 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-09-16T07:53:32.963 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-09-16T07:53:32.963 INFO:teuthology.orchestra.run.smithi027.stdout: Package Arch Version Repo Size 2024-09-16T07:53:32.963 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-09-16T07:53:32.963 INFO:teuthology.orchestra.run.smithi027.stdout:Installing: 2024-09-16T07:53:32.963 INFO:teuthology.orchestra.run.smithi027.stdout: rbd-mirror x86_64 2:19.3.0-4879.g81fedcb3.el9 ceph 3.5 M 2024-09-16T07:53:32.963 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-16T07:53:32.964 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2024-09-16T07:53:32.964 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-09-16T07:53:32.964 INFO:teuthology.orchestra.run.smithi027.stdout:Install 1 Package 2024-09-16T07:53:32.964 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-16T07:53:32.964 INFO:teuthology.orchestra.run.smithi027.stdout:Total download size: 3.5 M 2024-09-16T07:53:32.964 INFO:teuthology.orchestra.run.smithi027.stdout:Installed size: 30 M 2024-09-16T07:53:32.964 INFO:teuthology.orchestra.run.smithi027.stdout:Downloading Packages: 2024-09-16T07:53:33.143 INFO:teuthology.orchestra.run.smithi187.stdout: Preparing : 1/1 2024-09-16T07:53:33.229 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-volume-2:19.3.0-4879.g81fedcb3.el9.noarch 1/1 2024-09-16T07:53:33.246 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-volume-2:19.3.0-4879.g81fedcb3.el9.noarch 1/1 2024-09-16T07:53:33.246 INFO:teuthology.orchestra.run.smithi187.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T07:53:33.246 INFO:teuthology.orchestra.run.smithi187.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-16T07:53:33.246 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-16T07:53:33.298 INFO:teuthology.orchestra.run.smithi027.stdout:rbd-mirror-19.3.0-4879.g81fedcb3.el9.x86_64.rpm 11 MB/s | 3.5 MB 00:00 2024-09-16T07:53:33.298 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2024-09-16T07:53:33.299 INFO:teuthology.orchestra.run.smithi027.stdout:Total 10 MB/s | 3.5 MB 00:00 2024-09-16T07:53:33.299 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction check 2024-09-16T07:53:33.312 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction check succeeded. 2024-09-16T07:53:33.312 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction test 2024-09-16T07:53:33.400 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction test succeeded. 2024-09-16T07:53:33.400 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction 2024-09-16T07:53:34.029 INFO:teuthology.orchestra.run.smithi027.stdout: Preparing : 1/1 2024-09-16T07:53:34.159 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : rbd-mirror-2:19.3.0-4879.g81fedcb3.el9.x86_64 1/1 2024-09-16T07:53:34.159 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: rbd-mirror-2:19.3.0-4879.g81fedcb3.el9.x86_64 1/1 2024-09-16T07:53:34.159 INFO:teuthology.orchestra.run.smithi027.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T07:53:34.159 INFO:teuthology.orchestra.run.smithi027.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-16T07:53:34.160 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-16T07:53:34.160 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-16T07:53:34.160 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-16T07:53:34.266 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-volume-2:19.3.0-4879.g81fedcb3.el9.noarch 1/1 2024-09-16T07:53:34.266 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-16T07:53:34.266 INFO:teuthology.orchestra.run.smithi187.stdout:Installed: 2024-09-16T07:53:34.266 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-volume-2:19.3.0-4879.g81fedcb3.el9.noarch 2024-09-16T07:53:34.266 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-16T07:53:34.267 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2024-09-16T07:53:34.559 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum -y install librados-devel 2024-09-16T07:53:35.141 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:02:26 ago on Mon 16 Sep 2024 07:51:09 AM UTC. 2024-09-16T07:53:35.276 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2024-09-16T07:53:35.277 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-09-16T07:53:35.277 INFO:teuthology.orchestra.run.smithi187.stdout: Package Arch Version Repo Size 2024-09-16T07:53:35.277 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-09-16T07:53:35.277 INFO:teuthology.orchestra.run.smithi187.stdout:Installing: 2024-09-16T07:53:35.277 INFO:teuthology.orchestra.run.smithi187.stdout: librados-devel x86_64 2:19.3.0-4879.g81fedcb3.el9 ceph 125 k 2024-09-16T07:53:35.277 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-16T07:53:35.278 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction Summary 2024-09-16T07:53:35.278 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-09-16T07:53:35.278 INFO:teuthology.orchestra.run.smithi187.stdout:Install 1 Package 2024-09-16T07:53:35.278 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-16T07:53:35.278 INFO:teuthology.orchestra.run.smithi187.stdout:Total download size: 125 k 2024-09-16T07:53:35.278 INFO:teuthology.orchestra.run.smithi187.stdout:Installed size: 445 k 2024-09-16T07:53:35.278 INFO:teuthology.orchestra.run.smithi187.stdout:Downloading Packages: 2024-09-16T07:53:35.382 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : rbd-mirror-2:19.3.0-4879.g81fedcb3.el9.x86_64 1/1 2024-09-16T07:53:35.382 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-16T07:53:35.382 INFO:teuthology.orchestra.run.smithi027.stdout:Installed: 2024-09-16T07:53:35.382 INFO:teuthology.orchestra.run.smithi027.stdout: rbd-mirror-2:19.3.0-4879.g81fedcb3.el9.x86_64 2024-09-16T07:53:35.382 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-16T07:53:35.382 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-09-16T07:53:35.465 INFO:teuthology.orchestra.run.smithi187.stdout:librados-devel-19.3.0-4879.g81fedcb3.el9.x86_64 665 kB/s | 125 kB 00:00 2024-09-16T07:53:35.466 INFO:teuthology.orchestra.run.smithi187.stdout:-------------------------------------------------------------------------------- 2024-09-16T07:53:35.466 INFO:teuthology.orchestra.run.smithi187.stdout:Total 661 kB/s | 125 kB 00:00 2024-09-16T07:53:35.466 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction check 2024-09-16T07:53:35.472 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction check succeeded. 2024-09-16T07:53:35.472 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction test 2024-09-16T07:53:35.535 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction test succeeded. 2024-09-16T07:53:35.535 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction 2024-09-16T07:53:35.691 INFO:teuthology.orchestra.run.smithi187.stdout: Preparing : 1/1 2024-09-16T07:53:35.763 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install rbd-nbd 2024-09-16T07:53:35.781 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : librados-devel-2:19.3.0-4879.g81fedcb3.el9.x86_64 1/1 2024-09-16T07:53:36.231 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: librados-devel-2:19.3.0-4879.g81fedcb3.el9.x86_64 1/1 2024-09-16T07:53:36.359 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:02:24 ago on Mon 16 Sep 2024 07:51:12 AM UTC. 2024-09-16T07:53:36.489 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : librados-devel-2:19.3.0-4879.g81fedcb3.el9.x86_64 1/1 2024-09-16T07:53:36.489 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-16T07:53:36.489 INFO:teuthology.orchestra.run.smithi187.stdout:Installed: 2024-09-16T07:53:36.490 INFO:teuthology.orchestra.run.smithi187.stdout: librados-devel-2:19.3.0-4879.g81fedcb3.el9.x86_64 2024-09-16T07:53:36.490 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-16T07:53:36.490 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2024-09-16T07:53:36.496 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-09-16T07:53:36.497 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-09-16T07:53:36.497 INFO:teuthology.orchestra.run.smithi027.stdout: Package Architecture Version Repository Size 2024-09-16T07:53:36.497 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-09-16T07:53:36.497 INFO:teuthology.orchestra.run.smithi027.stdout:Installing: 2024-09-16T07:53:36.497 INFO:teuthology.orchestra.run.smithi027.stdout: rbd-nbd x86_64 2:19.3.0-4879.g81fedcb3.el9 ceph 177 k 2024-09-16T07:53:36.497 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-16T07:53:36.497 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2024-09-16T07:53:36.497 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-09-16T07:53:36.498 INFO:teuthology.orchestra.run.smithi027.stdout:Install 1 Package 2024-09-16T07:53:36.498 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-16T07:53:36.498 INFO:teuthology.orchestra.run.smithi027.stdout:Total download size: 177 k 2024-09-16T07:53:36.498 INFO:teuthology.orchestra.run.smithi027.stdout:Installed size: 685 k 2024-09-16T07:53:36.498 INFO:teuthology.orchestra.run.smithi027.stdout:Downloading Packages: 2024-09-16T07:53:36.719 INFO:teuthology.orchestra.run.smithi027.stdout:rbd-nbd-19.3.0-4879.g81fedcb3.el9.x86_64.rpm 800 kB/s | 177 kB 00:00 2024-09-16T07:53:36.720 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2024-09-16T07:53:36.720 INFO:teuthology.orchestra.run.smithi027.stdout:Total 795 kB/s | 177 kB 00:00 2024-09-16T07:53:36.720 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction check 2024-09-16T07:53:36.731 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction check succeeded. 2024-09-16T07:53:36.731 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction test 2024-09-16T07:53:36.749 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum -y install libcephfs2 2024-09-16T07:53:36.805 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction test succeeded. 2024-09-16T07:53:36.805 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction 2024-09-16T07:53:36.983 INFO:teuthology.orchestra.run.smithi027.stdout: Preparing : 1/1 2024-09-16T07:53:37.078 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : rbd-nbd-2:19.3.0-4879.g81fedcb3.el9.x86_64 1/1 2024-09-16T07:53:37.330 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:02:28 ago on Mon 16 Sep 2024 07:51:09 AM UTC. 2024-09-16T07:53:37.424 INFO:teuthology.orchestra.run.smithi187.stdout:Package libcephfs2-2:19.3.0-4879.g81fedcb3.el9.x86_64 is already installed. 2024-09-16T07:53:37.470 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2024-09-16T07:53:37.472 INFO:teuthology.orchestra.run.smithi187.stdout:Nothing to do. 2024-09-16T07:53:37.472 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2024-09-16T07:53:37.522 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: rbd-nbd-2:19.3.0-4879.g81fedcb3.el9.x86_64 1/1 2024-09-16T07:53:37.543 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum -y install libcephfs-devel 2024-09-16T07:53:37.752 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : rbd-nbd-2:19.3.0-4879.g81fedcb3.el9.x86_64 1/1 2024-09-16T07:53:37.752 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-16T07:53:37.752 INFO:teuthology.orchestra.run.smithi027.stdout:Installed: 2024-09-16T07:53:37.753 INFO:teuthology.orchestra.run.smithi027.stdout: rbd-nbd-2:19.3.0-4879.g81fedcb3.el9.x86_64 2024-09-16T07:53:37.753 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-16T07:53:37.753 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-09-16T07:53:38.022 DEBUG:teuthology.parallel:result is None 2024-09-16T07:53:38.111 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:02:29 ago on Mon 16 Sep 2024 07:51:09 AM UTC. 2024-09-16T07:53:38.246 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2024-09-16T07:53:38.247 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-09-16T07:53:38.247 INFO:teuthology.orchestra.run.smithi187.stdout: Package Arch Version Repo Size 2024-09-16T07:53:38.247 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-09-16T07:53:38.247 INFO:teuthology.orchestra.run.smithi187.stdout:Installing: 2024-09-16T07:53:38.247 INFO:teuthology.orchestra.run.smithi187.stdout: libcephfs-devel x86_64 2:19.3.0-4879.g81fedcb3.el9 ceph 32 k 2024-09-16T07:53:38.247 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-16T07:53:38.247 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction Summary 2024-09-16T07:53:38.247 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-09-16T07:53:38.248 INFO:teuthology.orchestra.run.smithi187.stdout:Install 1 Package 2024-09-16T07:53:38.248 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-16T07:53:38.248 INFO:teuthology.orchestra.run.smithi187.stdout:Total download size: 32 k 2024-09-16T07:53:38.248 INFO:teuthology.orchestra.run.smithi187.stdout:Installed size: 142 k 2024-09-16T07:53:38.248 INFO:teuthology.orchestra.run.smithi187.stdout:Downloading Packages: 2024-09-16T07:53:38.410 INFO:teuthology.orchestra.run.smithi187.stdout:libcephfs-devel-19.3.0-4879.g81fedcb3.el9.x86_6 195 kB/s | 32 kB 00:00 2024-09-16T07:53:38.411 INFO:teuthology.orchestra.run.smithi187.stdout:-------------------------------------------------------------------------------- 2024-09-16T07:53:38.411 INFO:teuthology.orchestra.run.smithi187.stdout:Total 194 kB/s | 32 kB 00:00 2024-09-16T07:53:38.411 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction check 2024-09-16T07:53:38.414 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction check succeeded. 2024-09-16T07:53:38.415 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction test 2024-09-16T07:53:38.431 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction test succeeded. 2024-09-16T07:53:38.431 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction 2024-09-16T07:53:38.521 INFO:teuthology.orchestra.run.smithi187.stdout: Preparing : 1/1 2024-09-16T07:53:38.632 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : libcephfs-devel-2:19.3.0-4879.g81fedcb3.el9.x86_64 1/1 2024-09-16T07:53:39.006 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: libcephfs-devel-2:19.3.0-4879.g81fedcb3.el9.x86_64 1/1 2024-09-16T07:53:39.238 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : libcephfs-devel-2:19.3.0-4879.g81fedcb3.el9.x86_64 1/1 2024-09-16T07:53:39.238 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-16T07:53:39.238 INFO:teuthology.orchestra.run.smithi187.stdout:Installed: 2024-09-16T07:53:39.238 INFO:teuthology.orchestra.run.smithi187.stdout: libcephfs-devel-2:19.3.0-4879.g81fedcb3.el9.x86_64 2024-09-16T07:53:39.238 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-16T07:53:39.239 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2024-09-16T07:53:39.415 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum -y install librados2 2024-09-16T07:53:39.983 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:02:30 ago on Mon 16 Sep 2024 07:51:09 AM UTC. 2024-09-16T07:53:40.074 INFO:teuthology.orchestra.run.smithi187.stdout:Package librados2-2:19.3.0-4879.g81fedcb3.el9.x86_64 is already installed. 2024-09-16T07:53:40.118 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2024-09-16T07:53:40.119 INFO:teuthology.orchestra.run.smithi187.stdout:Nothing to do. 2024-09-16T07:53:40.119 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2024-09-16T07:53:40.214 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum -y install librbd1 2024-09-16T07:53:40.788 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:02:31 ago on Mon 16 Sep 2024 07:51:09 AM UTC. 2024-09-16T07:53:40.879 INFO:teuthology.orchestra.run.smithi187.stdout:Package librbd1-2:19.3.0-4879.g81fedcb3.el9.x86_64 is already installed. 2024-09-16T07:53:40.922 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2024-09-16T07:53:40.923 INFO:teuthology.orchestra.run.smithi187.stdout:Nothing to do. 2024-09-16T07:53:40.923 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2024-09-16T07:53:40.995 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum -y install python3-rados 2024-09-16T07:53:41.564 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:02:32 ago on Mon 16 Sep 2024 07:51:09 AM UTC. 2024-09-16T07:53:41.655 INFO:teuthology.orchestra.run.smithi187.stdout:Package python3-rados-2:19.3.0-4879.g81fedcb3.el9.x86_64 is already installed. 2024-09-16T07:53:41.698 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2024-09-16T07:53:41.699 INFO:teuthology.orchestra.run.smithi187.stdout:Nothing to do. 2024-09-16T07:53:41.699 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2024-09-16T07:53:41.770 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum -y install python3-rgw 2024-09-16T07:53:42.340 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:02:33 ago on Mon 16 Sep 2024 07:51:09 AM UTC. 2024-09-16T07:53:42.430 INFO:teuthology.orchestra.run.smithi187.stdout:Package python3-rgw-2:19.3.0-4879.g81fedcb3.el9.x86_64 is already installed. 2024-09-16T07:53:42.474 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2024-09-16T07:53:42.475 INFO:teuthology.orchestra.run.smithi187.stdout:Nothing to do. 2024-09-16T07:53:42.475 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2024-09-16T07:53:42.543 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum -y install python3-cephfs 2024-09-16T07:53:43.113 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:02:34 ago on Mon 16 Sep 2024 07:51:09 AM UTC. 2024-09-16T07:53:43.203 INFO:teuthology.orchestra.run.smithi187.stdout:Package python3-cephfs-2:19.3.0-4879.g81fedcb3.el9.x86_64 is already installed. 2024-09-16T07:53:43.247 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2024-09-16T07:53:43.247 INFO:teuthology.orchestra.run.smithi187.stdout:Nothing to do. 2024-09-16T07:53:43.248 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2024-09-16T07:53:43.318 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum -y install python3-rbd 2024-09-16T07:53:43.888 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:02:34 ago on Mon 16 Sep 2024 07:51:09 AM UTC. 2024-09-16T07:53:43.978 INFO:teuthology.orchestra.run.smithi187.stdout:Package python3-rbd-2:19.3.0-4879.g81fedcb3.el9.x86_64 is already installed. 2024-09-16T07:53:44.021 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2024-09-16T07:53:44.022 INFO:teuthology.orchestra.run.smithi187.stdout:Nothing to do. 2024-09-16T07:53:44.022 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2024-09-16T07:53:44.097 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum -y install rbd-fuse 2024-09-16T07:53:44.668 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:02:35 ago on Mon 16 Sep 2024 07:51:09 AM UTC. 2024-09-16T07:53:44.802 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2024-09-16T07:53:44.802 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-09-16T07:53:44.802 INFO:teuthology.orchestra.run.smithi187.stdout: Package Architecture Version Repository Size 2024-09-16T07:53:44.802 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-09-16T07:53:44.803 INFO:teuthology.orchestra.run.smithi187.stdout:Installing: 2024-09-16T07:53:44.803 INFO:teuthology.orchestra.run.smithi187.stdout: rbd-fuse x86_64 2:19.3.0-4879.g81fedcb3.el9 ceph 91 k 2024-09-16T07:53:44.803 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-16T07:53:44.803 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction Summary 2024-09-16T07:53:44.803 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-09-16T07:53:44.803 INFO:teuthology.orchestra.run.smithi187.stdout:Install 1 Package 2024-09-16T07:53:44.803 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-16T07:53:44.803 INFO:teuthology.orchestra.run.smithi187.stdout:Total download size: 91 k 2024-09-16T07:53:44.803 INFO:teuthology.orchestra.run.smithi187.stdout:Installed size: 321 k 2024-09-16T07:53:44.803 INFO:teuthology.orchestra.run.smithi187.stdout:Downloading Packages: 2024-09-16T07:53:44.988 INFO:teuthology.orchestra.run.smithi187.stdout:rbd-fuse-19.3.0-4879.g81fedcb3.el9.x86_64.rpm 492 kB/s | 91 kB 00:00 2024-09-16T07:53:44.989 INFO:teuthology.orchestra.run.smithi187.stdout:-------------------------------------------------------------------------------- 2024-09-16T07:53:44.989 INFO:teuthology.orchestra.run.smithi187.stdout:Total 488 kB/s | 91 kB 00:00 2024-09-16T07:53:44.989 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction check 2024-09-16T07:53:44.998 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction check succeeded. 2024-09-16T07:53:44.998 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction test 2024-09-16T07:53:45.061 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction test succeeded. 2024-09-16T07:53:45.062 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction 2024-09-16T07:53:45.219 INFO:teuthology.orchestra.run.smithi187.stdout: Preparing : 1/1 2024-09-16T07:53:45.311 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : rbd-fuse-2:19.3.0-4879.g81fedcb3.el9.x86_64 1/1 2024-09-16T07:53:45.707 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: rbd-fuse-2:19.3.0-4879.g81fedcb3.el9.x86_64 1/1 2024-09-16T07:53:45.978 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : rbd-fuse-2:19.3.0-4879.g81fedcb3.el9.x86_64 1/1 2024-09-16T07:53:45.979 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-16T07:53:45.979 INFO:teuthology.orchestra.run.smithi187.stdout:Installed: 2024-09-16T07:53:45.979 INFO:teuthology.orchestra.run.smithi187.stdout: rbd-fuse-2:19.3.0-4879.g81fedcb3.el9.x86_64 2024-09-16T07:53:45.979 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-16T07:53:45.979 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2024-09-16T07:53:46.242 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum -y install rbd-mirror 2024-09-16T07:53:46.847 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:02:37 ago on Mon 16 Sep 2024 07:51:09 AM UTC. 2024-09-16T07:53:46.989 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2024-09-16T07:53:46.990 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-09-16T07:53:46.990 INFO:teuthology.orchestra.run.smithi187.stdout: Package Arch Version Repo Size 2024-09-16T07:53:46.990 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-09-16T07:53:46.990 INFO:teuthology.orchestra.run.smithi187.stdout:Installing: 2024-09-16T07:53:46.990 INFO:teuthology.orchestra.run.smithi187.stdout: rbd-mirror x86_64 2:19.3.0-4879.g81fedcb3.el9 ceph 3.5 M 2024-09-16T07:53:46.990 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-16T07:53:46.990 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction Summary 2024-09-16T07:53:46.990 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-09-16T07:53:46.990 INFO:teuthology.orchestra.run.smithi187.stdout:Install 1 Package 2024-09-16T07:53:46.991 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-16T07:53:46.991 INFO:teuthology.orchestra.run.smithi187.stdout:Total download size: 3.5 M 2024-09-16T07:53:46.991 INFO:teuthology.orchestra.run.smithi187.stdout:Installed size: 30 M 2024-09-16T07:53:46.991 INFO:teuthology.orchestra.run.smithi187.stdout:Downloading Packages: 2024-09-16T07:53:47.325 INFO:teuthology.orchestra.run.smithi187.stdout:rbd-mirror-19.3.0-4879.g81fedcb3.el9.x86_64.rpm 10 MB/s | 3.5 MB 00:00 2024-09-16T07:53:47.326 INFO:teuthology.orchestra.run.smithi187.stdout:-------------------------------------------------------------------------------- 2024-09-16T07:53:47.326 INFO:teuthology.orchestra.run.smithi187.stdout:Total 10 MB/s | 3.5 MB 00:00 2024-09-16T07:53:47.326 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction check 2024-09-16T07:53:47.338 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction check succeeded. 2024-09-16T07:53:47.339 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction test 2024-09-16T07:53:47.425 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction test succeeded. 2024-09-16T07:53:47.425 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction 2024-09-16T07:53:48.022 INFO:teuthology.orchestra.run.smithi187.stdout: Preparing : 1/1 2024-09-16T07:53:48.076 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : rbd-mirror-2:19.3.0-4879.g81fedcb3.el9.x86_64 1/1 2024-09-16T07:53:48.102 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: rbd-mirror-2:19.3.0-4879.g81fedcb3.el9.x86_64 1/1 2024-09-16T07:53:48.102 INFO:teuthology.orchestra.run.smithi187.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T07:53:48.103 INFO:teuthology.orchestra.run.smithi187.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-16T07:53:48.103 INFO:teuthology.orchestra.run.smithi187.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-16T07:53:48.103 INFO:teuthology.orchestra.run.smithi187.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-16T07:53:48.103 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-16T07:53:49.388 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : rbd-mirror-2:19.3.0-4879.g81fedcb3.el9.x86_64 1/1 2024-09-16T07:53:49.388 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-16T07:53:49.388 INFO:teuthology.orchestra.run.smithi187.stdout:Installed: 2024-09-16T07:53:49.388 INFO:teuthology.orchestra.run.smithi187.stdout: rbd-mirror-2:19.3.0-4879.g81fedcb3.el9.x86_64 2024-09-16T07:53:49.389 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-16T07:53:49.389 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2024-09-16T07:53:49.659 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum -y install rbd-nbd 2024-09-16T07:53:50.237 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:02:41 ago on Mon 16 Sep 2024 07:51:09 AM UTC. 2024-09-16T07:53:50.373 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2024-09-16T07:53:50.374 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-09-16T07:53:50.375 INFO:teuthology.orchestra.run.smithi187.stdout: Package Architecture Version Repository Size 2024-09-16T07:53:50.375 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-09-16T07:53:50.375 INFO:teuthology.orchestra.run.smithi187.stdout:Installing: 2024-09-16T07:53:50.375 INFO:teuthology.orchestra.run.smithi187.stdout: rbd-nbd x86_64 2:19.3.0-4879.g81fedcb3.el9 ceph 177 k 2024-09-16T07:53:50.375 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-16T07:53:50.375 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction Summary 2024-09-16T07:53:50.375 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-09-16T07:53:50.375 INFO:teuthology.orchestra.run.smithi187.stdout:Install 1 Package 2024-09-16T07:53:50.375 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-16T07:53:50.375 INFO:teuthology.orchestra.run.smithi187.stdout:Total download size: 177 k 2024-09-16T07:53:50.376 INFO:teuthology.orchestra.run.smithi187.stdout:Installed size: 685 k 2024-09-16T07:53:50.376 INFO:teuthology.orchestra.run.smithi187.stdout:Downloading Packages: 2024-09-16T07:53:50.587 INFO:teuthology.orchestra.run.smithi187.stdout:rbd-nbd-19.3.0-4879.g81fedcb3.el9.x86_64.rpm 834 kB/s | 177 kB 00:00 2024-09-16T07:53:50.587 INFO:teuthology.orchestra.run.smithi187.stdout:-------------------------------------------------------------------------------- 2024-09-16T07:53:50.588 INFO:teuthology.orchestra.run.smithi187.stdout:Total 828 kB/s | 177 kB 00:00 2024-09-16T07:53:50.588 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction check 2024-09-16T07:53:50.597 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction check succeeded. 2024-09-16T07:53:50.597 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction test 2024-09-16T07:53:50.667 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction test succeeded. 2024-09-16T07:53:50.667 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction 2024-09-16T07:53:50.833 INFO:teuthology.orchestra.run.smithi187.stdout: Preparing : 1/1 2024-09-16T07:53:50.926 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : rbd-nbd-2:19.3.0-4879.g81fedcb3.el9.x86_64 1/1 2024-09-16T07:53:51.370 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: rbd-nbd-2:19.3.0-4879.g81fedcb3.el9.x86_64 1/1 2024-09-16T07:53:51.622 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : rbd-nbd-2:19.3.0-4879.g81fedcb3.el9.x86_64 1/1 2024-09-16T07:53:51.622 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-16T07:53:51.622 INFO:teuthology.orchestra.run.smithi187.stdout:Installed: 2024-09-16T07:53:51.622 INFO:teuthology.orchestra.run.smithi187.stdout: rbd-nbd-2:19.3.0-4879.g81fedcb3.el9.x86_64 2024-09-16T07:53:51.622 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-16T07:53:51.622 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2024-09-16T07:53:51.865 DEBUG:teuthology.parallel:result is None 2024-09-16T07:53:51.866 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=81fedcb3b484c843d22ffbc8d179cb567b7083b5 2024-09-16T07:53:52.026 DEBUG:teuthology.orchestra.run.smithi027:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-16T07:53:52.053 INFO:teuthology.orchestra.run.smithi027.stdout:19.3.0-4879.g81fedcb3.el9 2024-09-16T07:53:52.054 INFO:teuthology.packaging:The installed version of ceph is 19.3.0-4879.g81fedcb3.el9 2024-09-16T07:53:52.054 INFO:teuthology.task.install:The correct ceph version 19.3.0-4879.g81fedcb3 is installed. 2024-09-16T07:53:52.056 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=81fedcb3b484c843d22ffbc8d179cb567b7083b5 2024-09-16T07:53:52.202 DEBUG:teuthology.orchestra.run.smithi187:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-16T07:53:52.224 INFO:teuthology.orchestra.run.smithi187.stdout:19.3.0-4879.g81fedcb3.el9 2024-09-16T07:53:52.225 INFO:teuthology.packaging:The installed version of ceph is 19.3.0-4879.g81fedcb3.el9 2024-09-16T07:53:52.225 INFO:teuthology.task.install:The correct ceph version 19.3.0-4879.g81fedcb3 is installed. 2024-09-16T07:53:52.226 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-16T07:53:52.227 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-09-16T07:53:52.227 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-16T07:53:52.258 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-09-16T07:53:52.258 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-16T07:53:52.291 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-16T07:53:52.291 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-09-16T07:53:52.291 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd of=/usr/bin/daemon-helper 2024-09-16T07:53:52.329 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-16T07:53:52.399 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-09-16T07:53:52.399 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd of=/usr/bin/daemon-helper 2024-09-16T07:53:52.424 DEBUG:teuthology.orchestra.run.smithi187:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-16T07:53:52.490 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-16T07:53:52.491 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-09-16T07:53:52.491 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-16T07:53:52.521 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-16T07:53:52.590 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-09-16T07:53:52.590 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-16T07:53:52.615 DEBUG:teuthology.orchestra.run.smithi187:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-16T07:53:52.681 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-16T07:53:52.681 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-09-16T07:53:52.681 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd of=/usr/bin/stdin-killer 2024-09-16T07:53:52.711 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-16T07:53:52.780 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-09-16T07:53:52.780 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd of=/usr/bin/stdin-killer 2024-09-16T07:53:52.804 DEBUG:teuthology.orchestra.run.smithi187:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-16T07:53:52.868 INFO:teuthology.run_tasks:Running task ceph... 2024-09-16T07:53:52.954 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-09-16T07:53:52.955 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod 777 /var/log/ceph 2024-09-16T07:53:52.957 DEBUG:teuthology.orchestra.run.smithi187:> sudo chmod 777 /var/log/ceph 2024-09-16T07:53:52.985 INFO:tasks.ceph:Disabling ceph logrotate... 2024-09-16T07:53:52.985 DEBUG:teuthology.orchestra.run.smithi027:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-16T07:53:53.029 DEBUG:teuthology.orchestra.run.smithi187:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-16T07:53:53.055 INFO:tasks.ceph:Creating extra log directories... 2024-09-16T07:53:53.055 DEBUG:teuthology.orchestra.run.smithi027:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-16T07:53:53.100 DEBUG:teuthology.orchestra.run.smithi187:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-16T07:53:53.135 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-09-16T07:53:53.135 INFO:tasks.ceph:config {'conf': {'mon': {'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10, 'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon warn on pool no app': False, 'osd pool default crimson': True}, 'osd': {'debug monc': 20, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug ms': 1, 'debug osd': 20, 'osd blocked scrub grace period': 3600, 'osd class default list': '*', 'osd class load list': '*', 'osd objectstore': 'seastore', 'osd shutdown pgref assert': True}, 'client': {'debug ms': 1}, 'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1, 'mgr stats period': 30}, 'osd.0': {'crimson seastar cpu cores': '0-2'}, 'osd.1': {'crimson seastar cpu cores': '3-5'}, 'osd.2': {'crimson seastar cpu cores': '0-2'}, 'osd.3': {'crimson seastar cpu cores': '3-5'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'reached quota', 'but it is still running', 'overall HEALTH_', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-09-16T07:53:53.136 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/matan-2024-09-16_07:15:55-crimson-rados-wip-matanb-crimson-testing-59543-distro-crimson-smithi/7906886', 'branch': 'wip-matanb-crimson-testing-59543', 'description': 'crimson-rados/basic/{clusters/fixed-2 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/seastore tasks/rados_api_tests}', 'email': None, 'first_in_suite': False, 'job_id': '7906886', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'matan-2024-09-16_07:15:55-crimson-rados-wip-matanb-crimson-testing-59543-distro-crimson-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-matanb-crimson-testing-59543'}, 'ceph': {'conf': {'client': {'debug ms': 1}, 'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1, 'mgr stats period': 30}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon warn on pool no app': False, 'osd pool default crimson': True}, 'osd': {'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug ms': 1, 'debug osd': 20, 'osd blocked scrub grace period': 3600, 'osd class default list': '*', 'osd class load list': '*', 'osd objectstore': 'seastore', 'osd shutdown pgref assert': True}, 'osd.0': {'crimson seastar cpu cores': '0-2'}, 'osd.1': {'crimson seastar cpu cores': '3-5'}, 'osd.2': {'crimson seastar cpu cores': '0-2'}, 'osd.3': {'crimson seastar cpu cores': '3-5'}}, 'flavor': 'crimson', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'reached quota', 'but it is still running', 'overall HEALTH_', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)'], 'sha1': '81fedcb3b484c843d22ffbc8d179cb567b7083b5'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'crimson', 'sha1': '81fedcb3b484c843d22ffbc8d179cb567b7083b5'}}, 'selinux': {'allowlist': ['scontext=system_u:system_r:getty_t:s0']}, 'workunit': {'branch': 'main', 'env': {'CRIMSON_COMPAT': '1'}, 'sha1': '0325fd3c11a225c0fd9eb38fab88aad1370ad779'}}, 'owner': 'scheduled_matan@teuthology', 'priority': 60, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'osd.0', 'osd.1', 'client.0', 'node-exporter.a'], ['mgr.x', 'osd.2', 'osd.3', 'client.1', 'prometheus.a', 'node-exporter.b']], 'seed': 6651, 'sha1': '81fedcb3b484c843d22ffbc8d179cb567b7083b5', 'sleep_before_teardown': 0, 'suite': 'crimson-rados', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_0325fd3c11a225c0fd9eb38fab88aad1370ad779/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': '0325fd3c11a225c0fd9eb38fab88aad1370ad779', 'targets': {'smithi027.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLpcwPaHyzmk8hQe88qHeZTFsVJ0Y/iA/xfFISw1dCCxtKNeHjvgrr2HIKX/kdv3qi12ZBBt00FizK4uPoFcr2k=', 'smithi187.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFvXrMa5rbcalzvURNr4bpBSSg2Hb6A1St9wR0SUv7oRgP1gtuEubBZCiD1S1b5rJOf7CrjqqbFJryMDNPVVifY='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'conf': {'mon': {'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10, 'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon warn on pool no app': False, 'osd pool default crimson': True}, 'osd': {'debug monc': 20, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug ms': 1, 'debug osd': 20, 'osd blocked scrub grace period': 3600, 'osd class default list': '*', 'osd class load list': '*', 'osd objectstore': 'seastore', 'osd shutdown pgref assert': True}, 'client': {'debug ms': 1}, 'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1, 'mgr stats period': 30}, 'osd.0': {'crimson seastar cpu cores': '0-2'}, 'osd.1': {'crimson seastar cpu cores': '3-5'}, 'osd.2': {'crimson seastar cpu cores': '0-2'}, 'osd.3': {'crimson seastar cpu cores': '3-5'}}, 'flavor': 'crimson', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'reached quota', 'but it is still running', 'overall HEALTH_', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)'], 'sha1': '81fedcb3b484c843d22ffbc8d179cb567b7083b5', 'cluster': 'ceph'}}, {'workunit': {'clients': {'client.0': ['rados/test.sh --crimson', 'rados/test_pool_quota.sh']}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a', 'timestamp': '2024-09-16_07:15:55', 'tube': 'smithi', 'user': 'matan', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856'} 2024-09-16T07:53:53.136 DEBUG:teuthology.orchestra.run.smithi027:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-16T07:53:53.168 DEBUG:teuthology.orchestra.run.smithi187:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-16T07:53:53.184 DEBUG:teuthology.orchestra.run.smithi027:> sudo install -d -m0777 -- /var/run/ceph 2024-09-16T07:53:53.226 DEBUG:teuthology.orchestra.run.smithi187:> sudo install -d -m0777 -- /var/run/ceph 2024-09-16T07:53:53.252 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-09-16T07:53:53.253 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/scratch_devs of=/dev/stdout 2024-09-16T07:53:53.311 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-16T07:53:53.311 DEBUG:teuthology.orchestra.run.smithi027:> stat /dev/vg_nvme/lv_1 2024-09-16T07:53:53.370 INFO:teuthology.orchestra.run.smithi027.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-16T07:53:53.370 INFO:teuthology.orchestra.run.smithi027.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T07:53:53.370 INFO:teuthology.orchestra.run.smithi027.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-09-16T07:53:53.370 INFO:teuthology.orchestra.run.smithi027.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T07:53:53.370 INFO:teuthology.orchestra.run.smithi027.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T07:53:53.370 INFO:teuthology.orchestra.run.smithi027.stdout:Access: 2024-09-16 07:53:34.399509015 +0000 2024-09-16T07:53:53.371 INFO:teuthology.orchestra.run.smithi027.stdout:Modify: 2024-09-16 07:50:26.893829069 +0000 2024-09-16T07:53:53.371 INFO:teuthology.orchestra.run.smithi027.stdout:Change: 2024-09-16 07:50:26.893829069 +0000 2024-09-16T07:53:53.371 INFO:teuthology.orchestra.run.smithi027.stdout: Birth: 2024-09-16 07:50:26.893829069 +0000 2024-09-16T07:53:53.371 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-16T07:53:53.438 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records in 2024-09-16T07:53:53.439 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records out 2024-09-16T07:53:53.439 INFO:teuthology.orchestra.run.smithi027.stderr:512 bytes copied, 0.000272268 s, 1.9 MB/s 2024-09-16T07:53:53.440 DEBUG:teuthology.orchestra.run.smithi027:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-16T07:53:53.498 DEBUG:teuthology.orchestra.run.smithi027:> stat /dev/vg_nvme/lv_2 2024-09-16T07:53:53.556 INFO:teuthology.orchestra.run.smithi027.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-16T07:53:53.556 INFO:teuthology.orchestra.run.smithi027.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T07:53:53.557 INFO:teuthology.orchestra.run.smithi027.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-09-16T07:53:53.557 INFO:teuthology.orchestra.run.smithi027.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T07:53:53.557 INFO:teuthology.orchestra.run.smithi027.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T07:53:53.557 INFO:teuthology.orchestra.run.smithi027.stdout:Access: 2024-09-16 07:53:34.400508989 +0000 2024-09-16T07:53:53.557 INFO:teuthology.orchestra.run.smithi027.stdout:Modify: 2024-09-16 07:50:27.233820177 +0000 2024-09-16T07:53:53.558 INFO:teuthology.orchestra.run.smithi027.stdout:Change: 2024-09-16 07:50:27.233820177 +0000 2024-09-16T07:53:53.558 INFO:teuthology.orchestra.run.smithi027.stdout: Birth: 2024-09-16 07:50:27.233820177 +0000 2024-09-16T07:53:53.558 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-16T07:53:53.622 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records in 2024-09-16T07:53:53.622 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records out 2024-09-16T07:53:53.622 INFO:teuthology.orchestra.run.smithi027.stderr:512 bytes copied, 0.000219294 s, 2.3 MB/s 2024-09-16T07:53:53.624 DEBUG:teuthology.orchestra.run.smithi027:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-16T07:53:53.681 DEBUG:teuthology.orchestra.run.smithi027:> stat /dev/vg_nvme/lv_3 2024-09-16T07:53:53.738 INFO:teuthology.orchestra.run.smithi027.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-16T07:53:53.738 INFO:teuthology.orchestra.run.smithi027.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T07:53:53.739 INFO:teuthology.orchestra.run.smithi027.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-09-16T07:53:53.739 INFO:teuthology.orchestra.run.smithi027.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T07:53:53.739 INFO:teuthology.orchestra.run.smithi027.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T07:53:53.739 INFO:teuthology.orchestra.run.smithi027.stdout:Access: 2024-09-16 07:53:34.400508989 +0000 2024-09-16T07:53:53.739 INFO:teuthology.orchestra.run.smithi027.stdout:Modify: 2024-09-16 07:50:27.569811390 +0000 2024-09-16T07:53:53.739 INFO:teuthology.orchestra.run.smithi027.stdout:Change: 2024-09-16 07:50:27.569811390 +0000 2024-09-16T07:53:53.739 INFO:teuthology.orchestra.run.smithi027.stdout: Birth: 2024-09-16 07:50:27.569811390 +0000 2024-09-16T07:53:53.739 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-16T07:53:53.804 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records in 2024-09-16T07:53:53.804 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records out 2024-09-16T07:53:53.804 INFO:teuthology.orchestra.run.smithi027.stderr:512 bytes copied, 0.000213284 s, 2.4 MB/s 2024-09-16T07:53:53.805 DEBUG:teuthology.orchestra.run.smithi027:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-16T07:53:53.863 DEBUG:teuthology.orchestra.run.smithi027:> stat /dev/vg_nvme/lv_4 2024-09-16T07:53:53.920 INFO:teuthology.orchestra.run.smithi027.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-16T07:53:53.920 INFO:teuthology.orchestra.run.smithi027.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T07:53:53.920 INFO:teuthology.orchestra.run.smithi027.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-09-16T07:53:53.921 INFO:teuthology.orchestra.run.smithi027.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T07:53:53.921 INFO:teuthology.orchestra.run.smithi027.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T07:53:53.921 INFO:teuthology.orchestra.run.smithi027.stdout:Access: 2024-09-16 07:53:34.401508963 +0000 2024-09-16T07:53:53.921 INFO:teuthology.orchestra.run.smithi027.stdout:Modify: 2024-09-16 07:50:27.917802289 +0000 2024-09-16T07:53:53.921 INFO:teuthology.orchestra.run.smithi027.stdout:Change: 2024-09-16 07:50:27.917802289 +0000 2024-09-16T07:53:53.921 INFO:teuthology.orchestra.run.smithi027.stdout: Birth: 2024-09-16 07:50:27.917802289 +0000 2024-09-16T07:53:53.921 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-16T07:53:53.986 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records in 2024-09-16T07:53:53.986 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records out 2024-09-16T07:53:53.986 INFO:teuthology.orchestra.run.smithi027.stderr:512 bytes copied, 0.000254722 s, 2.0 MB/s 2024-09-16T07:53:53.987 DEBUG:teuthology.orchestra.run.smithi027:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-16T07:53:54.045 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2'} 2024-09-16T07:53:54.045 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-09-16T07:53:54.045 DEBUG:teuthology.orchestra.run.smithi187:> dd if=/scratch_devs of=/dev/stdout 2024-09-16T07:53:54.063 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-16T07:53:54.064 DEBUG:teuthology.orchestra.run.smithi187:> stat /dev/vg_nvme/lv_1 2024-09-16T07:53:54.122 INFO:teuthology.orchestra.run.smithi187.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-16T07:53:54.123 INFO:teuthology.orchestra.run.smithi187.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T07:53:54.123 INFO:teuthology.orchestra.run.smithi187.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-09-16T07:53:54.123 INFO:teuthology.orchestra.run.smithi187.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T07:53:54.123 INFO:teuthology.orchestra.run.smithi187.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T07:53:54.123 INFO:teuthology.orchestra.run.smithi187.stdout:Access: 2024-09-16 07:53:48.385085697 +0000 2024-09-16T07:53:54.123 INFO:teuthology.orchestra.run.smithi187.stdout:Modify: 2024-09-16 07:50:26.253075844 +0000 2024-09-16T07:53:54.123 INFO:teuthology.orchestra.run.smithi187.stdout:Change: 2024-09-16 07:50:26.253075844 +0000 2024-09-16T07:53:54.124 INFO:teuthology.orchestra.run.smithi187.stdout: Birth: 2024-09-16 07:50:26.253075844 +0000 2024-09-16T07:53:54.124 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-16T07:53:54.189 INFO:teuthology.orchestra.run.smithi187.stderr:1+0 records in 2024-09-16T07:53:54.190 INFO:teuthology.orchestra.run.smithi187.stderr:1+0 records out 2024-09-16T07:53:54.190 INFO:teuthology.orchestra.run.smithi187.stderr:512 bytes copied, 0.000222507 s, 2.3 MB/s 2024-09-16T07:53:54.191 DEBUG:teuthology.orchestra.run.smithi187:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-16T07:53:54.250 DEBUG:teuthology.orchestra.run.smithi187:> stat /dev/vg_nvme/lv_2 2024-09-16T07:53:54.307 INFO:teuthology.orchestra.run.smithi187.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-16T07:53:54.308 INFO:teuthology.orchestra.run.smithi187.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T07:53:54.308 INFO:teuthology.orchestra.run.smithi187.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-09-16T07:53:54.308 INFO:teuthology.orchestra.run.smithi187.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T07:53:54.308 INFO:teuthology.orchestra.run.smithi187.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T07:53:54.308 INFO:teuthology.orchestra.run.smithi187.stdout:Access: 2024-09-16 07:53:48.386085680 +0000 2024-09-16T07:53:54.308 INFO:teuthology.orchestra.run.smithi187.stdout:Modify: 2024-09-16 07:50:26.597070354 +0000 2024-09-16T07:53:54.308 INFO:teuthology.orchestra.run.smithi187.stdout:Change: 2024-09-16 07:50:26.597070354 +0000 2024-09-16T07:53:54.308 INFO:teuthology.orchestra.run.smithi187.stdout: Birth: 2024-09-16 07:50:26.597070354 +0000 2024-09-16T07:53:54.309 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-16T07:53:54.373 INFO:teuthology.orchestra.run.smithi187.stderr:1+0 records in 2024-09-16T07:53:54.373 INFO:teuthology.orchestra.run.smithi187.stderr:1+0 records out 2024-09-16T07:53:54.373 INFO:teuthology.orchestra.run.smithi187.stderr:512 bytes copied, 0.000212498 s, 2.4 MB/s 2024-09-16T07:53:54.375 DEBUG:teuthology.orchestra.run.smithi187:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-16T07:53:54.432 DEBUG:teuthology.orchestra.run.smithi187:> stat /dev/vg_nvme/lv_3 2024-09-16T07:53:54.488 INFO:teuthology.orchestra.run.smithi187.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-16T07:53:54.488 INFO:teuthology.orchestra.run.smithi187.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T07:53:54.488 INFO:teuthology.orchestra.run.smithi187.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-09-16T07:53:54.489 INFO:teuthology.orchestra.run.smithi187.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T07:53:54.489 INFO:teuthology.orchestra.run.smithi187.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T07:53:54.489 INFO:teuthology.orchestra.run.smithi187.stdout:Access: 2024-09-16 07:53:48.386085680 +0000 2024-09-16T07:53:54.489 INFO:teuthology.orchestra.run.smithi187.stdout:Modify: 2024-09-16 07:50:26.920065200 +0000 2024-09-16T07:53:54.489 INFO:teuthology.orchestra.run.smithi187.stdout:Change: 2024-09-16 07:50:26.920065200 +0000 2024-09-16T07:53:54.489 INFO:teuthology.orchestra.run.smithi187.stdout: Birth: 2024-09-16 07:50:26.920065200 +0000 2024-09-16T07:53:54.489 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-16T07:53:54.553 INFO:teuthology.orchestra.run.smithi187.stderr:1+0 records in 2024-09-16T07:53:54.554 INFO:teuthology.orchestra.run.smithi187.stderr:1+0 records out 2024-09-16T07:53:54.554 INFO:teuthology.orchestra.run.smithi187.stderr:512 bytes copied, 0.000203102 s, 2.5 MB/s 2024-09-16T07:53:54.556 DEBUG:teuthology.orchestra.run.smithi187:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-16T07:53:54.614 DEBUG:teuthology.orchestra.run.smithi187:> stat /dev/vg_nvme/lv_4 2024-09-16T07:53:54.670 INFO:teuthology.orchestra.run.smithi187.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-16T07:53:54.670 INFO:teuthology.orchestra.run.smithi187.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T07:53:54.670 INFO:teuthology.orchestra.run.smithi187.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-09-16T07:53:54.670 INFO:teuthology.orchestra.run.smithi187.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T07:53:54.670 INFO:teuthology.orchestra.run.smithi187.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T07:53:54.670 INFO:teuthology.orchestra.run.smithi187.stdout:Access: 2024-09-16 07:53:48.386085680 +0000 2024-09-16T07:53:54.671 INFO:teuthology.orchestra.run.smithi187.stdout:Modify: 2024-09-16 07:50:27.264059711 +0000 2024-09-16T07:53:54.671 INFO:teuthology.orchestra.run.smithi187.stdout:Change: 2024-09-16 07:50:27.264059711 +0000 2024-09-16T07:53:54.671 INFO:teuthology.orchestra.run.smithi187.stdout: Birth: 2024-09-16 07:50:27.264059711 +0000 2024-09-16T07:53:54.671 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-16T07:53:54.734 INFO:teuthology.orchestra.run.smithi187.stderr:1+0 records in 2024-09-16T07:53:54.734 INFO:teuthology.orchestra.run.smithi187.stderr:1+0 records out 2024-09-16T07:53:54.734 INFO:teuthology.orchestra.run.smithi187.stderr:512 bytes copied, 0.000189117 s, 2.7 MB/s 2024-09-16T07:53:54.735 DEBUG:teuthology.orchestra.run.smithi187:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-16T07:53:54.792 INFO:tasks.ceph:osd dev map: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.3': '/dev/vg_nvme/lv_2'} 2024-09-16T07:53:54.792 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi027.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2'}, Remote(name='ubuntu@smithi187.front.sepia.ceph.com'): {'osd.2': '/dev/vg_nvme/lv_1', 'osd.3': '/dev/vg_nvme/lv_2'}} 2024-09-16T07:53:54.792 INFO:tasks.ceph:Generating config... 2024-09-16T07:53:54.793 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2024-09-16T07:53:54.794 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-09-16T07:53:54.794 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-09-16T07:53:54.794 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-09-16T07:53:54.794 INFO:tasks.ceph:[mon] paxos service trim min = 10 2024-09-16T07:53:54.794 INFO:tasks.ceph:[mon] debug mon = 20 2024-09-16T07:53:54.794 INFO:tasks.ceph:[mon] debug ms = 1 2024-09-16T07:53:54.794 INFO:tasks.ceph:[mon] debug paxos = 20 2024-09-16T07:53:54.794 INFO:tasks.ceph:[mon] mon warn on pool no app = False 2024-09-16T07:53:54.794 INFO:tasks.ceph:[mon] osd pool default crimson = True 2024-09-16T07:53:54.794 INFO:tasks.ceph:[osd] debug monc = 20 2024-09-16T07:53:54.795 INFO:tasks.ceph:[osd] crimson alien thread cpu cores = 6-7 2024-09-16T07:53:54.795 INFO:tasks.ceph:[osd] crimson osd obc lru size = 10 2024-09-16T07:53:54.795 INFO:tasks.ceph:[osd] debug ms = 1 2024-09-16T07:53:54.795 INFO:tasks.ceph:[osd] debug osd = 20 2024-09-16T07:53:54.795 INFO:tasks.ceph:[osd] osd blocked scrub grace period = 3600 2024-09-16T07:53:54.795 INFO:tasks.ceph:[osd] osd class default list = * 2024-09-16T07:53:54.795 INFO:tasks.ceph:[osd] osd class load list = * 2024-09-16T07:53:54.795 INFO:tasks.ceph:[osd] osd objectstore = seastore 2024-09-16T07:53:54.795 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-09-16T07:53:54.795 INFO:tasks.ceph:[client] debug ms = 1 2024-09-16T07:53:54.796 INFO:tasks.ceph:[global] enable experimental unrecoverable data corrupting features = crimson 2024-09-16T07:53:54.796 INFO:tasks.ceph:[global] ms client mode = crc 2024-09-16T07:53:54.796 INFO:tasks.ceph:[global] ms cluster mode = crc 2024-09-16T07:53:54.796 INFO:tasks.ceph:[global] ms mon client mode = crc 2024-09-16T07:53:54.796 INFO:tasks.ceph:[global] ms mon cluster mode = crc 2024-09-16T07:53:54.796 INFO:tasks.ceph:[global] ms mon service mode = crc 2024-09-16T07:53:54.796 INFO:tasks.ceph:[global] ms service mode = crc 2024-09-16T07:53:54.796 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-09-16T07:53:54.796 INFO:tasks.ceph:[mgr] debug ms = 1 2024-09-16T07:53:54.796 INFO:tasks.ceph:[mgr] mgr stats period = 30 2024-09-16T07:53:54.796 INFO:tasks.ceph:[osd.0] crimson seastar cpu cores = 0-2 2024-09-16T07:53:54.796 INFO:tasks.ceph:[osd.1] crimson seastar cpu cores = 3-5 2024-09-16T07:53:54.797 INFO:tasks.ceph:[osd.2] crimson seastar cpu cores = 0-2 2024-09-16T07:53:54.797 INFO:tasks.ceph:[osd.3] crimson seastar cpu cores = 3-5 2024-09-16T07:53:54.797 INFO:tasks.ceph:Setting up mon.a... 2024-09-16T07:53:54.797 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-09-16T07:53:54.874 INFO:teuthology.orchestra.run.smithi027.stdout:creating /etc/ceph/ceph.keyring 2024-09-16T07:53:54.878 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-09-16T07:53:54.936 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-16T07:53:54.963 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.27')] 2024-09-16T07:53:54.963 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.27', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug monc': 20, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug ms': 1, 'debug osd': 20, 'osd blocked scrub grace period': 3600, 'osd class default list': '*', 'osd class load list': '*', 'osd objectstore': 'seastore', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false', 'mgr stats period': 30}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10, 'mon warn on pool no app': False, 'osd pool default crimson': True}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug ms': 1}, 'mon.a': {}, 'osd.0': {'crimson seastar cpu cores': '0-2'}, 'osd.1': {'crimson seastar cpu cores': '3-5'}, 'osd.2': {'crimson seastar cpu cores': '0-2'}, 'osd.3': {'crimson seastar cpu cores': '3-5'}} 2024-09-16T07:53:54.965 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-09-16T07:53:54.965 DEBUG:teuthology.orchestra.run.smithi027:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-09-16T07:53:55.019 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.27 --print /home/ubuntu/cephtest/ceph.monmap 2024-09-16T07:53:55.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:53:55.113+0000 7f00bbcfdf00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T07:53:55.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:53:55.113+0000 7f00bbcfdf00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T07:53:55.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:53:55.114+0000 7f00bbcfdf00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T07:53:55.114 INFO:teuthology.orchestra.run.smithi027.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-09-16T07:53:55.114 INFO:teuthology.orchestra.run.smithi027.stdout:monmaptool: generated fsid 16b22dd1-2c19-4258-b5fa-641ce257086f 2024-09-16T07:53:55.114 INFO:teuthology.orchestra.run.smithi027.stdout:setting min_mon_release = quincy 2024-09-16T07:53:55.114 INFO:teuthology.orchestra.run.smithi027.stdout:epoch 0 2024-09-16T07:53:55.114 INFO:teuthology.orchestra.run.smithi027.stdout:fsid 16b22dd1-2c19-4258-b5fa-641ce257086f 2024-09-16T07:53:55.115 INFO:teuthology.orchestra.run.smithi027.stdout:last_changed 2024-09-16T07:53:55.115759+0000 2024-09-16T07:53:55.115 INFO:teuthology.orchestra.run.smithi027.stdout:created 2024-09-16T07:53:55.115759+0000 2024-09-16T07:53:55.115 INFO:teuthology.orchestra.run.smithi027.stdout:min_mon_release 17 (quincy) 2024-09-16T07:53:55.115 INFO:teuthology.orchestra.run.smithi027.stdout:election_strategy: 1 2024-09-16T07:53:55.115 INFO:teuthology.orchestra.run.smithi027.stdout:0: [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] mon.a 2024-09-16T07:53:55.115 INFO:teuthology.orchestra.run.smithi027.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (1 monitors) 2024-09-16T07:53:55.116 DEBUG:teuthology.orchestra.run.smithi027:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-09-16T07:53:55.132 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 16b22dd1-2c19-4258-b5fa-641ce257086f... 2024-09-16T07:53:55.133 DEBUG:teuthology.orchestra.run.smithi027:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-16T07:53:55.174 DEBUG:teuthology.orchestra.run.smithi187:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-16T07:53:55.217 INFO:teuthology.orchestra.run.smithi027.stdout:[global] 2024-09-16T07:53:55.217 INFO:teuthology.orchestra.run.smithi027.stdout: chdir = "" 2024-09-16T07:53:55.217 INFO:teuthology.orchestra.run.smithi027.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-16T07:53:55.217 INFO:teuthology.orchestra.run.smithi027.stdout: auth supported = cephx 2024-09-16T07:53:55.217 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-16T07:53:55.217 INFO:teuthology.orchestra.run.smithi027.stdout: filestore xattr use omap = true 2024-09-16T07:53:55.217 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-16T07:53:55.217 INFO:teuthology.orchestra.run.smithi027.stdout: mon clock drift allowed = 1.000 2024-09-16T07:53:55.217 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-16T07:53:55.217 INFO:teuthology.orchestra.run.smithi027.stdout: osd crush chooseleaf type = 0 2024-09-16T07:53:55.217 INFO:teuthology.orchestra.run.smithi027.stdout: auth debug = true 2024-09-16T07:53:55.217 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-16T07:53:55.218 INFO:teuthology.orchestra.run.smithi027.stdout: ms die on old message = true 2024-09-16T07:53:55.218 INFO:teuthology.orchestra.run.smithi027.stdout: ms die on bug = true 2024-09-16T07:53:55.218 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-16T07:53:55.218 INFO:teuthology.orchestra.run.smithi027.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-16T07:53:55.218 INFO:teuthology.orchestra.run.smithi027.stdout: mon pg warn max object skew = 0 2024-09-16T07:53:55.218 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-16T07:53:55.218 INFO:teuthology.orchestra.run.smithi027.stdout: # disable pg_autoscaler by default for new pools 2024-09-16T07:53:55.218 INFO:teuthology.orchestra.run.smithi027.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-16T07:53:55.218 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-16T07:53:55.218 INFO:teuthology.orchestra.run.smithi027.stdout: osd pool default size = 2 2024-09-16T07:53:55.218 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-16T07:53:55.218 INFO:teuthology.orchestra.run.smithi027.stdout: mon osd allow primary affinity = true 2024-09-16T07:53:55.218 INFO:teuthology.orchestra.run.smithi027.stdout: mon osd allow pg remap = true 2024-09-16T07:53:55.219 INFO:teuthology.orchestra.run.smithi027.stdout: mon warn on legacy crush tunables = false 2024-09-16T07:53:55.219 INFO:teuthology.orchestra.run.smithi027.stdout: mon warn on crush straw calc version zero = false 2024-09-16T07:53:55.219 INFO:teuthology.orchestra.run.smithi027.stdout: mon warn on no sortbitwise = false 2024-09-16T07:53:55.219 INFO:teuthology.orchestra.run.smithi027.stdout: mon warn on osd down out interval zero = false 2024-09-16T07:53:55.219 INFO:teuthology.orchestra.run.smithi027.stdout: mon warn on too few osds = false 2024-09-16T07:53:55.219 INFO:teuthology.orchestra.run.smithi027.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-16T07:53:55.219 INFO:teuthology.orchestra.run.smithi027.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-16T07:53:55.219 INFO:teuthology.orchestra.run.smithi027.stdout: mon_allow_pool_size_one = true 2024-09-16T07:53:55.219 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-16T07:53:55.219 INFO:teuthology.orchestra.run.smithi027.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-16T07:53:55.219 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-16T07:53:55.219 INFO:teuthology.orchestra.run.smithi027.stdout: osd default data pool replay window = 5 2024-09-16T07:53:55.219 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-16T07:53:55.220 INFO:teuthology.orchestra.run.smithi027.stdout: mon allow pool delete = true 2024-09-16T07:53:55.220 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-16T07:53:55.220 INFO:teuthology.orchestra.run.smithi027.stdout: mon cluster log file level = debug 2024-09-16T07:53:55.220 INFO:teuthology.orchestra.run.smithi027.stdout: debug asserts on shutdown = true 2024-09-16T07:53:55.220 INFO:teuthology.orchestra.run.smithi027.stdout: mon health detail to clog = false 2024-09-16T07:53:55.220 INFO:teuthology.orchestra.run.smithi027.stdout: mon host = 172.21.15.27 2024-09-16T07:53:55.220 INFO:teuthology.orchestra.run.smithi027.stdout: enable experimental unrecoverable data corrupting features = crimson 2024-09-16T07:53:55.220 INFO:teuthology.orchestra.run.smithi027.stdout: ms client mode = crc 2024-09-16T07:53:55.220 INFO:teuthology.orchestra.run.smithi027.stdout: ms cluster mode = crc 2024-09-16T07:53:55.220 INFO:teuthology.orchestra.run.smithi027.stdout: ms mon client mode = crc 2024-09-16T07:53:55.220 INFO:teuthology.orchestra.run.smithi027.stdout: ms mon cluster mode = crc 2024-09-16T07:53:55.220 INFO:teuthology.orchestra.run.smithi027.stdout: ms mon service mode = crc 2024-09-16T07:53:55.220 INFO:teuthology.orchestra.run.smithi027.stdout: ms service mode = crc 2024-09-16T07:53:55.220 INFO:teuthology.orchestra.run.smithi027.stdout: fsid = 16b22dd1-2c19-4258-b5fa-641ce257086f 2024-09-16T07:53:55.221 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-16T07:53:55.221 INFO:teuthology.orchestra.run.smithi027.stdout:[osd] 2024-09-16T07:53:55.221 INFO:teuthology.orchestra.run.smithi027.stdout: osd journal size = 100 2024-09-16T07:53:55.221 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-16T07:53:55.221 INFO:teuthology.orchestra.run.smithi027.stdout: osd scrub load threshold = 5.0 2024-09-16T07:53:55.221 INFO:teuthology.orchestra.run.smithi027.stdout: osd scrub max interval = 600 2024-09-16T07:53:55.221 INFO:teuthology.orchestra.run.smithi027.stdout: osd mclock profile = high_recovery_ops 2024-09-16T07:53:55.221 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-16T07:53:55.221 INFO:teuthology.orchestra.run.smithi027.stdout: osd recover clone overlap = true 2024-09-16T07:53:55.221 INFO:teuthology.orchestra.run.smithi027.stdout: osd recovery max chunk = 1048576 2024-09-16T07:53:55.221 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-16T07:53:55.221 INFO:teuthology.orchestra.run.smithi027.stdout: osd debug shutdown = true 2024-09-16T07:53:55.221 INFO:teuthology.orchestra.run.smithi027.stdout: osd debug op order = true 2024-09-16T07:53:55.221 INFO:teuthology.orchestra.run.smithi027.stdout: osd debug verify stray on activate = true 2024-09-16T07:53:55.222 INFO:teuthology.orchestra.run.smithi027.stdout: osd debug trim objects = true 2024-09-16T07:53:55.222 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-16T07:53:55.222 INFO:teuthology.orchestra.run.smithi027.stdout: osd open classes on start = true 2024-09-16T07:53:55.222 INFO:teuthology.orchestra.run.smithi027.stdout: osd debug pg log writeout = true 2024-09-16T07:53:55.222 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-16T07:53:55.222 INFO:teuthology.orchestra.run.smithi027.stdout: osd deep scrub update digest min age = 30 2024-09-16T07:53:55.222 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-16T07:53:55.222 INFO:teuthology.orchestra.run.smithi027.stdout: osd map max advance = 10 2024-09-16T07:53:55.222 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-16T07:53:55.222 INFO:teuthology.orchestra.run.smithi027.stdout: journal zero on create = true 2024-09-16T07:53:55.222 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-16T07:53:55.222 INFO:teuthology.orchestra.run.smithi027.stdout: filestore ondisk finisher threads = 3 2024-09-16T07:53:55.222 INFO:teuthology.orchestra.run.smithi027.stdout: filestore apply finisher threads = 3 2024-09-16T07:53:55.222 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-16T07:53:55.223 INFO:teuthology.orchestra.run.smithi027.stdout: bdev debug aio = true 2024-09-16T07:53:55.223 INFO:teuthology.orchestra.run.smithi027.stdout: osd debug misdirected ops = true 2024-09-16T07:53:55.223 INFO:teuthology.orchestra.run.smithi027.stdout: debug monc = 20 2024-09-16T07:53:55.223 INFO:teuthology.orchestra.run.smithi027.stdout: crimson alien thread cpu cores = 6-7 2024-09-16T07:53:55.223 INFO:teuthology.orchestra.run.smithi027.stdout: crimson osd obc lru size = 10 2024-09-16T07:53:55.223 INFO:teuthology.orchestra.run.smithi027.stdout: debug ms = 1 2024-09-16T07:53:55.223 INFO:teuthology.orchestra.run.smithi027.stdout: debug osd = 20 2024-09-16T07:53:55.223 INFO:teuthology.orchestra.run.smithi027.stdout: osd blocked scrub grace period = 3600 2024-09-16T07:53:55.223 INFO:teuthology.orchestra.run.smithi027.stdout: osd class default list = * 2024-09-16T07:53:55.223 INFO:teuthology.orchestra.run.smithi027.stdout: osd class load list = * 2024-09-16T07:53:55.223 INFO:teuthology.orchestra.run.smithi027.stdout: osd objectstore = seastore 2024-09-16T07:53:55.223 INFO:teuthology.orchestra.run.smithi027.stdout: osd shutdown pgref assert = True 2024-09-16T07:53:55.223 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-16T07:53:55.223 INFO:teuthology.orchestra.run.smithi027.stdout:[mgr] 2024-09-16T07:53:55.224 INFO:teuthology.orchestra.run.smithi027.stdout: debug ms = 1 2024-09-16T07:53:55.224 INFO:teuthology.orchestra.run.smithi027.stdout: debug mgr = 20 2024-09-16T07:53:55.224 INFO:teuthology.orchestra.run.smithi027.stdout: debug mon = 20 2024-09-16T07:53:55.224 INFO:teuthology.orchestra.run.smithi027.stdout: debug auth = 20 2024-09-16T07:53:55.224 INFO:teuthology.orchestra.run.smithi027.stdout: mon reweight min pgs per osd = 4 2024-09-16T07:53:55.224 INFO:teuthology.orchestra.run.smithi027.stdout: mon reweight min bytes per osd = 10 2024-09-16T07:53:55.224 INFO:teuthology.orchestra.run.smithi027.stdout: mgr/telemetry/nag = false 2024-09-16T07:53:55.224 INFO:teuthology.orchestra.run.smithi027.stdout: mgr stats period = 30 2024-09-16T07:53:55.224 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-16T07:53:55.224 INFO:teuthology.orchestra.run.smithi027.stdout:[mon] 2024-09-16T07:53:55.224 INFO:teuthology.orchestra.run.smithi027.stdout: debug ms = 1 2024-09-16T07:53:55.224 INFO:teuthology.orchestra.run.smithi027.stdout: debug mon = 20 2024-09-16T07:53:55.224 INFO:teuthology.orchestra.run.smithi027.stdout: debug paxos = 20 2024-09-16T07:53:55.224 INFO:teuthology.orchestra.run.smithi027.stdout: debug auth = 20 2024-09-16T07:53:55.224 INFO:teuthology.orchestra.run.smithi027.stdout: mon data avail warn = 5 2024-09-16T07:53:55.225 INFO:teuthology.orchestra.run.smithi027.stdout: mon mgr mkfs grace = 240 2024-09-16T07:53:55.225 INFO:teuthology.orchestra.run.smithi027.stdout: mon reweight min pgs per osd = 4 2024-09-16T07:53:55.225 INFO:teuthology.orchestra.run.smithi027.stdout: mon osd reporter subtree level = osd 2024-09-16T07:53:55.225 INFO:teuthology.orchestra.run.smithi027.stdout: mon osd prime pg temp = true 2024-09-16T07:53:55.225 INFO:teuthology.orchestra.run.smithi027.stdout: mon reweight min bytes per osd = 10 2024-09-16T07:53:55.225 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-16T07:53:55.225 INFO:teuthology.orchestra.run.smithi027.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-16T07:53:55.225 INFO:teuthology.orchestra.run.smithi027.stdout: auth mon ticket ttl = 660 # 11m 2024-09-16T07:53:55.225 INFO:teuthology.orchestra.run.smithi027.stdout: auth service ticket ttl = 240 # 4m 2024-09-16T07:53:55.225 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-16T07:53:55.225 INFO:teuthology.orchestra.run.smithi027.stdout: # don't complain about insecure global_id in the test suite 2024-09-16T07:53:55.225 INFO:teuthology.orchestra.run.smithi027.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-16T07:53:55.225 INFO:teuthology.orchestra.run.smithi027.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-16T07:53:55.225 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-16T07:53:55.225 INFO:teuthology.orchestra.run.smithi027.stdout: # 1m isn't quite enough 2024-09-16T07:53:55.226 INFO:teuthology.orchestra.run.smithi027.stdout: mon_down_mkfs_grace = 2m 2024-09-16T07:53:55.226 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-16T07:53:55.226 INFO:teuthology.orchestra.run.smithi027.stdout: mon_warn_on_filestore_osds = false 2024-09-16T07:53:55.226 INFO:teuthology.orchestra.run.smithi027.stdout: mon min osdmap epochs = 50 2024-09-16T07:53:55.226 INFO:teuthology.orchestra.run.smithi027.stdout: mon osdmap full prune interval = 2 2024-09-16T07:53:55.226 INFO:teuthology.orchestra.run.smithi027.stdout: mon osdmap full prune min = 15 2024-09-16T07:53:55.226 INFO:teuthology.orchestra.run.smithi027.stdout: mon osdmap full prune txsize = 2 2024-09-16T07:53:55.226 INFO:teuthology.orchestra.run.smithi027.stdout: paxos service trim min = 10 2024-09-16T07:53:55.226 INFO:teuthology.orchestra.run.smithi027.stdout: mon warn on pool no app = False 2024-09-16T07:53:55.226 INFO:teuthology.orchestra.run.smithi027.stdout: osd pool default crimson = True 2024-09-16T07:53:55.226 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-16T07:53:55.226 INFO:teuthology.orchestra.run.smithi027.stdout:[client] 2024-09-16T07:53:55.226 INFO:teuthology.orchestra.run.smithi027.stdout: rgw cache enabled = true 2024-09-16T07:53:55.226 INFO:teuthology.orchestra.run.smithi027.stdout: rgw enable ops log = true 2024-09-16T07:53:55.227 INFO:teuthology.orchestra.run.smithi027.stdout: rgw enable usage log = true 2024-09-16T07:53:55.227 INFO:teuthology.orchestra.run.smithi027.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-16T07:53:55.227 INFO:teuthology.orchestra.run.smithi027.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-16T07:53:55.227 INFO:teuthology.orchestra.run.smithi027.stdout: debug ms = 1 2024-09-16T07:53:55.227 INFO:teuthology.orchestra.run.smithi027.stdout:[mon.a] 2024-09-16T07:53:55.227 INFO:teuthology.orchestra.run.smithi027.stdout:[osd.0] 2024-09-16T07:53:55.227 INFO:teuthology.orchestra.run.smithi027.stdout: crimson seastar cpu cores = 0-2 2024-09-16T07:53:55.227 INFO:teuthology.orchestra.run.smithi027.stdout:[osd.1] 2024-09-16T07:53:55.227 INFO:teuthology.orchestra.run.smithi027.stdout: crimson seastar cpu cores = 3-5 2024-09-16T07:53:55.227 INFO:teuthology.orchestra.run.smithi027.stdout:[osd.2] 2024-09-16T07:53:55.227 INFO:teuthology.orchestra.run.smithi027.stdout: crimson seastar cpu cores = 0-2 2024-09-16T07:53:55.227 INFO:teuthology.orchestra.run.smithi027.stdout:[osd.3] 2024-09-16T07:53:55.227 INFO:teuthology.orchestra.run.smithi027.stdout: crimson seastar cpu cores = 3-5 2024-09-16T07:53:55.228 INFO:teuthology.orchestra.run.smithi187.stdout:[global] 2024-09-16T07:53:55.228 INFO:teuthology.orchestra.run.smithi187.stdout: chdir = "" 2024-09-16T07:53:55.228 INFO:teuthology.orchestra.run.smithi187.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-16T07:53:55.228 INFO:teuthology.orchestra.run.smithi187.stdout: auth supported = cephx 2024-09-16T07:53:55.228 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-16T07:53:55.228 INFO:teuthology.orchestra.run.smithi187.stdout: filestore xattr use omap = true 2024-09-16T07:53:55.228 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-16T07:53:55.228 INFO:teuthology.orchestra.run.smithi187.stdout: mon clock drift allowed = 1.000 2024-09-16T07:53:55.228 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-16T07:53:55.228 INFO:teuthology.orchestra.run.smithi187.stdout: osd crush chooseleaf type = 0 2024-09-16T07:53:55.228 INFO:teuthology.orchestra.run.smithi187.stdout: auth debug = true 2024-09-16T07:53:55.229 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-16T07:53:55.229 INFO:teuthology.orchestra.run.smithi187.stdout: ms die on old message = true 2024-09-16T07:53:55.229 INFO:teuthology.orchestra.run.smithi187.stdout: ms die on bug = true 2024-09-16T07:53:55.229 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-16T07:53:55.229 INFO:teuthology.orchestra.run.smithi187.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-16T07:53:55.229 INFO:teuthology.orchestra.run.smithi187.stdout: mon pg warn max object skew = 0 2024-09-16T07:53:55.229 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-16T07:53:55.229 INFO:teuthology.orchestra.run.smithi187.stdout: # disable pg_autoscaler by default for new pools 2024-09-16T07:53:55.229 INFO:teuthology.orchestra.run.smithi187.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-16T07:53:55.229 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-16T07:53:55.229 INFO:teuthology.orchestra.run.smithi187.stdout: osd pool default size = 2 2024-09-16T07:53:55.229 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-16T07:53:55.229 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd allow primary affinity = true 2024-09-16T07:53:55.230 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd allow pg remap = true 2024-09-16T07:53:55.230 INFO:teuthology.orchestra.run.smithi187.stdout: mon warn on legacy crush tunables = false 2024-09-16T07:53:55.230 INFO:teuthology.orchestra.run.smithi187.stdout: mon warn on crush straw calc version zero = false 2024-09-16T07:53:55.230 INFO:teuthology.orchestra.run.smithi187.stdout: mon warn on no sortbitwise = false 2024-09-16T07:53:55.230 INFO:teuthology.orchestra.run.smithi187.stdout: mon warn on osd down out interval zero = false 2024-09-16T07:53:55.230 INFO:teuthology.orchestra.run.smithi187.stdout: mon warn on too few osds = false 2024-09-16T07:53:55.230 INFO:teuthology.orchestra.run.smithi187.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-16T07:53:55.230 INFO:teuthology.orchestra.run.smithi187.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-16T07:53:55.230 INFO:teuthology.orchestra.run.smithi187.stdout: mon_allow_pool_size_one = true 2024-09-16T07:53:55.230 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-16T07:53:55.230 INFO:teuthology.orchestra.run.smithi187.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-16T07:53:55.230 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-16T07:53:55.230 INFO:teuthology.orchestra.run.smithi187.stdout: osd default data pool replay window = 5 2024-09-16T07:53:55.230 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-16T07:53:55.231 INFO:teuthology.orchestra.run.smithi187.stdout: mon allow pool delete = true 2024-09-16T07:53:55.231 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-16T07:53:55.231 INFO:teuthology.orchestra.run.smithi187.stdout: mon cluster log file level = debug 2024-09-16T07:53:55.231 INFO:teuthology.orchestra.run.smithi187.stdout: debug asserts on shutdown = true 2024-09-16T07:53:55.231 INFO:teuthology.orchestra.run.smithi187.stdout: mon health detail to clog = false 2024-09-16T07:53:55.231 INFO:teuthology.orchestra.run.smithi187.stdout: mon host = 172.21.15.27 2024-09-16T07:53:55.231 INFO:teuthology.orchestra.run.smithi187.stdout: enable experimental unrecoverable data corrupting features = crimson 2024-09-16T07:53:55.231 INFO:teuthology.orchestra.run.smithi187.stdout: ms client mode = crc 2024-09-16T07:53:55.231 INFO:teuthology.orchestra.run.smithi187.stdout: ms cluster mode = crc 2024-09-16T07:53:55.231 INFO:teuthology.orchestra.run.smithi187.stdout: ms mon client mode = crc 2024-09-16T07:53:55.231 INFO:teuthology.orchestra.run.smithi187.stdout: ms mon cluster mode = crc 2024-09-16T07:53:55.231 INFO:teuthology.orchestra.run.smithi187.stdout: ms mon service mode = crc 2024-09-16T07:53:55.231 INFO:teuthology.orchestra.run.smithi187.stdout: ms service mode = crc 2024-09-16T07:53:55.231 INFO:teuthology.orchestra.run.smithi187.stdout: fsid = 16b22dd1-2c19-4258-b5fa-641ce257086f 2024-09-16T07:53:55.232 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-16T07:53:55.232 INFO:teuthology.orchestra.run.smithi187.stdout:[osd] 2024-09-16T07:53:55.232 INFO:teuthology.orchestra.run.smithi187.stdout: osd journal size = 100 2024-09-16T07:53:55.232 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-16T07:53:55.232 INFO:teuthology.orchestra.run.smithi187.stdout: osd scrub load threshold = 5.0 2024-09-16T07:53:55.232 INFO:teuthology.orchestra.run.smithi187.stdout: osd scrub max interval = 600 2024-09-16T07:53:55.232 INFO:teuthology.orchestra.run.smithi187.stdout: osd mclock profile = high_recovery_ops 2024-09-16T07:53:55.232 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-16T07:53:55.232 INFO:teuthology.orchestra.run.smithi187.stdout: osd recover clone overlap = true 2024-09-16T07:53:55.232 INFO:teuthology.orchestra.run.smithi187.stdout: osd recovery max chunk = 1048576 2024-09-16T07:53:55.232 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-16T07:53:55.232 INFO:teuthology.orchestra.run.smithi187.stdout: osd debug shutdown = true 2024-09-16T07:53:55.232 INFO:teuthology.orchestra.run.smithi187.stdout: osd debug op order = true 2024-09-16T07:53:55.232 INFO:teuthology.orchestra.run.smithi187.stdout: osd debug verify stray on activate = true 2024-09-16T07:53:55.232 INFO:teuthology.orchestra.run.smithi187.stdout: osd debug trim objects = true 2024-09-16T07:53:55.233 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-16T07:53:55.233 INFO:teuthology.orchestra.run.smithi187.stdout: osd open classes on start = true 2024-09-16T07:53:55.233 INFO:teuthology.orchestra.run.smithi187.stdout: osd debug pg log writeout = true 2024-09-16T07:53:55.233 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-16T07:53:55.233 INFO:teuthology.orchestra.run.smithi187.stdout: osd deep scrub update digest min age = 30 2024-09-16T07:53:55.233 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-16T07:53:55.233 INFO:teuthology.orchestra.run.smithi187.stdout: osd map max advance = 10 2024-09-16T07:53:55.233 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-16T07:53:55.233 INFO:teuthology.orchestra.run.smithi187.stdout: journal zero on create = true 2024-09-16T07:53:55.233 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-16T07:53:55.233 INFO:teuthology.orchestra.run.smithi187.stdout: filestore ondisk finisher threads = 3 2024-09-16T07:53:55.233 INFO:teuthology.orchestra.run.smithi187.stdout: filestore apply finisher threads = 3 2024-09-16T07:53:55.233 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-16T07:53:55.233 INFO:teuthology.orchestra.run.smithi187.stdout: bdev debug aio = true 2024-09-16T07:53:55.234 INFO:teuthology.orchestra.run.smithi187.stdout: osd debug misdirected ops = true 2024-09-16T07:53:55.234 INFO:teuthology.orchestra.run.smithi187.stdout: debug monc = 20 2024-09-16T07:53:55.234 INFO:teuthology.orchestra.run.smithi187.stdout: crimson alien thread cpu cores = 6-7 2024-09-16T07:53:55.234 INFO:teuthology.orchestra.run.smithi187.stdout: crimson osd obc lru size = 10 2024-09-16T07:53:55.234 INFO:teuthology.orchestra.run.smithi187.stdout: debug ms = 1 2024-09-16T07:53:55.234 INFO:teuthology.orchestra.run.smithi187.stdout: debug osd = 20 2024-09-16T07:53:55.234 INFO:teuthology.orchestra.run.smithi187.stdout: osd blocked scrub grace period = 3600 2024-09-16T07:53:55.234 INFO:teuthology.orchestra.run.smithi187.stdout: osd class default list = * 2024-09-16T07:53:55.234 INFO:teuthology.orchestra.run.smithi187.stdout: osd class load list = * 2024-09-16T07:53:55.234 INFO:teuthology.orchestra.run.smithi187.stdout: osd objectstore = seastore 2024-09-16T07:53:55.234 INFO:teuthology.orchestra.run.smithi187.stdout: osd shutdown pgref assert = True 2024-09-16T07:53:55.234 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-16T07:53:55.234 INFO:teuthology.orchestra.run.smithi187.stdout:[mgr] 2024-09-16T07:53:55.234 INFO:teuthology.orchestra.run.smithi187.stdout: debug ms = 1 2024-09-16T07:53:55.235 INFO:teuthology.orchestra.run.smithi187.stdout: debug mgr = 20 2024-09-16T07:53:55.235 INFO:teuthology.orchestra.run.smithi187.stdout: debug mon = 20 2024-09-16T07:53:55.235 INFO:teuthology.orchestra.run.smithi187.stdout: debug auth = 20 2024-09-16T07:53:55.235 INFO:teuthology.orchestra.run.smithi187.stdout: mon reweight min pgs per osd = 4 2024-09-16T07:53:55.235 INFO:teuthology.orchestra.run.smithi187.stdout: mon reweight min bytes per osd = 10 2024-09-16T07:53:55.235 INFO:teuthology.orchestra.run.smithi187.stdout: mgr/telemetry/nag = false 2024-09-16T07:53:55.235 INFO:teuthology.orchestra.run.smithi187.stdout: mgr stats period = 30 2024-09-16T07:53:55.235 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-16T07:53:55.235 INFO:teuthology.orchestra.run.smithi187.stdout:[mon] 2024-09-16T07:53:55.235 INFO:teuthology.orchestra.run.smithi187.stdout: debug ms = 1 2024-09-16T07:53:55.235 INFO:teuthology.orchestra.run.smithi187.stdout: debug mon = 20 2024-09-16T07:53:55.235 INFO:teuthology.orchestra.run.smithi187.stdout: debug paxos = 20 2024-09-16T07:53:55.235 INFO:teuthology.orchestra.run.smithi187.stdout: debug auth = 20 2024-09-16T07:53:55.235 INFO:teuthology.orchestra.run.smithi187.stdout: mon data avail warn = 5 2024-09-16T07:53:55.235 INFO:teuthology.orchestra.run.smithi187.stdout: mon mgr mkfs grace = 240 2024-09-16T07:53:55.236 INFO:teuthology.orchestra.run.smithi187.stdout: mon reweight min pgs per osd = 4 2024-09-16T07:53:55.236 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd reporter subtree level = osd 2024-09-16T07:53:55.236 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd prime pg temp = true 2024-09-16T07:53:55.236 INFO:teuthology.orchestra.run.smithi187.stdout: mon reweight min bytes per osd = 10 2024-09-16T07:53:55.236 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-16T07:53:55.236 INFO:teuthology.orchestra.run.smithi187.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-16T07:53:55.236 INFO:teuthology.orchestra.run.smithi187.stdout: auth mon ticket ttl = 660 # 11m 2024-09-16T07:53:55.236 INFO:teuthology.orchestra.run.smithi187.stdout: auth service ticket ttl = 240 # 4m 2024-09-16T07:53:55.236 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-16T07:53:55.236 INFO:teuthology.orchestra.run.smithi187.stdout: # don't complain about insecure global_id in the test suite 2024-09-16T07:53:55.236 INFO:teuthology.orchestra.run.smithi187.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-16T07:53:55.236 INFO:teuthology.orchestra.run.smithi187.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-16T07:53:55.236 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-16T07:53:55.236 INFO:teuthology.orchestra.run.smithi187.stdout: # 1m isn't quite enough 2024-09-16T07:53:55.236 INFO:teuthology.orchestra.run.smithi187.stdout: mon_down_mkfs_grace = 2m 2024-09-16T07:53:55.237 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-16T07:53:55.237 INFO:teuthology.orchestra.run.smithi187.stdout: mon_warn_on_filestore_osds = false 2024-09-16T07:53:55.237 INFO:teuthology.orchestra.run.smithi187.stdout: mon min osdmap epochs = 50 2024-09-16T07:53:55.237 INFO:teuthology.orchestra.run.smithi187.stdout: mon osdmap full prune interval = 2 2024-09-16T07:53:55.237 INFO:teuthology.orchestra.run.smithi187.stdout: mon osdmap full prune min = 15 2024-09-16T07:53:55.237 INFO:teuthology.orchestra.run.smithi187.stdout: mon osdmap full prune txsize = 2 2024-09-16T07:53:55.237 INFO:teuthology.orchestra.run.smithi187.stdout: paxos service trim min = 10 2024-09-16T07:53:55.237 INFO:teuthology.orchestra.run.smithi187.stdout: mon warn on pool no app = False 2024-09-16T07:53:55.237 INFO:teuthology.orchestra.run.smithi187.stdout: osd pool default crimson = True 2024-09-16T07:53:55.237 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-16T07:53:55.237 INFO:teuthology.orchestra.run.smithi187.stdout:[client] 2024-09-16T07:53:55.237 INFO:teuthology.orchestra.run.smithi187.stdout: rgw cache enabled = true 2024-09-16T07:53:55.237 INFO:teuthology.orchestra.run.smithi187.stdout: rgw enable ops log = true 2024-09-16T07:53:55.237 INFO:teuthology.orchestra.run.smithi187.stdout: rgw enable usage log = true 2024-09-16T07:53:55.238 INFO:teuthology.orchestra.run.smithi187.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-16T07:53:55.238 INFO:teuthology.orchestra.run.smithi187.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-16T07:53:55.238 INFO:teuthology.orchestra.run.smithi187.stdout: debug ms = 1 2024-09-16T07:53:55.238 INFO:teuthology.orchestra.run.smithi187.stdout:[mon.a] 2024-09-16T07:53:55.238 INFO:teuthology.orchestra.run.smithi187.stdout:[osd.0] 2024-09-16T07:53:55.238 INFO:teuthology.orchestra.run.smithi187.stdout: crimson seastar cpu cores = 0-2 2024-09-16T07:53:55.238 INFO:teuthology.orchestra.run.smithi187.stdout:[osd.1] 2024-09-16T07:53:55.238 INFO:teuthology.orchestra.run.smithi187.stdout: crimson seastar cpu cores = 3-5 2024-09-16T07:53:55.238 INFO:teuthology.orchestra.run.smithi187.stdout:[osd.2] 2024-09-16T07:53:55.238 INFO:teuthology.orchestra.run.smithi187.stdout: crimson seastar cpu cores = 0-2 2024-09-16T07:53:55.238 INFO:teuthology.orchestra.run.smithi187.stdout:[osd.3] 2024-09-16T07:53:55.238 INFO:teuthology.orchestra.run.smithi187.stdout: crimson seastar cpu cores = 3-5 2024-09-16T07:53:55.240 INFO:tasks.ceph:Creating admin key on mon.a... 2024-09-16T07:53:55.240 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-09-16T07:53:55.347 INFO:tasks.ceph:Copying monmap to all nodes... 2024-09-16T07:53:55.348 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-09-16T07:53:55.348 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-09-16T07:53:55.363 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-09-16T07:53:55.364 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-09-16T07:53:55.419 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi027.front.sepia.ceph.com 2024-09-16T07:53:55.420 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-09-16T07:53:55.420 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-16T07:53:55.420 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-16T07:53:55.495 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-09-16T07:53:55.495 DEBUG:teuthology.orchestra.run.smithi027:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-16T07:53:55.552 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi187.front.sepia.ceph.com 2024-09-16T07:53:55.552 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-09-16T07:53:55.552 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-16T07:53:55.552 DEBUG:teuthology.orchestra.run.smithi187:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-16T07:53:55.589 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-09-16T07:53:55.589 DEBUG:teuthology.orchestra.run.smithi187:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-16T07:53:55.645 INFO:tasks.ceph:Setting up mon nodes... 2024-09-16T07:53:55.646 INFO:tasks.ceph:Setting up mgr nodes... 2024-09-16T07:53:55.646 DEBUG:teuthology.orchestra.run.smithi187:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-09-16T07:53:55.756 INFO:teuthology.orchestra.run.smithi187.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-09-16T07:53:55.759 INFO:tasks.ceph:Setting up mds nodes... 2024-09-16T07:53:55.759 INFO:tasks.ceph_client:Setting up client nodes... 2024-09-16T07:53:55.760 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-09-16T07:53:55.811 INFO:teuthology.orchestra.run.smithi027.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-09-16T07:53:55.825 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-09-16T07:53:55.874 INFO:teuthology.orchestra.run.smithi187.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-09-16T07:53:55.887 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-09-16T07:53:55.888 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi027.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2'}, Remote(name='ubuntu@smithi187.front.sepia.ceph.com'): {'osd.2': '/dev/vg_nvme/lv_1', 'osd.3': '/dev/vg_nvme/lv_2'}} 2024-09-16T07:53:55.888 DEBUG:teuthology.orchestra.run.smithi027:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-09-16T07:53:55.913 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2'} 2024-09-16T07:53:55.913 INFO:tasks.ceph:role: osd.0 2024-09-16T07:53:55.913 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi027.front.sepia.ceph.com 2024-09-16T07:53:55.913 DEBUG:teuthology.orchestra.run.smithi027:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-16T07:53:55.987 INFO:teuthology.orchestra.run.smithi027.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T07:53:55.987 INFO:teuthology.orchestra.run.smithi027.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T07:53:55.987 INFO:teuthology.orchestra.run.smithi027.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T07:53:55.987 INFO:teuthology.orchestra.run.smithi027.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-16T07:53:55.988 INFO:teuthology.orchestra.run.smithi027.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T07:53:55.988 INFO:teuthology.orchestra.run.smithi027.stdout: = sunit=0 swidth=0 blks 2024-09-16T07:53:55.988 INFO:teuthology.orchestra.run.smithi027.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T07:53:55.988 INFO:teuthology.orchestra.run.smithi027.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-16T07:53:55.988 INFO:teuthology.orchestra.run.smithi027.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T07:53:55.988 INFO:teuthology.orchestra.run.smithi027.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T07:53:56.481 INFO:teuthology.orchestra.run.smithi027.stdout:Discarding blocks...Done. 2024-09-16T07:53:56.483 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi027.front.sepia.ceph.com -o noatime 2024-09-16T07:53:56.484 DEBUG:teuthology.orchestra.run.smithi027:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-09-16T07:53:56.625 DEBUG:teuthology.orchestra.run.smithi027:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-09-16T07:53:56.659 DEBUG:teuthology.orchestra.run.smithi027:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-09-16T07:53:56.726 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2'} 2024-09-16T07:53:56.727 INFO:tasks.ceph:role: osd.1 2024-09-16T07:53:56.727 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi027.front.sepia.ceph.com 2024-09-16T07:53:56.727 DEBUG:teuthology.orchestra.run.smithi027:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-16T07:53:56.802 INFO:teuthology.orchestra.run.smithi027.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T07:53:56.802 INFO:teuthology.orchestra.run.smithi027.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T07:53:56.802 INFO:teuthology.orchestra.run.smithi027.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T07:53:56.802 INFO:teuthology.orchestra.run.smithi027.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-16T07:53:56.802 INFO:teuthology.orchestra.run.smithi027.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T07:53:56.802 INFO:teuthology.orchestra.run.smithi027.stdout: = sunit=0 swidth=0 blks 2024-09-16T07:53:56.802 INFO:teuthology.orchestra.run.smithi027.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T07:53:56.803 INFO:teuthology.orchestra.run.smithi027.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-16T07:53:56.803 INFO:teuthology.orchestra.run.smithi027.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T07:53:56.803 INFO:teuthology.orchestra.run.smithi027.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T07:53:57.246 INFO:teuthology.orchestra.run.smithi027.stdout:Discarding blocks...Done. 2024-09-16T07:53:57.249 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi027.front.sepia.ceph.com -o noatime 2024-09-16T07:53:57.249 DEBUG:teuthology.orchestra.run.smithi027:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-09-16T07:53:57.350 DEBUG:teuthology.orchestra.run.smithi027:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-09-16T07:53:57.381 DEBUG:teuthology.orchestra.run.smithi027:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T07:53:57.719 INFO:teuthology.orchestra.run.smithi027.stderr:WARNING: debug mode. Not for benchmarking or production 2024-09-16T07:53:57.720 INFO:teuthology.orchestra.run.smithi027.stderr:INFO 2024-09-16 07:53:57,722 seastar - Reactor backend: linux-aio 2024-09-16T07:53:57.737 INFO:teuthology.orchestra.run.smithi027.stderr:==27767==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-09-16T07:53:57.863 INFO:teuthology.orchestra.run.smithi027.stderr:INFO 2024-09-16 07:53:57,865 [shard 0:main] osd - bypassing the config fetch due to --no-mon-config 2024-09-16T07:53:57.863 INFO:teuthology.orchestra.run.smithi027.stderr:INFO 2024-09-16 07:53:57,866 [shard 0:main] osd - get_early_config: set --thread-affinity 1 --cpuset 0-2 2024-09-16T07:53:57.864 INFO:teuthology.orchestra.run.smithi027.stderr:DEBUG 2024-09-16 07:53:57,866 [shard 0:main] seastar - reactor::drain 2024-09-16T07:53:58.412 INFO:teuthology.orchestra.run.smithi027.stderr:----------------------------------------------------- 2024-09-16T07:53:58.412 INFO:teuthology.orchestra.run.smithi027.stderr:Suppressions used: 2024-09-16T07:53:58.412 INFO:teuthology.orchestra.run.smithi027.stderr: count bytes template 2024-09-16T07:53:58.413 INFO:teuthology.orchestra.run.smithi027.stderr: 1 8 MallocExtension::Register 2024-09-16T07:53:58.413 INFO:teuthology.orchestra.run.smithi027.stderr:----------------------------------------------------- 2024-09-16T07:53:58.413 INFO:teuthology.orchestra.run.smithi027.stderr: 2024-09-16T07:53:58.475 INFO:teuthology.orchestra.run.smithi027.stderr:WARNING: debug mode. Not for benchmarking or production 2024-09-16T07:53:58.479 INFO:teuthology.orchestra.run.smithi027.stderr:INFO 2024-09-16 07:53:58,482 seastar - Reactor backend: linux-aio 2024-09-16T07:53:58.489 INFO:teuthology.orchestra.run.smithi027.stderr:==27763==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-09-16T07:53:58.554 INFO:teuthology.orchestra.run.smithi027.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbca834f 0xbb80cd9 0xbb82244 0xbb8245a 0xbb825b0 0xbb82a79 0x3e6ef 0x4b82e 0x5fdd56d 0x5fdd8fa 0x6038b24 0x5fb8d06 0x5fb8ee0 0x5fb946c 0x5e00fef 0x5e19231 0x71579d2 0x71691a5 0x716a642 0x716a6c7 0x716a7c9 0x716a8c1 0x716aace 0x716acb4 0x716b003 0x716b3a7 0x716b5d7 0x716b841 0x716bb0e 0x716bd2c 0x716bde4 0x716beaa 0x716bf8b 0x5c63e30 0x5c63f3a 0x5c63ff6 0x5c640b2 0x5c6416e 0x5c642bb 0x5c64446 0x5c64807 0x5c7e60a 0xbb95287 0xbbad105 0xbc5a9af 0xbc5bffd 0xb92a8cd 0xb92b105 0x3805438 0x2958f 0x2963f 0x35ec9b4 2024-09-16T07:53:58.554 INFO:teuthology.orchestra.run.smithi027.stderr:kernel callstack: 2024-09-16T07:53:58.739 INFO:teuthology.orchestra.run.smithi027.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbca834f 0xbb80cd9 0xbb82244 0xbb8245a 0xbb825b0 0xbb82a79 0x3e6ef 0xb9ece4b 0xb9d8a8c 0xb9dc6f8 0xb9c7696 0xb9c7a7a 0x9af88b7 0x9b11b72 0x99a2168 0x9942951 0x927469d 0x932dfed 0x932ef8c 0x932f00f 0x932f094 0x932f196 0x932f28e 0x932f314 0x932f3cd 0x932f583 0x932f6bb 0x932f736 0x932f7bb 0x932f916 0x932fa30 0x932fab5 0x932fb93 0x9552886 0x961b5f3 0x961b770 0x96a9bec 0x96a9f88 0x96aa10d 0x96aa1af 0x96aa2c8 0xbe45609 0xbe45810 0xbe458e6 0xbe45a21 0xbe45e15 0xbe451d8 0xbe452b8 0xbe453cc 0xbe4508f 0x9507f90 0x95ecfd2 0x9324493 0x932466d 0x93247a8 0x9324862 0x932493c 0x9324e88 0x9325160 0x9325215 0x959c95c 0x959cbc1 0x95cadf8 0x96ce5a6 0xbb95287 0xbbad105 0xbc5a9af 0xbc5bffd 0xb92a8cd 0xb92b105 0x3805438 0x2958f 0x2963f 0x35ec9b4 2024-09-16T07:53:58.739 INFO:teuthology.orchestra.run.smithi027.stderr:kernel callstack: 2024-09-16T07:53:58.796 INFO:teuthology.orchestra.run.smithi027.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbca834f 0xbb80cd9 0xbb82244 0xbb8245a 0xbb825b0 0xbb82a79 0x3e6ef 0x10797d 0xc9e1d 0x36084f8 0x3b8c080 0x3b8c278 0xb9e9267 0xb9ea53a 0xb9ea7b4 0xb9ea95c 0xb9ec717 0xb9ec913 0xba317fd 0xb9d8ccf 0xb9dc6f8 0xb9c7696 0xb9c7a7a 0x9afa023 0x9b11b72 0x99a2168 0x9942951 0x927469d 0x932dfed 0x932ef8c 0x932f00f 0x932f094 0x932f196 0x932f28e 0x932f314 0x932f3cd 0x932f583 0x932f6bb 0x932f736 0x932f7bb 0x932f916 0x932fa30 0x932fab5 0x932fb93 0x9552886 0x961b5f3 0x961b770 0x96dc4d1 0xbb95287 0xbbad105 0xbc5a9af 0xbc6cfab 0xbc6d62d 0x7b30c45 0xc602d44 0x5e669 0x89c51 0x10ec7f 2024-09-16T07:53:58.796 INFO:teuthology.orchestra.run.smithi027.stderr:kernel callstack: 2024-09-16T07:53:58.796 INFO:teuthology.orchestra.run.smithi027.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: 0x82dff 0xbca834f 0xbb80cd9 0xbb82244 0xbb8245a 0xbb825b0 0xbb82a79 0x3e6ef 0xb9fdcd9 0xba39123 0xba394ae 0xb9d98a8 0xb9dc6f8 0xb9c7696 0xb9c7a7a 0x9afa023 0x9b11b72 0x99a2168 0x9942951 0x927469d 0x932dfed 0x932ef8c 0x932f00f 0x932f094 0x932f196 0x932f28e 0x932f314 0x932f3cd 0x932f583 0x932f6bb 0x932f736 0x932f7bb 0x932f916 0x932fa30 0x932fab5 0x932fb93 0x9552886 0x961b5f3 0x961b770 0x96dc4d1 0xbb95287 0xbbad105 0xbc5a9af 0xbc6cfab 0xbc6d62d 0x7b30c45 0xc602d44 0x5e669 0x89c51 0x10ec7f 2024-09-16T07:53:58.797 INFO:teuthology.orchestra.run.smithi027.stderr:kernel callstack: 2024-09-16T07:53:58.829 INFO:teuthology.orchestra.run.smithi027.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbca834f 0xbb80cd9 0xbb82244 0xbb8245a 0xbb825b0 0xbb82a79 0x3e6ef 0x44699 0xfe5da 0x5cc3466 0x5cc3640 0x5cc38f7 0x5cc39ec 0xba1cab4 0xb9dc9ce 0xb9c7696 0xb9c7a7a 0x9af88b7 0x9b11b72 0x99a2168 0x9942951 0x927469d 0x9274db2 0x92752ce 0x9275457 0x927569e 0x9275976 0x9275a2b 0x9592ce8 0x9592f4d 0x95caf33 0x96cf51e 0xbb95287 0xbbad105 0xbc5a9af 0xbc5bffd 0xb92a8cd 0xb92b105 0x3805438 0x2958f 0x2963f 0x35ec9b4 2024-09-16T07:53:58.829 INFO:teuthology.orchestra.run.smithi027.stderr:kernel callstack: 2024-09-16T07:53:58.854 INFO:teuthology.orchestra.run.smithi027.stderr:Reactor stalled for 90 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbca834f 0xbb80cd9 0xbb82244 0xbb8245a 0xbb825b0 0xbb82a79 0x3e6ef 0x4205d 0x3ef7b 0xfd531 0xb85348d 0xb853540 0xb853656 0xb853736 0xb8539b6 0xb853a6f 0xb853f61 0xb85436c 0xb85462b 0xba26389 0xba28397 0xba2861f 0xba286cf 0xb9d809d 0xb9dc6f8 0xb9c7696 0xb9c7a7a 0x9ab25f7 0x9afe286 0x9b11b72 0x99a2168 0x9942951 0x927469d 0x932dfed 0x932ef8c 0x932f00f 0x932f094 0x932f196 0x932f28e 0x932f314 0x932f3cd 0x932f583 0x932f6bb 0x932f736 0x932f7bb 0x932f916 0x932fa30 0x932fab5 0x932fb93 0x9552886 0x961b5f3 0x961b770 0x96dc4d1 0xbb95287 0xbbad105 0xbc5a9af 0xbc6cfab 0xbc6d62d 0x7b30c45 0xc602d44 0x5e669 0x89c51 0x10ec7f 2024-09-16T07:53:58.854 INFO:teuthology.orchestra.run.smithi027.stderr:Reactor stalled for 90 ms on shard 2, in scheduling group main. Backtrace: 0x82dff 0xbca834f 0xbb80cd9 0xbb82244 0xbb8245a 0xbb825b0 0xbb82a79 0x3e6ef 0x8b56c 0x10d76c 0x10a21a 0xfd59e 0xb85348d 0xb853540 0xb853656 0xb853736 0xb8539b6 0xb853a6f 0xb853f61 0xb85436c 0xb85462b 0xb9dab79 0xb9c7696 0xb9c7a7a 0x9ab25f7 0x9afe0f2 0x9b11b72 0x99a2168 0x9942951 0x927469d 0x932dfed 0x932ef8c 0x932f00f 0x932f094 0x932f196 0x932f28e 0x932f314 0x932f3cd 0x932f583 0x932f6bb 0x932f736 0x932f7bb 0x932f916 0x932fa30 0x932fab5 0x932fb93 0x9552886 0x961b5f3 0x961b770 0x96dc4d1 0xbb95287 0xbbad105 0xbc5a9af 0xbc6cfab 0xbc6d62d 0x7b30c45 0xc602d44 0x5e669 0x89c51 0x10ec7f 2024-09-16T07:53:58.854 INFO:teuthology.orchestra.run.smithi027.stderr:kernel callstack: 2024-09-16T07:53:58.854 INFO:teuthology.orchestra.run.smithi027.stderr:kernel callstack: 2024-09-16T07:53:58.887 INFO:teuthology.orchestra.run.smithi027.stderr:Reactor stalled for 123 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbca834f 0xbb80cd9 0xbb82244 0xbb8245a 0xbb825b0 0xbb82a79 0x3e6ef 0x119673 0x11d39b 0x41608 0x41bbe 0x41c92 0x41f68 0x4214e 0x3ecea 0xfc9b1 0x386a219 0xb9ce334 0x9843791 0x9af9f5a 0x9b11b72 0x99a2168 0x9942951 0x927469d 0x9274db2 0x92752ce 0x9275457 0x927569e 0x9275976 0x9275a2b 0x9592ce8 0x9592f4d 0x95caf33 0x96cf51e 0xbb95287 0xbbad105 0xbc5a9af 0xbc5bffd 0xb92a8cd 0xb92b105 0x3805438 0x2958f 0x2963f 0x35ec9b4 2024-09-16T07:53:58.887 INFO:teuthology.orchestra.run.smithi027.stderr:kernel callstack: 0xffffffffffffff80 0xffffffffb428f540 0xffffffffb429355f 0xffffffffb4294fb4 0xffffffffb3f9c498 0xffffffffb3f9e012 0xffffffffb3f9e1b8 0xffffffffb3f9e5c2 0xffffffffb3f9ff39 0xffffffffb3fa0b9d 0xffffffffb3f6cd4d 0xffffffffb48cd45f 0xffffffffb4a00130 2024-09-16T07:53:59.009 INFO:teuthology.orchestra.run.smithi027.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbca834f 0xbb80cd9 0xbb82244 0xbb8245a 0xbb825b0 0xbb82a79 0x3e6ef 0x9e33966 0x9ea2e39 0x9ed3631 0x9e3abc4 0x9959157 0x9878562 0x989cc22 0x989cd81 0x98ae61e 0x98ae7ee 0x98aeafc 0x98aee0b 0x98aef6a 0x98aefe5 0x98af22c 0x98af4f0 0x98af580 0x98af624 0x95c34b2 0x95c3717 0x95cbcbc 0x96be366 0xbb95287 0xbbad105 0xbc5a9af 0xbc5bffd 0xb92a8cd 0xb92b105 0x3805438 0x2958f 0x2963f 0x35ec9b4 2024-09-16T07:53:59.009 INFO:teuthology.orchestra.run.smithi027.stderr:kernel callstack: 2024-09-16T07:53:59.100 INFO:teuthology.orchestra.run.smithi027.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbca834f 0xbb80cd9 0xbb82244 0xbb8245a 0xbb825b0 0xbb82a79 0x3e6ef 0x9e33934 0x9ea2e39 0x9ed3631 0x9e3abc4 0x9959157 0x9878562 0x989cc22 0x989cd81 0x98ae61e 0x98ae7ee 0x98aeafc 0x98aee0b 0x98aef6a 0x98aefe5 0x98af22c 0x98af4f0 0x98af580 0x98af624 0x95c34b2 0x95c3717 0x95cbcbc 0x96be366 0xbb95287 0xbbad105 0xbc5a9af 0xbc6cfab 0xbc6d62d 0x7b30c45 0xc602d44 0x5e669 0x89c51 0x10ec7f 2024-09-16T07:53:59.100 INFO:teuthology.orchestra.run.smithi027.stderr:kernel callstack: 2024-09-16T07:53:59.101 INFO:teuthology.orchestra.run.smithi027.stderr:Reactor stalled for 65 ms on shard 2, in scheduling group main. Backtrace: 0x82dff 0xbca834f 0xbb80cd9 0xbb82244 0xbb8245a 0xbb825b0 0xbb82a79 0x3e6ef 0x55427bf 0x9e3397b 0x9ea2e39 0x9ed3631 0x9e3abc4 0x9959157 0x9878562 0x989cc22 0x989cd81 0x98ae61e 0x98ae7ee 0x98aeafc 0x98aee0b 0x98aef6a 0x98aefe5 0x98af22c 0x98af4f0 0x98af580 0x98af624 0x95c34b2 0x95c3717 0x95cbcbc 0x96be366 0xbb95287 0xbbad105 0xbc5a9af 0xbc6cfab 0xbc6d62d 0x7b30c45 0xc602d44 0x5e669 0x89c51 0x10ec7f 2024-09-16T07:53:59.102 INFO:teuthology.orchestra.run.smithi027.stderr:kernel callstack: 2024-09-16T07:53:59.158 INFO:teuthology.orchestra.run.smithi027.stderr:Reactor stalled for 122 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbca834f 0xbb80cd9 0xbb82244 0xbb8245a 0xbb825b0 0xbb82a79 0x3e6ef 0x133d18 0xdd332 0x4ab1c 0x5542dfb 0x9e33940 0x9ea2e39 0x9ed3631 0x9e3abc4 0x9959157 0x9878562 0x989cc22 0x989cd81 0x98ae61e 0x98ae7ee 0x98aeafc 0x98aee0b 0x98aef6a 0x98aefe5 0x98af22c 0x98af4f0 0x98af580 0x98af624 0x95c34b2 0x95c3717 0x95cbcbc 0x96be366 0xbb95287 0xbbad105 0xbc5a9af 0xbc6cfab 0xbc6d62d 0x7b30c45 0xc602d44 0x5e669 0x89c51 0x10ec7f 2024-09-16T07:53:59.158 INFO:teuthology.orchestra.run.smithi027.stderr:kernel callstack: 2024-09-16T07:53:59.159 INFO:teuthology.orchestra.run.smithi027.stderr:Reactor stalled for 122 ms on shard 2, in scheduling group main. Backtrace: 0x82dff 0xbca834f 0xbb80cd9 0xbb82244 0xbb8245a 0xbb825b0 0xbb82a79 0x3e6ef 0x5542de6 0x9e33940 0x9ea2e39 0x9ed3631 0x9e3abc4 0x9959157 0x9878562 0x989cc22 0x989cd81 0x98ae61e 0x98ae7ee 0x98aeafc 0x98aee0b 0x98aef6a 0x98aefe5 0x98af22c 0x98af4f0 0x98af580 0x98af624 0x95c34b2 0x95c3717 0x95cbcbc 0x96be366 0xbb95287 0xbbad105 0xbc5a9af 0xbc6cfab 0xbc6d62d 0x7b30c45 0xc602d44 0x5e669 0x89c51 0x10ec7f 2024-09-16T07:53:59.159 INFO:teuthology.orchestra.run.smithi027.stderr:kernel callstack: 2024-09-16T07:53:59.265 INFO:teuthology.orchestra.run.smithi027.stderr:Reactor stalled for 230 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbca834f 0xbb80cd9 0xbb82244 0xbb8245a 0xbb825b0 0xbb82a79 0x3e6ef 0x156e8 0xdd312 0x4ab1c 0x5542dfb 0x9e33940 0x9ea2e39 0x9ed3631 0x9e3abc4 0x9959157 0x9878562 0x989cc22 0x989cd81 0x98ae61e 0x98ae7ee 0x98aeafc 0x98aee0b 0x98aef6a 0x98aefe5 0x98af22c 0x98af4f0 0x98af580 0x98af624 0x95c34b2 0x95c3717 0x95cbcbc 0x96be366 0xbb95287 0xbbad105 0xbc5a9af 0xbc6cfab 0xbc6d62d 0x7b30c45 0xc602d44 0x5e669 0x89c51 0x10ec7f 2024-09-16T07:53:59.265 INFO:teuthology.orchestra.run.smithi027.stderr:kernel callstack: 2024-09-16T07:53:59.267 INFO:teuthology.orchestra.run.smithi027.stderr:Reactor stalled for 230 ms on shard 2, in scheduling group main. Backtrace: 0x82dff 0xbca834f 0xbb80cd9 0xbb82244 0xbb8245a 0xbb825b0 0xbb82a79 0x3e6ef 0x9e33940 0x9ea2e39 0x9ed3631 0x9e3abc4 0x9959157 0x9878562 0x989cc22 0x989cd81 0x98ae61e 0x98ae7ee 0x98aeafc 0x98aee0b 0x98aef6a 0x98aefe5 0x98af22c 0x98af4f0 0x98af580 0x98af624 0x95c34b2 0x95c3717 0x95cbcbc 0x96be366 0xbb95287 0xbbad105 0xbc5a9af 0xbc6cfab 0xbc6d62d 0x7b30c45 0xc602d44 0x5e669 0x89c51 0x10ec7f 2024-09-16T07:53:59.267 INFO:teuthology.orchestra.run.smithi027.stderr:kernel callstack: 2024-09-16T07:54:00.512 INFO:teuthology.orchestra.run.smithi027.stderr:----------------------------------------------------- 2024-09-16T07:54:00.512 INFO:teuthology.orchestra.run.smithi027.stderr:Suppressions used: 2024-09-16T07:54:00.513 INFO:teuthology.orchestra.run.smithi027.stderr: count bytes template 2024-09-16T07:54:00.513 INFO:teuthology.orchestra.run.smithi027.stderr: 1 8 MallocExtension::Register 2024-09-16T07:54:00.513 INFO:teuthology.orchestra.run.smithi027.stderr:----------------------------------------------------- 2024-09-16T07:54:00.513 INFO:teuthology.orchestra.run.smithi027.stderr: 2024-09-16T07:54:00.540 DEBUG:teuthology.orchestra.run.smithi027:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-16T07:54:00.574 DEBUG:teuthology.orchestra.run.smithi027:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T07:54:00.867 INFO:teuthology.orchestra.run.smithi027.stderr:WARNING: debug mode. Not for benchmarking or production 2024-09-16T07:54:00.868 INFO:teuthology.orchestra.run.smithi027.stderr:INFO 2024-09-16 07:54:00,870 seastar - Reactor backend: linux-aio 2024-09-16T07:54:00.874 INFO:teuthology.orchestra.run.smithi027.stderr:==27815==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-09-16T07:54:00.982 INFO:teuthology.orchestra.run.smithi027.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbca834f 0xbb80cd9 0xbb82244 0xbb8245a 0xbb825b0 0xbb82a79 0x3e6ef 0x5c55b6f 0x5fe9434 0x5fe9a0d 0x6038b5b 0x5fb8d06 0x5fb923a 0x5e00fef 0x5e19231 0x71579d2 0x71691a5 0x716a642 0x716a6c7 0x716a7c9 0x716a8c1 0x716aace 0x716acb4 0x716b003 0x716b3a7 0x716b5d7 0x716b841 0x716bb0e 0x716bd2c 0x716bde4 0x716beaa 0x716bf8b 0x5c63e30 0x5c63f3a 0x5c63ff6 0x5c640b2 0x5c6416e 0x5c642bb 0x5c64446 0x5c64807 0x5c7e60a 0xbb95287 0xbbad105 0xbc5a9af 0xbc5bffd 0xb92a8cd 0xb92b105 0x3962bc0 0x396439a 0x3804c28 0x2958f 0x2963f 0x35ec9b4 2024-09-16T07:54:00.982 INFO:teuthology.orchestra.run.smithi027.stderr:kernel callstack: 2024-09-16T07:54:01.000 INFO:teuthology.orchestra.run.smithi027.stderr:INFO 2024-09-16 07:54:01,002 [shard 0:main] osd - bypassing the config fetch due to --no-mon-config 2024-09-16T07:54:01.000 INFO:teuthology.orchestra.run.smithi027.stderr:INFO 2024-09-16 07:54:01,002 [shard 0:main] osd - get_early_config: set --thread-affinity 1 --cpuset 3-5 2024-09-16T07:54:01.001 INFO:teuthology.orchestra.run.smithi027.stderr:DEBUG 2024-09-16 07:54:01,003 [shard 0:main] seastar - reactor::drain 2024-09-16T07:54:01.531 INFO:teuthology.orchestra.run.smithi027.stderr:----------------------------------------------------- 2024-09-16T07:54:01.531 INFO:teuthology.orchestra.run.smithi027.stderr:Suppressions used: 2024-09-16T07:54:01.531 INFO:teuthology.orchestra.run.smithi027.stderr: count bytes template 2024-09-16T07:54:01.531 INFO:teuthology.orchestra.run.smithi027.stderr: 1 8 MallocExtension::Register 2024-09-16T07:54:01.531 INFO:teuthology.orchestra.run.smithi027.stderr:----------------------------------------------------- 2024-09-16T07:54:01.531 INFO:teuthology.orchestra.run.smithi027.stderr: 2024-09-16T07:54:01.596 INFO:teuthology.orchestra.run.smithi027.stderr:WARNING: debug mode. Not for benchmarking or production 2024-09-16T07:54:01.600 INFO:teuthology.orchestra.run.smithi027.stderr:INFO 2024-09-16 07:54:01,602 seastar - Reactor backend: linux-aio 2024-09-16T07:54:01.609 INFO:teuthology.orchestra.run.smithi027.stderr:==27813==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-09-16T07:54:01.674 INFO:teuthology.orchestra.run.smithi027.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbca834f 0xbb80cd9 0xbb82244 0xbb8245a 0xbb825b0 0xbb82a79 0x3e6ef 0x3614016 0x5fdcdd0 0x5fdd224 0x5fdd7df 0x5fdd905 0x6038b24 0x5fb8d06 0x5fb8ee0 0x5fb946c 0x5e00fef 0x5e19231 0x71579d2 0x71691a5 0x716a642 0x716a6c7 0x716a7c9 0x716a8c1 0x716aace 0x716acb4 0x716b003 0x716b3a7 0x716b5d7 0x716b841 0x716bb0e 0x716bd2c 0x716bde4 0x716beaa 0x716bf8b 0x5c63e30 0x5c63f3a 0x5c63ff6 0x5c640b2 0x5c6416e 0x5c642bb 0x5c64446 0x5c64807 0x5c7e60a 0xbb95287 0xbbad105 0xbc5a9af 0xbc5bffd 0xb92a8cd 0xb92b105 0x3805438 0x2958f 0x2963f 0x35ec9b4 2024-09-16T07:54:01.674 INFO:teuthology.orchestra.run.smithi027.stderr:kernel callstack: 2024-09-16T07:54:01.910 INFO:teuthology.orchestra.run.smithi027.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: 0x82dff 0xbca834f 0xbb80cd9 0xbb82244 0xbb8245a 0xbb825b0 0xbb82a79 0x3e6ef 0xb9ea6d3 0xb9ea95c 0xb9ec717 0xb9ecb62 0xb9d8a8c 0xb9dc6f8 0xb9c7696 0xb9c7a7a 0x9afa023 0x9b11b72 0x99a2168 0x9942951 0x927469d 0x932dfed 0x932ef8c 0x932f00f 0x932f094 0x932f196 0x932f28e 0x932f314 0x932f3cd 0x932f583 0x932f6bb 0x932f736 0x932f7bb 0x932f916 0x932fa30 0x932fab5 0x932fb93 0x9552886 0x961b5f3 0x961b770 0x96dc4d1 0xbb95287 0xbbad105 0xbc5a9af 0xbc6cfab 0xbc6d62d 0x7b30c45 0xc602d44 0x5e669 0x89c51 0x10ec7f 2024-09-16T07:54:01.910 INFO:teuthology.orchestra.run.smithi027.stderr:kernel callstack: 2024-09-16T07:54:01.910 INFO:teuthology.orchestra.run.smithi027.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbca834f 0xbb80cd9 0xbb82244 0xbb8245a 0xbb825b0 0xbb82a79 0x3e6ef 0xba102a0 0xb9ceb2f 0x9843791 0x9af9974 0x9b11b72 0x99a2168 0x9942951 0x927469d 0x932dfed 0x932ef8c 0x932f00f 0x932f094 0x932f196 0x932f28e 0x932f314 0x932f3cd 0x932f583 0x932f6bb 0x932f736 0x932f7bb 0x932f916 0x932fa30 0x932fab5 0x932fb93 0x9552886 0x961b5f3 0x961b770 0x96dc4d1 0xbb95287 0xbbad105 0xbc5a9af 0xbc6cfab 0xbc6d62d 0x7b30c45 0xc602d44 0x5e669 0x89c51 0x10ec7f 2024-09-16T07:54:01.910 INFO:teuthology.orchestra.run.smithi027.stderr:kernel callstack: 2024-09-16T07:54:01.968 INFO:teuthology.orchestra.run.smithi027.stderr:Reactor stalled for 90 ms on shard 2, in scheduling group main. Backtrace: 0x82dff 0xbca834f 0xbb80cd9 0xbb82244 0xbb8245a 0xbb825b0 0xbb82a79 0x3e6ef 0xdd2cf 0x4ab1c 0x3ac7b0e 0x3b8c26d 0xb9e9267 0xb9ea53a 0xb9ea7b4 0xb9ea95c 0xb9ec717 0xb9ecb62 0xb9d8a8c 0xb9dc6f8 0xb9c7696 0xb9c7a7a 0x9afe8a1 0x9b11b72 0x99a2168 0x9942951 0x927469d 0x932dfed 0x932ef8c 0x932f00f 0x932f094 0x932f196 0x932f28e 0x932f314 0x932f3cd 0x932f583 0x932f6bb 0x932f736 0x932f7bb 0x932f916 0x932fa30 0x932fab5 0x932fb93 0x9552886 0x961b5f3 0x961b770 0x96dc4d1 0xbb95287 0xbbad105 0xbc5a9af 0xbc6cfab 0xbc6d62d 0x7b30c45 0xc602d44 0x5e669 0x89c51 0x10ec7f 2024-09-16T07:54:01.968 INFO:teuthology.orchestra.run.smithi027.stderr:kernel callstack: 2024-09-16T07:54:01.968 INFO:teuthology.orchestra.run.smithi027.stderr:Reactor stalled for 90 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbca834f 0xbb80cd9 0xbb82244 0xbb8245a 0xbb825b0 0xbb82a79 0x3e6ef 0x133c1f 0xdd332 0x4ab1c 0x3ac7b0e 0x3b8c26d 0xba3462e 0xb9cef3d 0x9843791 0x9ab1ada 0x9afe286 0x9b11b72 0x99a2168 0x9942951 0x927469d 0x932dfed 0x932ef8c 0x932f00f 0x932f094 0x932f196 0x932f28e 0x932f314 0x932f3cd 0x932f583 0x932f6bb 0x932f736 0x932f7bb 0x932f916 0x932fa30 0x932fab5 0x932fb93 0x9552886 0x961b5f3 0x961b770 0x96dc4d1 0xbb95287 0xbbad105 0xbc5a9af 0xbc6cfab 0xbc6d62d 0x7b30c45 0xc602d44 0x5e669 0x89c51 0x10ec7f 2024-09-16T07:54:01.968 INFO:teuthology.orchestra.run.smithi027.stderr:kernel callstack: 2024-09-16T07:54:02.125 INFO:teuthology.orchestra.run.smithi027.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbca834f 0xbb80cd9 0xbb82244 0xbb8245a 0xbb825b0 0xbb82a79 0x3e6ef 0x8b571 0x10d76c 0xdd31a 0x4ab1c 0x5542dfb 0x9e33940 0x9ea2e39 0x9ed3631 0x9e3abc4 0x9959157 0x9878562 0x989cc22 0x989cd81 0x98ae61e 0x98ae7ee 0x98aeafc 0x98aee0b 0x98aef6a 0x98aefe5 0x98af22c 0x98af4f0 0x98af580 0x98af624 0x95c34b2 0x95c3717 0x95cbcbc 0x96be366 0xbb95287 0xbbad105 0xbc5a9af 0xbc5bffd 0xb92a8cd 0xb92b105 0x3805438 0x2958f 0x2963f 0x35ec9b4 2024-09-16T07:54:02.125 INFO:teuthology.orchestra.run.smithi027.stderr:kernel callstack: 2024-09-16T07:54:02.182 INFO:teuthology.orchestra.run.smithi027.stderr:Reactor stalled for 122 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbca834f 0xbb80cd9 0xbb82244 0xbb8245a 0xbb825b0 0xbb82a79 0x3e6ef 0x5541dd2 0x5542d9c 0x9e33940 0x9ea2e39 0x9ed3631 0x9e3abc4 0x9959157 0x9878562 0x989cc22 0x989cd81 0x98ae61e 0x98ae7ee 0x98aeafc 0x98aee0b 0x98aef6a 0x98aefe5 0x98af22c 0x98af4f0 0x98af580 0x98af624 0x95c34b2 0x95c3717 0x95cbcbc 0x96be366 0xbb95287 0xbbad105 0xbc5a9af 0xbc5bffd 0xb92a8cd 0xb92b105 0x3805438 0x2958f 0x2963f 0x35ec9b4 2024-09-16T07:54:02.182 INFO:teuthology.orchestra.run.smithi027.stderr:kernel callstack: 2024-09-16T07:54:02.236 INFO:teuthology.orchestra.run.smithi027.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbca834f 0xbb80cd9 0xbb82244 0xbb8245a 0xbb825b0 0xbb82a79 0x3e6ef 0x4ab7c 0x5542dfb 0x9e33940 0x9ea2e39 0x9ed3631 0x9e3abc4 0x9959157 0x9878562 0x989cc22 0x989cd81 0x98ae61e 0x98ae7ee 0x98aeafc 0x98aee0b 0x98aef6a 0x98aefe5 0x98af22c 0x98af4f0 0x98af580 0x98af624 0x95c34b2 0x95c3717 0x95cbcbc 0x96be366 0xbb95287 0xbbad105 0xbc5a9af 0xbc6cfab 0xbc6d62d 0x7b30c45 0xc602d44 0x5e669 0x89c51 0x10ec7f 2024-09-16T07:54:02.236 INFO:teuthology.orchestra.run.smithi027.stderr:kernel callstack: 2024-09-16T07:54:02.241 INFO:teuthology.orchestra.run.smithi027.stderr:Reactor stalled for 65 ms on shard 2, in scheduling group main. Backtrace: 0x82dff 0xbca834f 0xbb80cd9 0xbb82244 0xbb8245a 0xbb825b0 0xbb82a79 0x3e6ef 0x5542be7 0x9e33940 0x9ea2e39 0x9ed3631 0x9e3abc4 0x9959157 0x9878562 0x989cc22 0x989cd81 0x98ae61e 0x98ae7ee 0x98aeafc 0x98aee0b 0x98aef6a 0x98aefe5 0x98af22c 0x98af4f0 0x98af580 0x98af624 0x95c34b2 0x95c3717 0x95cbcbc 0x96be366 0xbb95287 0xbbad105 0xbc5a9af 0xbc6cfab 0xbc6d62d 0x7b30c45 0xc602d44 0x5e669 0x89c51 0x10ec7f 2024-09-16T07:54:02.241 INFO:teuthology.orchestra.run.smithi027.stderr:kernel callstack: 2024-09-16T07:54:02.290 INFO:teuthology.orchestra.run.smithi027.stderr:Reactor stalled for 230 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbca834f 0xbb80cd9 0xbb82244 0xbb8245a 0xbb825b0 0xbb82a79 0x3e6ef 0x4ab7c 0x5542dfb 0x9e33940 0x9ea2e39 0x9ed3631 0x9e3abc4 0x9959157 0x9878562 0x989cc22 0x989cd81 0x98ae61e 0x98ae7ee 0x98aeafc 0x98aee0b 0x98aef6a 0x98aefe5 0x98af22c 0x98af4f0 0x98af580 0x98af624 0x95c34b2 0x95c3717 0x95cbcbc 0x96be366 0xbb95287 0xbbad105 0xbc5a9af 0xbc5bffd 0xb92a8cd 0xb92b105 0x3805438 0x2958f 0x2963f 0x35ec9b4 2024-09-16T07:54:02.290 INFO:teuthology.orchestra.run.smithi027.stderr:kernel callstack: 2024-09-16T07:54:02.294 INFO:teuthology.orchestra.run.smithi027.stderr:Reactor stalled for 122 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbca834f 0xbb80cd9 0xbb82244 0xbb8245a 0xbb825b0 0xbb82a79 0x3e6ef 0xdd2e1 0x4ab1c 0x5542dfb 0x9e33940 0x9ea2e39 0x9ed3631 0x9e3abc4 0x9959157 0x9878562 0x989cc22 0x989cd81 0x98ae61e 0x98ae7ee 0x98aeafc 0x98aee0b 0x98aef6a 0x98aefe5 0x98af22c 0x98af4f0 0x98af580 0x98af624 0x95c34b2 0x95c3717 0x95cbcbc 0x96be366 0xbb95287 0xbbad105 0xbc5a9af 0xbc6cfab 0xbc6d62d 0x7b30c45 0xc602d44 0x5e669 0x89c51 0x10ec7f 2024-09-16T07:54:02.294 INFO:teuthology.orchestra.run.smithi027.stderr:kernel callstack: 2024-09-16T07:54:02.296 INFO:teuthology.orchestra.run.smithi027.stderr:Reactor stalled for 122 ms on shard 2, in scheduling group main. Backtrace: 0x82dff 0xbca834f 0xbb80cd9 0xbb82244 0xbb8245a 0xbb825b0 0xbb82a79 0x3e6ef 0x133cc6 0xdd332 0x4ab1c 0x5542dfb 0x9e33940 0x9ea2e39 0x9ed3631 0x9e3abc4 0x9959157 0x9878562 0x989cc22 0x989cd81 0x98ae61e 0x98ae7ee 0x98aeafc 0x98aee0b 0x98aef6a 0x98aefe5 0x98af22c 0x98af4f0 0x98af580 0x98af624 0x95c34b2 0x95c3717 0x95cbcbc 0x96be366 0xbb95287 0xbbad105 0xbc5a9af 0xbc6cfab 0xbc6d62d 0x7b30c45 0xc602d44 0x5e669 0x89c51 0x10ec7f 2024-09-16T07:54:02.296 INFO:teuthology.orchestra.run.smithi027.stderr:kernel callstack: 2024-09-16T07:54:02.401 INFO:teuthology.orchestra.run.smithi027.stderr:Reactor stalled for 230 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbca834f 0xbb80cd9 0xbb82244 0xbb8245a 0xbb825b0 0xbb82a79 0x3e6ef 0x4ab7c 0x5542dfb 0x9e33940 0x9ea2e39 0x9ed3631 0x9e3abc4 0x9959157 0x9878562 0x989cc22 0x989cd81 0x98ae61e 0x98ae7ee 0x98aeafc 0x98aee0b 0x98aef6a 0x98aefe5 0x98af22c 0x98af4f0 0x98af580 0x98af624 0x95c34b2 0x95c3717 0x95cbcbc 0x96be366 0xbb95287 0xbbad105 0xbc5a9af 0xbc6cfab 0xbc6d62d 0x7b30c45 0xc602d44 0x5e669 0x89c51 0x10ec7f 2024-09-16T07:54:02.401 INFO:teuthology.orchestra.run.smithi027.stderr:kernel callstack: 2024-09-16T07:54:02.404 INFO:teuthology.orchestra.run.smithi027.stderr:Reactor stalled for 230 ms on shard 2, in scheduling group main. Backtrace: 0x82dff 0xbca834f 0xbb80cd9 0xbb82244 0xbb8245a 0xbb825b0 0xbb82a79 0x3e6ef 0x5541fff 0x9e33959 0x9ea2e39 0x9ed3631 0x9e3abc4 0x9959157 0x9878562 0x989cc22 0x989cd81 0x98ae61e 0x98ae7ee 0x98aeafc 0x98aee0b 0x98aef6a 0x98aefe5 0x98af22c 0x98af4f0 0x98af580 0x98af624 0x95c34b2 0x95c3717 0x95cbcbc 0x96be366 0xbb95287 0xbbad105 0xbc5a9af 0xbc6cfab 0xbc6d62d 0x7b30c45 0xc602d44 0x5e669 0x89c51 0x10ec7f 2024-09-16T07:54:02.404 INFO:teuthology.orchestra.run.smithi027.stderr:kernel callstack: 2024-09-16T07:54:02.700 INFO:teuthology.orchestra.run.smithi027.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbca834f 0xbb80cd9 0xbb82244 0xbb8245a 0xbb825b0 0xbb82a79 0x3e6ef 0x5542113 0x9e33959 0x9ea2e39 0x9ed3631 0x9e3abc4 0x9959157 0x9878562 0x989cc22 0x989cd81 0x98ae61e 0x98ae7ee 0x98aeafc 0x98aee0b 0x98aef6a 0x98aefe5 0x98af22c 0x98af4f0 0x98af580 0x98af624 0x95c34b2 0x95c3717 0x95cbcbc 0x96be366 0xbb95287 0xbbad105 0xbc5a9af 0xbc5bffd 0xb92a8cd 0xb92b105 0x3805438 0x2958f 0x2963f 0x35ec9b4 2024-09-16T07:54:02.701 INFO:teuthology.orchestra.run.smithi027.stderr:kernel callstack: 2024-09-16T07:54:03.666 INFO:teuthology.orchestra.run.smithi027.stderr:----------------------------------------------------- 2024-09-16T07:54:03.666 INFO:teuthology.orchestra.run.smithi027.stderr:Suppressions used: 2024-09-16T07:54:03.666 INFO:teuthology.orchestra.run.smithi027.stderr: count bytes template 2024-09-16T07:54:03.666 INFO:teuthology.orchestra.run.smithi027.stderr: 1 8 MallocExtension::Register 2024-09-16T07:54:03.666 INFO:teuthology.orchestra.run.smithi027.stderr:----------------------------------------------------- 2024-09-16T07:54:03.666 INFO:teuthology.orchestra.run.smithi027.stderr: 2024-09-16T07:54:03.693 DEBUG:teuthology.orchestra.run.smithi027:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-16T07:54:03.719 DEBUG:teuthology.orchestra.run.smithi187:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-09-16T07:54:03.750 INFO:tasks.ceph:roles_to_devs: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.3': '/dev/vg_nvme/lv_2'} 2024-09-16T07:54:03.750 INFO:tasks.ceph:role: osd.2 2024-09-16T07:54:03.750 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi187.front.sepia.ceph.com 2024-09-16T07:54:03.750 DEBUG:teuthology.orchestra.run.smithi187:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-16T07:54:03.825 INFO:teuthology.orchestra.run.smithi187.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T07:54:03.826 INFO:teuthology.orchestra.run.smithi187.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T07:54:03.826 INFO:teuthology.orchestra.run.smithi187.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T07:54:03.826 INFO:teuthology.orchestra.run.smithi187.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-16T07:54:03.826 INFO:teuthology.orchestra.run.smithi187.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T07:54:03.826 INFO:teuthology.orchestra.run.smithi187.stdout: = sunit=0 swidth=0 blks 2024-09-16T07:54:03.826 INFO:teuthology.orchestra.run.smithi187.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T07:54:03.826 INFO:teuthology.orchestra.run.smithi187.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-16T07:54:03.826 INFO:teuthology.orchestra.run.smithi187.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T07:54:03.826 INFO:teuthology.orchestra.run.smithi187.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T07:54:04.340 INFO:teuthology.orchestra.run.smithi187.stdout:Discarding blocks...Done. 2024-09-16T07:54:04.342 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi187.front.sepia.ceph.com -o noatime 2024-09-16T07:54:04.342 DEBUG:teuthology.orchestra.run.smithi187:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-2 2024-09-16T07:54:04.492 DEBUG:teuthology.orchestra.run.smithi187:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-09-16T07:54:04.526 DEBUG:teuthology.orchestra.run.smithi187:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-09-16T07:54:04.594 INFO:tasks.ceph:roles_to_devs: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.3': '/dev/vg_nvme/lv_2'} 2024-09-16T07:54:04.594 INFO:tasks.ceph:role: osd.3 2024-09-16T07:54:04.594 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi187.front.sepia.ceph.com 2024-09-16T07:54:04.595 DEBUG:teuthology.orchestra.run.smithi187:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-16T07:54:04.668 INFO:teuthology.orchestra.run.smithi187.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T07:54:04.668 INFO:teuthology.orchestra.run.smithi187.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T07:54:04.668 INFO:teuthology.orchestra.run.smithi187.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T07:54:04.668 INFO:teuthology.orchestra.run.smithi187.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-16T07:54:04.668 INFO:teuthology.orchestra.run.smithi187.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T07:54:04.669 INFO:teuthology.orchestra.run.smithi187.stdout: = sunit=0 swidth=0 blks 2024-09-16T07:54:04.669 INFO:teuthology.orchestra.run.smithi187.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T07:54:04.669 INFO:teuthology.orchestra.run.smithi187.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-16T07:54:04.669 INFO:teuthology.orchestra.run.smithi187.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T07:54:04.669 INFO:teuthology.orchestra.run.smithi187.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T07:54:05.111 INFO:teuthology.orchestra.run.smithi187.stdout:Discarding blocks...Done. 2024-09-16T07:54:05.114 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi187.front.sepia.ceph.com -o noatime 2024-09-16T07:54:05.114 DEBUG:teuthology.orchestra.run.smithi187:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-3 2024-09-16T07:54:05.193 DEBUG:teuthology.orchestra.run.smithi187:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-09-16T07:54:05.263 DEBUG:teuthology.orchestra.run.smithi187:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T07:54:05.572 INFO:teuthology.orchestra.run.smithi187.stderr:WARNING: debug mode. Not for benchmarking or production 2024-09-16T07:54:05.573 INFO:teuthology.orchestra.run.smithi187.stderr:INFO 2024-09-16 07:54:05,575 seastar - Reactor backend: linux-aio 2024-09-16T07:54:05.588 INFO:teuthology.orchestra.run.smithi187.stderr:==27604==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-09-16T07:54:05.709 INFO:teuthology.orchestra.run.smithi187.stderr:INFO 2024-09-16 07:54:05,712 [shard 0:main] osd - bypassing the config fetch due to --no-mon-config 2024-09-16T07:54:05.709 INFO:teuthology.orchestra.run.smithi187.stderr:INFO 2024-09-16 07:54:05,712 [shard 0:main] osd - get_early_config: set --thread-affinity 1 --cpuset 0-2 2024-09-16T07:54:05.710 INFO:teuthology.orchestra.run.smithi187.stderr:DEBUG 2024-09-16 07:54:05,713 [shard 0:main] seastar - reactor::drain 2024-09-16T07:54:06.236 INFO:teuthology.orchestra.run.smithi187.stderr:----------------------------------------------------- 2024-09-16T07:54:06.236 INFO:teuthology.orchestra.run.smithi187.stderr:Suppressions used: 2024-09-16T07:54:06.236 INFO:teuthology.orchestra.run.smithi187.stderr: count bytes template 2024-09-16T07:54:06.236 INFO:teuthology.orchestra.run.smithi187.stderr: 1 8 MallocExtension::Register 2024-09-16T07:54:06.236 INFO:teuthology.orchestra.run.smithi187.stderr:----------------------------------------------------- 2024-09-16T07:54:06.236 INFO:teuthology.orchestra.run.smithi187.stderr: 2024-09-16T07:54:06.298 INFO:teuthology.orchestra.run.smithi187.stderr:WARNING: debug mode. Not for benchmarking or production 2024-09-16T07:54:06.302 INFO:teuthology.orchestra.run.smithi187.stderr:INFO 2024-09-16 07:54:06,304 seastar - Reactor backend: linux-aio 2024-09-16T07:54:06.311 INFO:teuthology.orchestra.run.smithi187.stderr:==27602==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-09-16T07:54:06.376 INFO:teuthology.orchestra.run.smithi187.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbca834f 0xbb80cd9 0xbb82244 0xbb8245a 0xbb825b0 0xbb82a79 0x3e6ef 0x5e01003 0x5e19231 0x71579d2 0x71691a5 0x716a642 0x716a6c7 0x716a7c9 0x716a8c1 0x716aace 0x716acb4 0x716b003 0x716b3a7 0x716b5d7 0x716b841 0x716bb0e 0x716bd2c 0x716bde4 0x716beaa 0x716bf8b 0x5c63e30 0x5c63f3a 0x5c63ff6 0x5c640b2 0x5c6416e 0x5c642bb 0x5c64446 0x5c64807 0x5c7e60a 0xbb95287 0xbbad105 0xbc5a9af 0xbc5bffd 0xb92a8cd 0xb92b105 0x3805438 0x2958f 0x2963f 0x35ec9b4 2024-09-16T07:54:06.376 INFO:teuthology.orchestra.run.smithi187.stderr:kernel callstack: 2024-09-16T07:54:06.594 INFO:teuthology.orchestra.run.smithi187.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbca834f 0xbb80cd9 0xbb82244 0xbb8245a 0xbb825b0 0xbb82a79 0x3e6ef 0xb9d8abc 0xb9dc6f8 0xb9c7696 0xb9c7a7a 0x9afa023 0x9b11b72 0x99a2168 0x9942951 0x927469d 0x932dfed 0x932ef8c 0x932f00f 0x932f094 0x932f196 0x932f28e 0x932f314 0x932f3cd 0x932f583 0x932f6bb 0x932f736 0x932f7bb 0x932f916 0x932fa30 0x932fab5 0x932fb93 0x9552886 0x961b5f3 0x961b770 0x96dc4d1 0xbb95287 0xbbad105 0xbc5a9af 0xbc6cfab 0xbc6d62d 0x7b30c45 0xc602d44 0x5e669 0x89c51 0x10ec7f 2024-09-16T07:54:06.595 INFO:teuthology.orchestra.run.smithi187.stderr:kernel callstack: 2024-09-16T07:54:06.595 INFO:teuthology.orchestra.run.smithi187.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: 0x82dff 0xbca834f 0xbb80cd9 0xbb82244 0xbb8245a 0xbb825b0 0xbb82a79 0x3e6ef 0xba27128 0xba2757f 0xba27a41 0xba27e99 0xb9cba66 0xba283d8 0xba2861f 0xba286cf 0xb9d809d 0xb9dc6f8 0xb9c7696 0xb9c7a7a 0x9afa023 0x9b11b72 0x99a2168 0x9942951 0x927469d 0x932dfed 0x932ef8c 0x932f00f 0x932f094 0x932f196 0x932f28e 0x932f314 0x932f3cd 0x932f583 0x932f6bb 0x932f736 0x932f7bb 0x932f916 0x932fa30 0x932fab5 0x932fb93 0x9552886 0x961b5f3 0x961b770 0x96dc4d1 0xbb95287 0xbbad105 0xbc5a9af 0xbc6cfab 0xbc6d62d 0x7b30c45 0xc602d44 0x5e669 0x89c51 0x10ec7f 2024-09-16T07:54:06.595 INFO:teuthology.orchestra.run.smithi187.stderr:kernel callstack: 2024-09-16T07:54:06.652 INFO:teuthology.orchestra.run.smithi187.stderr:Reactor stalled for 90 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbca834f 0xbb80cd9 0xbb82244 0xbb8245a 0xbb825b0 0xbb82a79 0x3e6ef 0xba2028f 0xba2ad3d 0xba2af09 0xb9d201b 0xb9d2514 0xb9d9f7f 0xb9dc6f8 0xb9c7696 0xb9c7a7a 0x9e12036 0x9e2237e 0x99fffb1 0x9a005b7 0x98f54f0 0x932e1f1 0x932ef8c 0x932f00f 0x932f094 0x932f196 0x932f28e 0x932f314 0x932f3cd 0x932f583 0x932f6bb 0x932f736 0x932f7bb 0x932f916 0x932fa30 0x932fab5 0x932fb93 0x9552886 0x961b5f3 0x961b770 0x96dc4d1 0xbb95287 0xbbad105 0xbc5a9af 0xbc6cfab 0xbc6d62d 0x7b30c45 0xc602d44 0x5e669 0x89c51 0x10ec7f 2024-09-16T07:54:06.652 INFO:teuthology.orchestra.run.smithi187.stderr:Reactor stalled for 90 ms on shard 2, in scheduling group main. Backtrace: 0x82dff 0xbca834f 0xbb80cd9 0xbb82244 0xbb8245a 0xbb825b0 0xbb82a79 0x3e6ef 0x400b7 0x42076 0x3ef7b 0xfd531 0x3c34f60 0x55593bc 0x9ed0efd 0x9ed14b6 0x9ed185e 0x9ed4c38 0x9ed5894 0x9ed5b57 0x9ed67ce 0x9e21d32 0x99fffb1 0x9a005b7 0x98f54f0 0x932e1f1 0x932ef8c 0x932f00f 0x932f094 0x932f196 0x932f28e 0x932f314 0x932f3cd 0x932f583 0x932f6bb 0x932f736 0x932f7bb 0x932f916 0x932fa30 0x932fab5 0x932fb93 0x9552886 0x961b5f3 0x961b770 0x96dc4d1 0xbb95287 0xbbad105 0xbc5a9af 0xbc6cfab 0xbc6d62d 0x7b30c45 0xc602d44 0x5e669 0x89c51 0x10ec7f 2024-09-16T07:54:06.653 INFO:teuthology.orchestra.run.smithi187.stderr:kernel callstack: 2024-09-16T07:54:06.653 INFO:teuthology.orchestra.run.smithi187.stderr:kernel callstack: 2024-09-16T07:54:06.775 INFO:teuthology.orchestra.run.smithi187.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbca834f 0xbb80cd9 0xbb82244 0xbb8245a 0xbb825b0 0xbb82a79 0x3e6ef 0x5541f18 0x9e33959 0x9ea2e39 0x9ed3631 0x9e3abc4 0x9959157 0x9878562 0x989cc22 0x989cd81 0x98ae61e 0x98ae7ee 0x98aeafc 0x98aee0b 0x98aef6a 0x98aefe5 0x98af22c 0x98af4f0 0x98af580 0x98af624 0x95c34b2 0x95c3717 0x95cbcbc 0x96be366 0xbb95287 0xbbad105 0xbc5a9af 0xbc5bffd 0xb92a8cd 0xb92b105 0x3805438 0x2958f 0x2963f 0x35ec9b4 2024-09-16T07:54:06.775 INFO:teuthology.orchestra.run.smithi187.stderr:kernel callstack: 2024-09-16T07:54:06.832 INFO:teuthology.orchestra.run.smithi187.stderr:Reactor stalled for 122 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbca834f 0xbb80cd9 0xbb82244 0xbb8245a 0xbb825b0 0xbb82a79 0x3e6ef 0x5542db1 0x9e33940 0x9ea2e39 0x9ed3631 0x9e3abc4 0x9959157 0x9878562 0x989cc22 0x989cd81 0x98ae61e 0x98ae7ee 0x98aeafc 0x98aee0b 0x98aef6a 0x98aefe5 0x98af22c 0x98af4f0 0x98af580 0x98af624 0x95c34b2 0x95c3717 0x95cbcbc 0x96be366 0xbb95287 0xbbad105 0xbc5a9af 0xbc5bffd 0xb92a8cd 0xb92b105 0x3805438 0x2958f 0x2963f 0x35ec9b4 2024-09-16T07:54:06.832 INFO:teuthology.orchestra.run.smithi187.stderr:kernel callstack: 2024-09-16T07:54:06.863 INFO:teuthology.orchestra.run.smithi187.stderr:Reactor stalled for 65 ms on shard 2, in scheduling group main. Backtrace: 0x82dff 0xbca834f 0xbb80cd9 0xbb82244 0xbb8245a 0xbb825b0 0xbb82a79 0x3e6ef 0x4ab04 0x5542dfb 0x9e33940 0x9ea2e39 0x9ed3631 0x9e3abc4 0x9959157 0x9878562 0x989cc22 0x989cd81 0x98ae61e 0x98ae7ee 0x98aeafc 0x98aee0b 0x98aef6a 0x98aefe5 0x98af22c 0x98af4f0 0x98af580 0x98af624 0x95c34b2 0x95c3717 0x95cbcbc 0x96be366 0xbb95287 0xbbad105 0xbc5a9af 0xbc6cfab 0xbc6d62d 0x7b30c45 0xc602d44 0x5e669 0x89c51 0x10ec7f 2024-09-16T07:54:06.863 INFO:teuthology.orchestra.run.smithi187.stderr:kernel callstack: 2024-09-16T07:54:06.863 INFO:teuthology.orchestra.run.smithi187.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbca834f 0xbb80cd9 0xbb82244 0xbb8245a 0xbb825b0 0xbb82a79 0x3e6ef 0x133c44 0xdd332 0x4ab1c 0x5542dfb 0x9e33940 0x9ea2e39 0x9ed3631 0x9e3abc4 0x9959157 0x9878562 0x989cc22 0x989cd81 0x98ae61e 0x98ae7ee 0x98aeafc 0x98aee0b 0x98aef6a 0x98aefe5 0x98af22c 0x98af4f0 0x98af580 0x98af624 0x95c34b2 0x95c3717 0x95cbcbc 0x96be366 0xbb95287 0xbbad105 0xbc5a9af 0xbc6cfab 0xbc6d62d 0x7b30c45 0xc602d44 0x5e669 0x89c51 0x10ec7f 2024-09-16T07:54:06.863 INFO:teuthology.orchestra.run.smithi187.stderr:kernel callstack: 2024-09-16T07:54:06.921 INFO:teuthology.orchestra.run.smithi187.stderr:Reactor stalled for 122 ms on shard 2, in scheduling group main. Backtrace: 0x82dff 0xbca834f 0xbb80cd9 0xbb82244 0xbb8245a 0xbb825b0 0xbb82a79 0x3e6ef 0x55422c4 0x9e3396e 0x9ea2e39 0x9ed3631 0x9e3abc4 0x9959157 0x9878562 0x989cc22 0x989cd81 0x98ae61e 0x98ae7ee 0x98aeafc 0x98aee0b 0x98aef6a 0x98aefe5 0x98af22c 0x98af4f0 0x98af580 0x98af624 0x95c34b2 0x95c3717 0x95cbcbc 0x96be366 0xbb95287 0xbbad105 0xbc5a9af 0xbc6cfab 0xbc6d62d 0x7b30c45 0xc602d44 0x5e669 0x89c51 0x10ec7f 2024-09-16T07:54:06.921 INFO:teuthology.orchestra.run.smithi187.stderr:kernel callstack: 2024-09-16T07:54:06.921 INFO:teuthology.orchestra.run.smithi187.stderr:Reactor stalled for 122 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbca834f 0xbb80cd9 0xbb82244 0xbb8245a 0xbb825b0 0xbb82a79 0x3e6ef 0x133c27 0xdd332 0x4ab1c 0x5542dfb 0x9e33940 0x9ea2e39 0x9ed3631 0x9e3abc4 0x9959157 0x9878562 0x989cc22 0x989cd81 0x98ae61e 0x98ae7ee 0x98aeafc 0x98aee0b 0x98aef6a 0x98aefe5 0x98af22c 0x98af4f0 0x98af580 0x98af624 0x95c34b2 0x95c3717 0x95cbcbc 0x96be366 0xbb95287 0xbbad105 0xbc5a9af 0xbc6cfab 0xbc6d62d 0x7b30c45 0xc602d44 0x5e669 0x89c51 0x10ec7f 2024-09-16T07:54:06.921 INFO:teuthology.orchestra.run.smithi187.stderr:kernel callstack: 2024-09-16T07:54:06.940 INFO:teuthology.orchestra.run.smithi187.stderr:Reactor stalled for 230 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbca834f 0xbb80cd9 0xbb82244 0xbb8245a 0xbb825b0 0xbb82a79 0x3e6ef 0x55422d9 0x9e3396e 0x9ea2e39 0x9ed3631 0x9e3abc4 0x9959157 0x9878562 0x989cc22 0x989cd81 0x98ae61e 0x98ae7ee 0x98aeafc 0x98aee0b 0x98aef6a 0x98aefe5 0x98af22c 0x98af4f0 0x98af580 0x98af624 0x95c34b2 0x95c3717 0x95cbcbc 0x96be366 0xbb95287 0xbbad105 0xbc5a9af 0xbc5bffd 0xb92a8cd 0xb92b105 0x3805438 0x2958f 0x2963f 0x35ec9b4 2024-09-16T07:54:06.940 INFO:teuthology.orchestra.run.smithi187.stderr:kernel callstack: 2024-09-16T07:54:07.028 INFO:teuthology.orchestra.run.smithi187.stderr:Reactor stalled for 230 ms on shard 2, in scheduling group main. Backtrace: 0x82dff 0xbca834f 0xbb80cd9 0xbb82244 0xbb8245a 0xbb825b0 0xbb82a79 0x3e6ef 0x5542726 0x9e3397b 0x9ea2e39 0x9ed3631 0x9e3abc4 0x9959157 0x9878562 0x989cc22 0x989cd81 0x98ae61e 0x98ae7ee 0x98aeafc 0x98aee0b 0x98aef6a 0x98aefe5 0x98af22c 0x98af4f0 0x98af580 0x98af624 0x95c34b2 0x95c3717 0x95cbcbc 0x96be366 0xbb95287 0xbbad105 0xbc5a9af 0xbc6cfab 0xbc6d62d 0x7b30c45 0xc602d44 0x5e669 0x89c51 0x10ec7f 2024-09-16T07:54:07.029 INFO:teuthology.orchestra.run.smithi187.stderr:kernel callstack: 2024-09-16T07:54:07.029 INFO:teuthology.orchestra.run.smithi187.stderr:Reactor stalled for 230 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbca834f 0xbb80cd9 0xbb82244 0xbb8245a 0xbb825b0 0xbb82a79 0x3e6ef 0x8b571 0x10d76c 0xdd31a 0x4ab1c 0x5542dfb 0x9e33940 0x9ea2e39 0x9ed3631 0x9e3abc4 0x9959157 0x9878562 0x989cc22 0x989cd81 0x98ae61e 0x98ae7ee 0x98aeafc 0x98aee0b 0x98aef6a 0x98aefe5 0x98af22c 0x98af4f0 0x98af580 0x98af624 0x95c34b2 0x95c3717 0x95cbcbc 0x96be366 0xbb95287 0xbbad105 0xbc5a9af 0xbc6cfab 0xbc6d62d 0x7b30c45 0xc602d44 0x5e669 0x89c51 0x10ec7f 2024-09-16T07:54:07.029 INFO:teuthology.orchestra.run.smithi187.stderr:kernel callstack: 2024-09-16T07:54:07.336 INFO:teuthology.orchestra.run.smithi187.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbca834f 0xbb80cd9 0xbb82244 0xbb8245a 0xbb825b0 0xbb82a79 0x3e6ef 0x5542d1c 0x9e33940 0x9ea2e39 0x9ed3631 0x9e3abc4 0x9959157 0x9878562 0x989cc22 0x989cd81 0x98ae61e 0x98ae7ee 0x98aeafc 0x98aee0b 0x98aef6a 0x98aefe5 0x98af22c 0x98af4f0 0x98af580 0x98af624 0x95c34b2 0x95c3717 0x95cbcbc 0x96be366 0xbb95287 0xbbad105 0xbc5a9af 0xbc5bffd 0xb92a8cd 0xb92b105 0x3805438 0x2958f 0x2963f 0x35ec9b4 2024-09-16T07:54:07.336 INFO:teuthology.orchestra.run.smithi187.stderr:kernel callstack: 2024-09-16T07:54:08.262 INFO:teuthology.orchestra.run.smithi187.stderr:----------------------------------------------------- 2024-09-16T07:54:08.263 INFO:teuthology.orchestra.run.smithi187.stderr:Suppressions used: 2024-09-16T07:54:08.263 INFO:teuthology.orchestra.run.smithi187.stderr: count bytes template 2024-09-16T07:54:08.263 INFO:teuthology.orchestra.run.smithi187.stderr: 1 8 MallocExtension::Register 2024-09-16T07:54:08.263 INFO:teuthology.orchestra.run.smithi187.stderr:----------------------------------------------------- 2024-09-16T07:54:08.263 INFO:teuthology.orchestra.run.smithi187.stderr: 2024-09-16T07:54:08.290 DEBUG:teuthology.orchestra.run.smithi187:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-16T07:54:08.316 DEBUG:teuthology.orchestra.run.smithi187:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T07:54:08.594 INFO:teuthology.orchestra.run.smithi187.stderr:WARNING: debug mode. Not for benchmarking or production 2024-09-16T07:54:08.595 INFO:teuthology.orchestra.run.smithi187.stderr:INFO 2024-09-16 07:54:08,598 seastar - Reactor backend: linux-aio 2024-09-16T07:54:08.601 INFO:teuthology.orchestra.run.smithi187.stderr:==27652==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-09-16T07:54:08.719 INFO:teuthology.orchestra.run.smithi187.stderr:INFO 2024-09-16 07:54:08,722 [shard 0:main] osd - bypassing the config fetch due to --no-mon-config 2024-09-16T07:54:08.719 INFO:teuthology.orchestra.run.smithi187.stderr:INFO 2024-09-16 07:54:08,722 [shard 0:main] osd - get_early_config: set --thread-affinity 1 --cpuset 3-5 2024-09-16T07:54:08.720 INFO:teuthology.orchestra.run.smithi187.stderr:DEBUG 2024-09-16 07:54:08,723 [shard 0:main] seastar - reactor::drain 2024-09-16T07:54:09.230 INFO:teuthology.orchestra.run.smithi187.stderr:----------------------------------------------------- 2024-09-16T07:54:09.230 INFO:teuthology.orchestra.run.smithi187.stderr:Suppressions used: 2024-09-16T07:54:09.230 INFO:teuthology.orchestra.run.smithi187.stderr: count bytes template 2024-09-16T07:54:09.230 INFO:teuthology.orchestra.run.smithi187.stderr: 1 8 MallocExtension::Register 2024-09-16T07:54:09.230 INFO:teuthology.orchestra.run.smithi187.stderr:----------------------------------------------------- 2024-09-16T07:54:09.230 INFO:teuthology.orchestra.run.smithi187.stderr: 2024-09-16T07:54:09.287 INFO:teuthology.orchestra.run.smithi187.stderr:WARNING: debug mode. Not for benchmarking or production 2024-09-16T07:54:09.291 INFO:teuthology.orchestra.run.smithi187.stderr:INFO 2024-09-16 07:54:09,293 seastar - Reactor backend: linux-aio 2024-09-16T07:54:09.300 INFO:teuthology.orchestra.run.smithi187.stderr:==27650==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-09-16T07:54:09.391 INFO:teuthology.orchestra.run.smithi187.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbca834f 0xbb80cd9 0xbb82244 0xbb8245a 0xbb825b0 0xbb82a79 0x3e6ef 0x381b079 0x381b468 0x5fdd485 0x5fdd8fa 0x6038b24 0x5fb8d06 0x5fb8ee0 0x5fb946c 0x5e00fef 0x5e19231 0x71579d2 0x71691a5 0x716a642 0x716a6c7 0x716a7c9 0x716a8c1 0x716aace 0x716acb4 0x716b003 0x716b3a7 0x716b5d7 0x716b841 0x716bb0e 0x716bd2c 0x716bde4 0x716beaa 0x716bf8b 0x5c63e30 0x5c63f3a 0x5c63ff6 0x5c640b2 0x5c6416e 0x5c642bb 0x5c64446 0x5c64807 0x5c7e60a 0xbb95287 0xbbad105 0xbc5a9af 0xbc5bffd 0xb92a8cd 0xb92b105 0x3805438 0x2958f 0x2963f 0x35ec9b4 2024-09-16T07:54:09.392 INFO:teuthology.orchestra.run.smithi187.stderr:kernel callstack: 2024-09-16T07:54:09.584 INFO:teuthology.orchestra.run.smithi187.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: 0x82dff 0xbca834f 0xbb80cd9 0xbb82244 0xbb8245a 0xbb825b0 0xbb82a79 0x3e6ef 0x3611987 0x3611d77 0xba28a97 0xba28dcc 0xba28e7c 0xba293a7 0xba29e31 0xb9dc97b 0xb9c7696 0xb9c7a7a 0x9afa023 0x9b11b72 0x99a2168 0x9942951 0x927469d 0x932dfed 0x932ef8c 0x932f00f 0x932f094 0x932f196 0x932f28e 0x932f314 0x932f3cd 0x932f583 0x932f6bb 0x932f736 0x932f7bb 0x932f916 0x932fa30 0x932fab5 0x932fb93 0x9552886 0x961b5f3 0x961b770 0x96dc4d1 0xbb95287 0xbbad105 0xbc5a9af 0xbc6cfab 0xbc6d62d 0x7b30c45 0xc602d44 0x5e669 0x89c51 0x10ec7f 2024-09-16T07:54:09.585 INFO:teuthology.orchestra.run.smithi187.stderr:kernel callstack: 2024-09-16T07:54:09.585 INFO:teuthology.orchestra.run.smithi187.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbca834f 0xbb80cd9 0xbb82244 0xbb8245a 0xbb825b0 0xbb82a79 0x3e6ef 0x162c83 0x422c4 0x3ef7b 0xfd531 0x361b453 0x361b65c 0x362a2bb 0x382b1d8 0x4550650 0xb9d216a 0xb9d2514 0xb9d84d7 0xb9dc6f8 0xb9c7696 0xb9c7a7a 0x9afa023 0x9b11b72 0x99a2168 0x9942951 0x927469d 0x932dfed 0x932ef8c 0x932f00f 0x932f094 0x932f196 0x932f28e 0x932f314 0x932f3cd 0x932f583 0x932f6bb 0x932f736 0x932f7bb 0x932f916 0x932fa30 0x932fab5 0x932fb93 0x9552886 0x961b5f3 0x961b770 0x96dc4d1 0xbb95287 0xbbad105 0xbc5a9af 0xbc6cfab 0xbc6d62d 0x7b30c45 0xc602d44 0x5e669 0x89c51 0x10ec7f 2024-09-16T07:54:09.585 INFO:teuthology.orchestra.run.smithi187.stderr:kernel callstack: 0xffffffffffffff80 0xffffffffb2dbb2b2 0xffffffffb2dbb929 0xffffffffb2dbd9d9 0xffffffffb2dbee42 0xffffffffb2dbf6b7 0xffffffffb2de4853 0xffffffffb2d8ebde 0xffffffffb2d96a3b 0xffffffffb2d96ebc 0xffffffffb2a8c884 0xffffffffb36d1d62 0xffffffffb3800bb2 2024-09-16T07:54:09.642 INFO:teuthology.orchestra.run.smithi187.stderr:Reactor stalled for 90 ms on shard 2, in scheduling group main. Backtrace: 0x82dff 0xbca834f 0xbb80cd9 0xbb82244 0xbb8245a 0xbb825b0 0xbb82a79 0x3e6ef 0x10797f 0xdd2fc 0x4b14c 0x3b8c0f7 0x3b8c278 0xb9e909a 0xb9ea53a 0xb9ea7b4 0xb9ea95c 0xb9ec717 0xb9ecb62 0xb9d8a8c 0xb9dc6f8 0xb9c7696 0xb9c7a7a 0x9afefe9 0x9b11b72 0x99a2168 0x9942951 0x927469d 0x932dfed 0x932ef8c 0x932f00f 0x932f094 0x932f196 0x932f28e 0x932f314 0x932f3cd 0x932f583 0x932f6bb 0x932f736 0x932f7bb 0x932f916 0x932fa30 0x932fab5 0x932fb93 0x9552886 0x961b5f3 0x961b770 0x96dc4d1 0xbb95287 0xbbad105 0xbc5a9af 0xbc6cfab 0xbc6d62d 0x7b30c45 0xc602d44 0x5e669 0x89c51 0x10ec7f 2024-09-16T07:54:09.642 INFO:teuthology.orchestra.run.smithi187.stderr:Reactor stalled for 90 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbca834f 0xbb80cd9 0xbb82244 0xbb8245a 0xbb825b0 0xbb82a79 0x3e6ef 0x400db 0x42076 0x3ef7b 0xfd531 0xb85348d 0xb853540 0xb853656 0xb853736 0xb8539b6 0xb853f61 0xb85436c 0xb85462b 0xb8550e8 0xba25db6 0xb9caf71 0x9e0d3f7 0x9e2237e 0x99fffb1 0x9a005b7 0x98f54f0 0x932e1f1 0x932ef8c 0x932f00f 0x932f094 0x932f196 0x932f28e 0x932f314 0x932f3cd 0x932f583 0x932f6bb 0x932f736 0x932f7bb 0x932f916 0x932fa30 0x932fab5 0x932fb93 0x9552886 0x961b5f3 0x961b770 0x96dc4d1 0xbb95287 0xbbad105 0xbc5a9af 0xbc6cfab 0xbc6d62d 0x7b30c45 0xc602d44 0x5e669 0x89c51 0x10ec7f 2024-09-16T07:54:09.642 INFO:teuthology.orchestra.run.smithi187.stderr:kernel callstack: 2024-09-16T07:54:09.643 INFO:teuthology.orchestra.run.smithi187.stderr:kernel callstack: 2024-09-16T07:54:09.774 INFO:teuthology.orchestra.run.smithi187.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbca834f 0xbb80cd9 0xbb82244 0xbb8245a 0xbb825b0 0xbb82a79 0x3e6ef 0x8b56c 0x10d76c 0xdd31a 0x4ab1c 0x5542dfb 0x9e33940 0x9ea2e39 0x9ed3631 0x9e3abc4 0x9959157 0x9878562 0x989cc22 0x989cd81 0x98ae61e 0x98ae7ee 0x98aeafc 0x98aee0b 0x98aef6a 0x98aefe5 0x98af22c 0x98af4f0 0x98af580 0x98af624 0x95c34b2 0x95c3717 0x95cbcbc 0x96be366 0xbb95287 0xbbad105 0xbc5a9af 0xbc5bffd 0xb92a8cd 0xb92b105 0x3805438 0x2958f 0x2963f 0x35ec9b4 2024-09-16T07:54:09.774 INFO:teuthology.orchestra.run.smithi187.stderr:kernel callstack: 2024-09-16T07:54:09.831 INFO:teuthology.orchestra.run.smithi187.stderr:Reactor stalled for 122 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbca834f 0xbb80cd9 0xbb82244 0xbb8245a 0xbb825b0 0xbb82a79 0x3e6ef 0x55420de 0x9e33959 0x9ea2e39 0x9ed3631 0x9e3abc4 0x9959157 0x9878562 0x989cc22 0x989cd81 0x98ae61e 0x98ae7ee 0x98aeafc 0x98aee0b 0x98aef6a 0x98aefe5 0x98af22c 0x98af4f0 0x98af580 0x98af624 0x95c34b2 0x95c3717 0x95cbcbc 0x96be366 0xbb95287 0xbbad105 0xbc5a9af 0xbc5bffd 0xb92a8cd 0xb92b105 0x3805438 0x2958f 0x2963f 0x35ec9b4 2024-09-16T07:54:09.832 INFO:teuthology.orchestra.run.smithi187.stderr:kernel callstack: 2024-09-16T07:54:09.841 INFO:teuthology.orchestra.run.smithi187.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbca834f 0xbb80cd9 0xbb82244 0xbb8245a 0xbb825b0 0xbb82a79 0x3e6ef 0x5542f29 0x9e33940 0x9ea2e39 0x9ed3631 0x9e3abc4 0x9959157 0x9878562 0x989cc22 0x989cd81 0x98ae61e 0x98ae7ee 0x98aeafc 0x98aee0b 0x98aef6a 0x98aefe5 0x98af22c 0x98af4f0 0x98af580 0x98af624 0x95c34b2 0x95c3717 0x95cbcbc 0x96be366 0xbb95287 0xbbad105 0xbc5a9af 0xbc6cfab 0xbc6d62d 0x7b30c45 0xc602d44 0x5e669 0x89c51 0x10ec7f 2024-09-16T07:54:09.841 INFO:teuthology.orchestra.run.smithi187.stderr:kernel callstack: 2024-09-16T07:54:09.883 INFO:teuthology.orchestra.run.smithi187.stderr:Reactor stalled for 65 ms on shard 2, in scheduling group main. Backtrace: 0x82dff 0xbca834f 0xbb80cd9 0xbb82244 0xbb8245a 0xbb825b0 0xbb82a79 0x3e6ef 0x5542f29 0x9e33940 0x9ea2e39 0x9ed3631 0x9e3abc4 0x9959157 0x9878562 0x989cc22 0x989cd81 0x98ae61e 0x98ae7ee 0x98aeafc 0x98aee0b 0x98aef6a 0x98aefe5 0x98af22c 0x98af4f0 0x98af580 0x98af624 0x95c34b2 0x95c3717 0x95cbcbc 0x96be366 0xbb95287 0xbbad105 0xbc5a9af 0xbc6cfab 0xbc6d62d 0x7b30c45 0xc602d44 0x5e669 0x89c51 0x10ec7f 2024-09-16T07:54:09.883 INFO:teuthology.orchestra.run.smithi187.stderr:kernel callstack: 2024-09-16T07:54:09.899 INFO:teuthology.orchestra.run.smithi187.stderr:Reactor stalled for 90 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbca834f 0xbb80cd9 0xbb82244 0xbb8245a 0xbb825b0 0xbb82a79 0x3e6ef 0x357849f 0x5542dfb 0x9e33940 0x9ea2e39 0x9ed3631 0x9e3abc4 0x9959157 0x9878562 0x989cc22 0x989cd81 0x98ae61e 0x98ae7ee 0x98aeafc 0x98aee0b 0x98aef6a 0x98aefe5 0x98af22c 0x98af4f0 0x98af580 0x98af624 0x95c34b2 0x95c3717 0x95cbcbc 0x96be366 0xbb95287 0xbbad105 0xbc5a9af 0xbc6cfab 0xbc6d62d 0x7b30c45 0xc602d44 0x5e669 0x89c51 0x10ec7f 2024-09-16T07:54:09.899 INFO:teuthology.orchestra.run.smithi187.stderr:kernel callstack: 2024-09-16T07:54:09.939 INFO:teuthology.orchestra.run.smithi187.stderr:Reactor stalled for 230 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbca834f 0xbb80cd9 0xbb82244 0xbb8245a 0xbb825b0 0xbb82a79 0x3e6ef 0x9e33959 0x9ea2e39 0x9ed3631 0x9e3abc4 0x9959157 0x9878562 0x989cc22 0x989cd81 0x98ae61e 0x98ae7ee 0x98aeafc 0x98aee0b 0x98aef6a 0x98aefe5 0x98af22c 0x98af4f0 0x98af580 0x98af624 0x95c34b2 0x95c3717 0x95cbcbc 0x96be366 0xbb95287 0xbbad105 0xbc5a9af 0xbc5bffd 0xb92a8cd 0xb92b105 0x3805438 0x2958f 0x2963f 0x35ec9b4 2024-09-16T07:54:09.939 INFO:teuthology.orchestra.run.smithi187.stderr:kernel callstack: 2024-09-16T07:54:09.941 INFO:teuthology.orchestra.run.smithi187.stderr:Reactor stalled for 122 ms on shard 2, in scheduling group main. Backtrace: 0x82dff 0xbca834f 0xbb80cd9 0xbb82244 0xbb8245a 0xbb825b0 0xbb82a79 0x3e6ef 0x8b56c 0x10d76c 0xdd31a 0x4ab1c 0x5542dfb 0x9e33940 0x9ea2e39 0x9ed3631 0x9e3abc4 0x9959157 0x9878562 0x989cc22 0x989cd81 0x98ae61e 0x98ae7ee 0x98aeafc 0x98aee0b 0x98aef6a 0x98aefe5 0x98af22c 0x98af4f0 0x98af580 0x98af624 0x95c34b2 0x95c3717 0x95cbcbc 0x96be366 0xbb95287 0xbbad105 0xbc5a9af 0xbc6cfab 0xbc6d62d 0x7b30c45 0xc602d44 0x5e669 0x89c51 0x10ec7f 2024-09-16T07:54:09.941 INFO:teuthology.orchestra.run.smithi187.stderr:kernel callstack: 2024-09-16T07:54:10.007 INFO:teuthology.orchestra.run.smithi187.stderr:Reactor stalled for 197 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbca834f 0xbb80cd9 0xbb82244 0xbb8245a 0xbb825b0 0xbb82a79 0x3e6ef 0x5541df2 0x5542d9c 0x9e33940 0x9ea2e39 0x9ed3631 0x9e3abc4 0x9959157 0x9878562 0x989cc22 0x989cd81 0x98ae61e 0x98ae7ee 0x98aeafc 0x98aee0b 0x98aef6a 0x98aefe5 0x98af22c 0x98af4f0 0x98af580 0x98af624 0x95c34b2 0x95c3717 0x95cbcbc 0x96be366 0xbb95287 0xbbad105 0xbc5a9af 0xbc6cfab 0xbc6d62d 0x7b30c45 0xc602d44 0x5e669 0x89c51 0x10ec7f 2024-09-16T07:54:10.007 INFO:teuthology.orchestra.run.smithi187.stderr:kernel callstack: 2024-09-16T07:54:10.048 INFO:teuthology.orchestra.run.smithi187.stderr:Reactor stalled for 230 ms on shard 2, in scheduling group main. Backtrace: 0x82dff 0xbca834f 0xbb80cd9 0xbb82244 0xbb8245a 0xbb825b0 0xbb82a79 0x3e6ef 0x4ab82 0x5542dfb 0x9e33940 0x9ea2e39 0x9ed3631 0x9e3abc4 0x9959157 0x9878562 0x989cc22 0x989cd81 0x98ae61e 0x98ae7ee 0x98aeafc 0x98aee0b 0x98aef6a 0x98aefe5 0x98af22c 0x98af4f0 0x98af580 0x98af624 0x95c34b2 0x95c3717 0x95cbcbc 0x96be366 0xbb95287 0xbbad105 0xbc5a9af 0xbc6cfab 0xbc6d62d 0x7b30c45 0xc602d44 0x5e669 0x89c51 0x10ec7f 2024-09-16T07:54:10.049 INFO:teuthology.orchestra.run.smithi187.stderr:kernel callstack: 2024-09-16T07:54:10.358 INFO:teuthology.orchestra.run.smithi187.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbca834f 0xbb80cd9 0xbb82244 0xbb8245a 0xbb825b0 0xbb82a79 0x3e6ef 0x4abb7 0x5542dfb 0x9e33940 0x9ea2e39 0x9ed3631 0x9e3abc4 0x9959157 0x9878562 0x989cc22 0x989cd81 0x98ae61e 0x98ae7ee 0x98aeafc 0x98aee0b 0x98aef6a 0x98aefe5 0x98af22c 0x98af4f0 0x98af580 0x98af624 0x95c34b2 0x95c3717 0x95cbcbc 0x96be366 0xbb95287 0xbbad105 0xbc5a9af 0xbc5bffd 0xb92a8cd 0xb92b105 0x3805438 0x2958f 0x2963f 0x35ec9b4 2024-09-16T07:54:10.358 INFO:teuthology.orchestra.run.smithi187.stderr:kernel callstack: 2024-09-16T07:54:11.290 INFO:teuthology.orchestra.run.smithi187.stderr:----------------------------------------------------- 2024-09-16T07:54:11.290 INFO:teuthology.orchestra.run.smithi187.stderr:Suppressions used: 2024-09-16T07:54:11.290 INFO:teuthology.orchestra.run.smithi187.stderr: count bytes template 2024-09-16T07:54:11.291 INFO:teuthology.orchestra.run.smithi187.stderr: 1 8 MallocExtension::Register 2024-09-16T07:54:11.291 INFO:teuthology.orchestra.run.smithi187.stderr:----------------------------------------------------- 2024-09-16T07:54:11.291 INFO:teuthology.orchestra.run.smithi187.stderr: 2024-09-16T07:54:11.317 DEBUG:teuthology.orchestra.run.smithi187:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-09-16T07:54:11.342 INFO:tasks.ceph:Reading keys from all nodes... 2024-09-16T07:54:11.342 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-09-16T07:54:11.342 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-09-16T07:54:11.374 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-09-16T07:54:11.374 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-09-16T07:54:11.443 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-09-16T07:54:11.443 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-09-16T07:54:11.468 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-09-16T07:54:11.468 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-09-16T07:54:11.533 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-09-16T07:54:11.533 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-09-16T07:54:11.597 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-09-16T07:54:11.597 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-09-16T07:54:11.616 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-09-16T07:54:11.616 DEBUG:teuthology.orchestra.run.smithi187:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-09-16T07:54:11.652 INFO:tasks.ceph:Adding keys to all mons... 2024-09-16T07:54:11.653 DEBUG:teuthology.orchestra.run.smithi027:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-16T07:54:11.684 INFO:teuthology.orchestra.run.smithi027.stdout:[osd.0] 2024-09-16T07:54:11.684 INFO:teuthology.orchestra.run.smithi027.stdout: key = AQAW5OdmdRezJhAAsfEaAX61ZmheRGJgujJpvQ== 2024-09-16T07:54:11.684 INFO:teuthology.orchestra.run.smithi027.stdout:[osd.1] 2024-09-16T07:54:11.684 INFO:teuthology.orchestra.run.smithi027.stdout: key = AQAZ5OdmrIe5LRAAl5HYKrUimsjxhvx3+dDm+A== 2024-09-16T07:54:11.684 INFO:teuthology.orchestra.run.smithi027.stdout:[mgr.x] 2024-09-16T07:54:11.684 INFO:teuthology.orchestra.run.smithi027.stdout: key = AQAT5OdmxDk2LRAAGs4HDkXfslN1M1OZtDp6xQ== 2024-09-16T07:54:11.684 INFO:teuthology.orchestra.run.smithi027.stdout:[osd.2] 2024-09-16T07:54:11.684 INFO:teuthology.orchestra.run.smithi027.stdout: key = AQAe5OdmYVh4GxAA5HITvgwhRUjqoy400iLcGA== 2024-09-16T07:54:11.684 INFO:teuthology.orchestra.run.smithi027.stdout:[osd.3] 2024-09-16T07:54:11.684 INFO:teuthology.orchestra.run.smithi027.stdout: key = AQAh5Odm7UzQGhAAm38YNkSku6+46eLdCcL77Q== 2024-09-16T07:54:11.685 INFO:teuthology.orchestra.run.smithi027.stdout:[client.0] 2024-09-16T07:54:11.685 INFO:teuthology.orchestra.run.smithi027.stdout: key = AQAT5Odm4zmDMBAAW8i4/yDlhqABjK1ZosCVGw== 2024-09-16T07:54:11.685 INFO:teuthology.orchestra.run.smithi027.stdout:[client.1] 2024-09-16T07:54:11.685 INFO:teuthology.orchestra.run.smithi027.stdout: key = AQAT5OdmVfBNNBAArKFVlSoLUM/4f+oFrC1bMg== 2024-09-16T07:54:11.686 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T07:54:11.797 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T07:54:11.862 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-16T07:54:11.924 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T07:54:11.988 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T07:54:12.051 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-16T07:54:12.116 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-16T07:54:12.183 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-09-16T07:54:12.183 DEBUG:teuthology.orchestra.run.smithi027:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-09-16T07:54:12.208 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-16T07:54:12.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:12.327+0000 7f56c822bc00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T07:54:12.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:12.327+0000 7f56c822bc00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T07:54:12.341 DEBUG:teuthology.orchestra.run.smithi027:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-09-16T07:54:12.366 DEBUG:teuthology.orchestra.run.smithi027:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-16T07:54:12.422 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-09-16T07:54:12.422 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-16T07:54:12.422 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-16T07:54:12.464 INFO:tasks.ceph.mon.a:Started 2024-09-16T07:54:12.465 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-09-16T07:54:12.465 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-09-16T07:54:12.465 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-09-16T07:54:12.468 INFO:tasks.ceph.mgr.x:Started 2024-09-16T07:54:12.468 DEBUG:tasks.ceph:set 0 configs 2024-09-16T07:54:12.468 INFO:tasks.ceph:Setting crush tunables to default 2024-09-16T07:54:12.468 DEBUG:teuthology.orchestra.run.smithi027:> sudo ceph --cluster ceph osd crush tunables default 2024-09-16T07:54:12.549 INFO:tasks.ceph.mon.a.smithi027.stderr:2024-09-16T07:54:12.550+0000 7f3ceb44bc00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T07:54:12.549 INFO:tasks.ceph.mon.a.smithi027.stderr:2024-09-16T07:54:12.550+0000 7f3ceb44bc00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T07:54:12.549 INFO:tasks.ceph.mon.a.smithi027.stderr:2024-09-16T07:54:12.551+0000 7f3ceb44bc00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T07:54:12.557 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-09-16T07:54:12.559+0000 7f5f4a401000 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T07:54:12.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:12.577+0000 7fedbc191640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T07:54:12.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:12.585+0000 7fedbc191640 1 Processor -- start 2024-09-16T07:54:12.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:12.585+0000 7fedbc191640 1 -- start start 2024-09-16T07:54:12.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:12.587+0000 7fedbc191640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fedb4138720 0x7fedb413ab40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:54:12.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:12.587+0000 7fedbc191640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fedb4133040 con 0x7fedb4132520 2024-09-16T07:54:12.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:12.587+0000 7fedbc191640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 -- 0x7fedb406f6d0 con 0x7fedb4138720 2024-09-16T07:54:12.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:12.587+0000 7fedb972c640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fedb4138720 0x7fedb413ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:54:12.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:12.587+0000 7fedb9f2d640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fedb4132520 0x7fedb4132930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:55898/0 (socket says 172.21.15.27:55898) 2024-09-16T07:54:12.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:12.587+0000 7fedb9f2d640 1 -- 172.21.15.27:0/3371433709 learned_addr learned my addr 172.21.15.27:0/3371433709 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-09-16T07:54:12.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:12.587+0000 7fedb972c640 1 -- 172.21.15.27:0/3371433709 >> v1:172.21.15.27:6789/0 conn(0x7fedb4132520 legacy=0x7fedb4132930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:54:12.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:12.587+0000 7fedb8f2b640 1 -- 172.21.15.27:0/3371433709 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 998926521 0 0) 0x7fedb0001110 con 0x7fedb4132520 2024-09-16T07:54:12.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:12.587+0000 7fedb972c640 1 -- 172.21.15.27:0/3371433709 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fedb406f3d0 con 0x7fedb4138720 2024-09-16T07:54:12.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:12.588+0000 7fedb972c640 1 --2- 172.21.15.27:0/3371433709 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fedb4138720 0x7fedb413ab40 crc :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db5e070886cc1bc server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:54:12.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:12.589+0000 7fedb8f2b640 1 -- 172.21.15.27:0/3371433709 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7feda4009d90 con 0x7fedb4138720 2024-09-16T07:54:12.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:12.589+0000 7fedb8f2b640 1 -- 172.21.15.27:0/3371433709 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7feda400e230 con 0x7fedb4138720 2024-09-16T07:54:12.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:12.589+0000 7fedb8f2b640 1 -- 172.21.15.27:0/3371433709 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7feda400e4d0 con 0x7fedb4138720 2024-09-16T07:54:12.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:12.589+0000 7fedbc191640 1 -- 172.21.15.27:0/3371433709 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fedb4138720 msgr2=0x7fedb413ab40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:54:12.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:12.589+0000 7fedbc191640 1 --2- 172.21.15.27:0/3371433709 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fedb4138720 0x7fedb413ab40 crc :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:54:12.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:12.589+0000 7fedbc191640 1 -- 172.21.15.27:0/3371433709 shutdown_connections 2024-09-16T07:54:12.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:12.589+0000 7fedbc191640 1 --2- 172.21.15.27:0/3371433709 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fedb4138720 0x7fedb413ab40 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:54:12.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:12.589+0000 7fedbc191640 1 -- 172.21.15.27:0/3371433709 >> 172.21.15.27:0/3371433709 conn(0x7fedb406e490 msgr2=0x7fedb406e8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:54:12.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:12.590+0000 7fedbc191640 1 -- 172.21.15.27:0/3371433709 shutdown_connections 2024-09-16T07:54:12.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:12.590+0000 7fedbc191640 1 -- 172.21.15.27:0/3371433709 wait complete. 2024-09-16T07:54:12.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:12.590+0000 7fedbc191640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T07:54:12.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:12.592+0000 7fedbc191640 1 Processor -- start 2024-09-16T07:54:12.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:12.592+0000 7fedbc191640 1 -- start start 2024-09-16T07:54:12.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:12.593+0000 7fedbc191640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fedb4132520 0x7fedb41c9fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:54:12.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:12.593+0000 7fedbc191640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fedb40a14c0 con 0x7fedb4132520 2024-09-16T07:54:12.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:12.593+0000 7fedb9f2d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fedb4132520 0x7fedb41c9fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:54:12.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:12.593+0000 7fedb9f2d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fedb4132520 0x7fedb41c9fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54338/0 (socket says 172.21.15.27:54338) 2024-09-16T07:54:12.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:12.593+0000 7fedb9f2d640 1 -- 172.21.15.27:0/3573179855 learned_addr learned my addr 172.21.15.27:0/3573179855 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-09-16T07:54:12.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:12.593+0000 7fedb9f2d640 1 -- 172.21.15.27:0/3573179855 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fedb40a0e20 con 0x7fedb4132520 2024-09-16T07:54:12.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:12.594+0000 7fedb9f2d640 1 --2- 172.21.15.27:0/3573179855 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fedb4132520 0x7fedb41c9fe0 crc :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:54:12.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:12.594+0000 7fedaa7fc640 1 -- 172.21.15.27:0/3573179855 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fedb0004b80 con 0x7fedb4132520 2024-09-16T07:54:12.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:12.594+0000 7fedaa7fc640 1 -- 172.21.15.27:0/3573179855 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7fedb0004ce0 con 0x7fedb4132520 2024-09-16T07:54:12.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:12.594+0000 7fedbc191640 1 -- 172.21.15.27:0/3573179855 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fedb411eb30 con 0x7fedb4132520 2024-09-16T07:54:12.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:12.594+0000 7fedaa7fc640 1 -- 172.21.15.27:0/3573179855 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fedb0007120 con 0x7fedb4132520 2024-09-16T07:54:12.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:12.594+0000 7fedbc191640 1 -- 172.21.15.27:0/3573179855 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fedb41ca520 con 0x7fedb4132520 2024-09-16T07:54:12.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:12.594+0000 7fedaa7fc640 1 -- 172.21.15.27:0/3573179855 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (crc 0 0 0) 0x7fedb000c520 con 0x7fedb4132520 2024-09-16T07:54:12.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:12.594+0000 7fedaa7fc640 1 -- 172.21.15.27:0/3573179855 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (crc 0 0 0) 0x7fedb000fa20 con 0x7fedb4132520 2024-09-16T07:54:12.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:12.595+0000 7fedbc191640 1 -- 172.21.15.27:0/3573179855 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fedb406a300 con 0x7fedb4132520 2024-09-16T07:54:12.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:12.597+0000 7fedaa7fc640 1 -- 172.21.15.27:0/3573179855 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (crc 0 0 0) 0x7fedb40a0e20 con 0x7fedb4132520 2024-09-16T07:54:12.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:12.663+0000 7fedbc191640 1 -- 172.21.15.27:0/3573179855 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7fedb40a0ba0 con 0x7fedb4132520 2024-09-16T07:54:12.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:12.664+0000 7fedaa7fc640 1 -- 172.21.15.27:0/3573179855 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) ==== 124+0+0 (crc 0 0 0) 0x7fedb4133040 con 0x7fedb4132520 2024-09-16T07:54:12.662 INFO:teuthology.orchestra.run.smithi027.stderr:adjusted tunables profile to default 2024-09-16T07:54:12.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:12.664+0000 7fedbc191640 1 -- 172.21.15.27:0/3573179855 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fedb4132520 msgr2=0x7fedb41c9fe0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:54:12.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:12.664+0000 7fedbc191640 1 --2- 172.21.15.27:0/3573179855 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fedb4132520 0x7fedb41c9fe0 crc :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:54:12.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:12.665+0000 7fedbc191640 1 -- 172.21.15.27:0/3573179855 shutdown_connections 2024-09-16T07:54:12.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:12.665+0000 7fedbc191640 1 --2- 172.21.15.27:0/3573179855 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fedb4132520 0x7fedb41c9fe0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:54:12.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:12.665+0000 7fedbc191640 1 -- 172.21.15.27:0/3573179855 >> 172.21.15.27:0/3573179855 conn(0x7fedb406e490 msgr2=0x7fedb40a5840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:54:12.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:12.665+0000 7fedbc191640 1 -- 172.21.15.27:0/3573179855 shutdown_connections 2024-09-16T07:54:12.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:12.665+0000 7fedbc191640 1 -- 172.21.15.27:0/3573179855 wait complete. 2024-09-16T07:54:12.673 INFO:tasks.ceph:check_enable_crimson: True 2024-09-16T07:54:12.673 INFO:tasks.ceph:check_enable_crimson: setting set-allow-crimson 2024-09-16T07:54:12.674 DEBUG:teuthology.orchestra.run.smithi027:> sudo ceph --cluster ceph osd set-allow-crimson --yes-i-really-mean-it 2024-09-16T07:54:12.763 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-09-16T07:54:12.764+0000 7f5f4a401000 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T07:54:12.763 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-09-16T07:54:12.765+0000 7f5f4a401000 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T07:54:12.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:12.780+0000 7ff34b0d1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T07:54:12.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:12.787+0000 7ff34b0d1640 1 Processor -- start 2024-09-16T07:54:12.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:12.787+0000 7ff34b0d1640 1 -- start start 2024-09-16T07:54:12.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:12.788+0000 7ff34b0d1640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ff344138720 0x7ff34413ab40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:54:12.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:12.788+0000 7ff34b0d1640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff344133040 con 0x7ff344132520 2024-09-16T07:54:12.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:12.788+0000 7ff34b0d1640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 -- 0x7ff34406f6d0 con 0x7ff344138720 2024-09-16T07:54:12.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:12.788+0000 7ff348e6d640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7ff344132520 0x7ff344132930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:55900/0 (socket says 172.21.15.27:55900) 2024-09-16T07:54:12.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:12.788+0000 7ff348e6d640 1 -- 172.21.15.27:0/1352791177 learned_addr learned my addr 172.21.15.27:0/1352791177 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-09-16T07:54:12.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:12.789+0000 7ff343fff640 1 --2- 172.21.15.27:0/1352791177 >> v2:172.21.15.27:3300/0 conn(0x7ff344138720 0x7ff34413ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:54:12.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:12.789+0000 7ff3437fe640 1 -- 172.21.15.27:0/1352791177 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4218482462 0 0) 0x7ff338001110 con 0x7ff344132520 2024-09-16T07:54:12.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:12.789+0000 7ff3437fe640 1 -- 172.21.15.27:0/1352791177 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff334003580 con 0x7ff344132520 2024-09-16T07:54:12.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:12.789+0000 7ff343fff640 1 -- 172.21.15.27:0/1352791177 >> v1:172.21.15.27:6789/0 conn(0x7ff344132520 legacy=0x7ff344132930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:54:12.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:12.789+0000 7ff343fff640 1 -- 172.21.15.27:0/1352791177 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff34406f3d0 con 0x7ff344138720 2024-09-16T07:54:12.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:12.789+0000 7ff343fff640 1 --2- 172.21.15.27:0/1352791177 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff344138720 0x7ff34413ab40 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4bf0aff69d9f553f server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:54:12.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:12.789+0000 7ff3437fe640 1 -- 172.21.15.27:0/1352791177 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7ff330009d90 con 0x7ff344138720 2024-09-16T07:54:12.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:12.789+0000 7ff3437fe640 1 -- 172.21.15.27:0/1352791177 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7ff33000e230 con 0x7ff344138720 2024-09-16T07:54:12.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:12.789+0000 7ff3437fe640 1 -- 172.21.15.27:0/1352791177 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7ff33000e4d0 con 0x7ff344138720 2024-09-16T07:54:12.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:12.790+0000 7ff34b0d1640 1 -- 172.21.15.27:0/1352791177 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff344138720 msgr2=0x7ff34413ab40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:54:12.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:12.790+0000 7ff34b0d1640 1 --2- 172.21.15.27:0/1352791177 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff344138720 0x7ff34413ab40 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:54:12.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:12.790+0000 7ff34b0d1640 1 -- 172.21.15.27:0/1352791177 shutdown_connections 2024-09-16T07:54:12.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:12.790+0000 7ff34b0d1640 1 --2- 172.21.15.27:0/1352791177 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff344138720 0x7ff34413ab40 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:54:12.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:12.790+0000 7ff34b0d1640 1 -- 172.21.15.27:0/1352791177 >> 172.21.15.27:0/1352791177 conn(0x7ff34406e490 msgr2=0x7ff34406e8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:54:12.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:12.790+0000 7ff34b0d1640 1 -- 172.21.15.27:0/1352791177 shutdown_connections 2024-09-16T07:54:12.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:12.790+0000 7ff34b0d1640 1 -- 172.21.15.27:0/1352791177 wait complete. 2024-09-16T07:54:12.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:12.790+0000 7ff34b0d1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T07:54:12.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:12.792+0000 7ff34b0d1640 1 Processor -- start 2024-09-16T07:54:12.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:12.792+0000 7ff34b0d1640 1 -- start start 2024-09-16T07:54:12.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:12.794+0000 7ff34b0d1640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff344132520 0x7ff344130c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:54:12.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:12.794+0000 7ff34b0d1640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7ff3441313d0 con 0x7ff344132520 2024-09-16T07:54:12.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:12.794+0000 7ff348e6d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff344132520 0x7ff344130c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:54:12.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:12.794+0000 7ff348e6d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff344132520 0x7ff344130c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54350/0 (socket says 172.21.15.27:54350) 2024-09-16T07:54:12.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:12.794+0000 7ff348e6d640 1 -- 172.21.15.27:0/74386107 learned_addr learned my addr 172.21.15.27:0/74386107 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-09-16T07:54:12.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:12.794+0000 7ff348e6d640 1 -- 172.21.15.27:0/74386107 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3440a07b0 con 0x7ff344132520 2024-09-16T07:54:12.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:12.794+0000 7ff348e6d640 1 --2- 172.21.15.27:0/74386107 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff344132520 0x7ff344130c70 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:54:12.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:12.794+0000 7ff3417fa640 1 -- 172.21.15.27:0/74386107 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7ff338005930 con 0x7ff344132520 2024-09-16T07:54:12.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:12.794+0000 7ff3417fa640 1 -- 172.21.15.27:0/74386107 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7ff338005a90 con 0x7ff344132520 2024-09-16T07:54:12.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:12.794+0000 7ff3417fa640 1 -- 172.21.15.27:0/74386107 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7ff338005d30 con 0x7ff344132520 2024-09-16T07:54:12.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:12.794+0000 7ff34b0d1640 1 -- 172.21.15.27:0/74386107 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff344120bc0 con 0x7ff344132520 2024-09-16T07:54:12.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:12.794+0000 7ff34b0d1640 1 -- 172.21.15.27:0/74386107 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff344120030 con 0x7ff344132520 2024-09-16T07:54:12.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:12.795+0000 7ff3417fa640 1 -- 172.21.15.27:0/74386107 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (crc 0 0 0) 0x7ff33800cb90 con 0x7ff344132520 2024-09-16T07:54:12.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:12.795+0000 7ff3417fa640 1 -- 172.21.15.27:0/74386107 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 725+0+0 (crc 0 0 0) 0x7ff33800f8e0 con 0x7ff344132520 2024-09-16T07:54:12.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:12.795+0000 7ff34b0d1640 1 -- 172.21.15.27:0/74386107 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff34406a300 con 0x7ff344132520 2024-09-16T07:54:12.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:12.798+0000 7ff3417fa640 1 -- 172.21.15.27:0/74386107 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (crc 0 0 0) 0x7ff344133040 con 0x7ff344132520 2024-09-16T07:54:12.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:12.873+0000 7ff34b0d1640 1 -- 172.21.15.27:0/74386107 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd set-allow-crimson", "yes_i_really_mean_it": true} v 0) -- 0x7ff344132930 con 0x7ff344132520 2024-09-16T07:54:12.871 INFO:tasks.ceph.mon.a.smithi027.stderr:2024-09-16T07:54:12.873+0000 7f3ce1df4640 -1 WARNING: experimental feature 'crimson' is enabled 2024-09-16T07:54:12.871 INFO:tasks.ceph.mon.a.smithi027.stderr:Please be aware that this feature is experimental, untested, 2024-09-16T07:54:12.871 INFO:tasks.ceph.mon.a.smithi027.stderr:unsupported, and may result in data corruption, data loss, 2024-09-16T07:54:12.872 INFO:tasks.ceph.mon.a.smithi027.stderr:and/or irreparable damage to your cluster. Do not use 2024-09-16T07:54:12.872 INFO:tasks.ceph.mon.a.smithi027.stderr:feature with important data. 2024-09-16T07:54:12.872 INFO:tasks.ceph.mon.a.smithi027.stderr: 2024-09-16T07:54:13.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:13.664+0000 7ff3417fa640 1 -- 172.21.15.27:0/74386107 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd set-allow-crimson", "yes_i_really_mean_it": true}]=0 v3) ==== 99+0+0 (crc 0 0 0) 0x7ff33800fb40 con 0x7ff344132520 2024-09-16T07:54:13.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:13.666+0000 7ff34b0d1640 1 -- 172.21.15.27:0/74386107 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff344132520 msgr2=0x7ff344130c70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:54:13.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:13.666+0000 7ff34b0d1640 1 --2- 172.21.15.27:0/74386107 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff344132520 0x7ff344130c70 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:54:13.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:13.666+0000 7ff34b0d1640 1 -- 172.21.15.27:0/74386107 shutdown_connections 2024-09-16T07:54:13.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:13.666+0000 7ff34b0d1640 1 --2- 172.21.15.27:0/74386107 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff344132520 0x7ff344130c70 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:54:13.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:13.666+0000 7ff34b0d1640 1 -- 172.21.15.27:0/74386107 >> 172.21.15.27:0/74386107 conn(0x7ff34406e490 msgr2=0x7ff3440a3190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:54:13.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:13.666+0000 7ff34b0d1640 1 -- 172.21.15.27:0/74386107 shutdown_connections 2024-09-16T07:54:13.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:13.666+0000 7ff34b0d1640 1 -- 172.21.15.27:0/74386107 wait complete. 2024-09-16T07:54:13.677 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-09-16T07:54:13.677 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-09-16T07:54:13.677 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-09-16T07:54:13.705 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-09-16T07:54:13.705 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-09-16T07:54:13.773 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-09-16T07:54:13.773 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-09-16T07:54:13.798 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-09-16T07:54:13.798 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-09-16T07:54:13.863 DEBUG:teuthology.orchestra.run.smithi187:> sudo ceph --cluster ceph osd new 9a91e4ff-5797-4c44-a7e4-c088c97e1063 0 2024-09-16T07:54:14.012 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.008+0000 7f3893d91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T07:54:14.013 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.014+0000 7f3893d91640 1 Processor -- start 2024-09-16T07:54:14.013 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.014+0000 7f3893d91640 1 -- start start 2024-09-16T07:54:14.013 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.015+0000 7f3893d91640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f388c117020 0x7f388c117430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:54:14.013 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.015+0000 7f3893d91640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f388c112c30 con 0x7f388c112110 2024-09-16T07:54:14.014 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.015+0000 7f3893d91640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 -- 0x7f388c112dc0 con 0x7f388c117020 2024-09-16T07:54:14.014 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.015+0000 7f3891b2d640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f388c117020 0x7f388c117430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:54:14.014 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.015+0000 7f3891b2d640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f388c117020 0x7f388c117430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.187:48420/0 (socket says 172.21.15.187:48420) 2024-09-16T07:54:14.014 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.015+0000 7f3891b2d640 1 -- 172.21.15.187:0/4036844725 learned_addr learned my addr 172.21.15.187:0/4036844725 (peer_addr_for_me v2:172.21.15.187:0/0) 2024-09-16T07:54:14.014 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.015+0000 7f389132c640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f388c112110 0x7f388c112520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.187:50858/0 (socket says 172.21.15.187:50858) 2024-09-16T07:54:14.018 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.015+0000 7f3891b2d640 1 -- 172.21.15.187:0/4036844725 >> v1:172.21.15.27:6789/0 conn(0x7f388c112110 legacy=0x7f388c112520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:54:14.018 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.015+0000 7f3891b2d640 1 -- 172.21.15.187:0/4036844725 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f388c112f30 con 0x7f388c117020 2024-09-16T07:54:14.018 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.016+0000 7f3891b2d640 1 --2- 172.21.15.187:0/4036844725 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f388c117020 0x7f388c117430 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eab699e6498cf160 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:54:14.018 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.016+0000 7f3890b2b640 1 -- 172.21.15.187:0/4036844725 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f387c009e50 con 0x7f388c117020 2024-09-16T07:54:14.018 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.016+0000 7f3890b2b640 1 -- 172.21.15.187:0/4036844725 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f387c00e230 con 0x7f388c117020 2024-09-16T07:54:14.018 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.016+0000 7f3890b2b640 1 -- 172.21.15.187:0/4036844725 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f387c00e4d0 con 0x7f388c117020 2024-09-16T07:54:14.018 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.017+0000 7f3893d91640 1 -- 172.21.15.187:0/4036844725 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f388c117020 msgr2=0x7f388c117430 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:54:14.018 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.017+0000 7f3893d91640 1 --2- 172.21.15.187:0/4036844725 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f388c117020 0x7f388c117430 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:54:14.019 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.017+0000 7f3893d91640 1 -- 172.21.15.187:0/4036844725 shutdown_connections 2024-09-16T07:54:14.019 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.017+0000 7f3893d91640 1 --2- 172.21.15.187:0/4036844725 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f388c117020 0x7f388c117430 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:54:14.019 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.017+0000 7f3893d91640 1 -- 172.21.15.187:0/4036844725 >> 172.21.15.187:0/4036844725 conn(0x7f388c0726d0 msgr2=0x7f388c074b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:54:14.019 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.017+0000 7f3893d91640 1 -- 172.21.15.187:0/4036844725 shutdown_connections 2024-09-16T07:54:14.019 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.017+0000 7f3893d91640 1 -- 172.21.15.187:0/4036844725 wait complete. 2024-09-16T07:54:14.019 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.017+0000 7f3893d91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T07:54:14.019 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.018+0000 7f3893d91640 1 Processor -- start 2024-09-16T07:54:14.019 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.018+0000 7f3893d91640 1 -- start start 2024-09-16T07:54:14.019 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.018+0000 7f3893d91640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f388c112110 0x7f388c1a6f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:54:14.019 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.018+0000 7f3893d91640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f388c1a7550 con 0x7f388c112110 2024-09-16T07:54:14.019 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.018+0000 7f3891b2d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f388c112110 0x7f388c1a6f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:54:14.019 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.018+0000 7f3891b2d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f388c112110 0x7f388c1a6f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.187:48430/0 (socket says 172.21.15.187:48430) 2024-09-16T07:54:14.020 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.018+0000 7f3891b2d640 1 -- 172.21.15.187:0/299999940 learned_addr learned my addr 172.21.15.187:0/299999940 (peer_addr_for_me v2:172.21.15.187:0/0) 2024-09-16T07:54:14.020 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.019+0000 7f3891b2d640 1 -- 172.21.15.187:0/299999940 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f388c1a7690 con 0x7f388c112110 2024-09-16T07:54:14.020 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.019+0000 7f3891b2d640 1 --2- 172.21.15.187:0/299999940 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f388c112110 0x7f388c1a6f80 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:54:14.020 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.019+0000 7f38827fc640 1 -- 172.21.15.187:0/299999940 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f387c00eea0 con 0x7f388c112110 2024-09-16T07:54:14.020 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.019+0000 7f38827fc640 1 -- 172.21.15.187:0/299999940 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f387c015420 con 0x7f388c112110 2024-09-16T07:54:14.020 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.019+0000 7f3893d91640 1 -- 172.21.15.187:0/299999940 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f388c1a78e0 con 0x7f388c112110 2024-09-16T07:54:14.020 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.019+0000 7f38827fc640 1 -- 172.21.15.187:0/299999940 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f387c0156c0 con 0x7f388c112110 2024-09-16T07:54:14.020 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.019+0000 7f3893d91640 1 -- 172.21.15.187:0/299999940 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f388c1ac7e0 con 0x7f388c112110 2024-09-16T07:54:14.020 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.019+0000 7f38827fc640 1 -- 172.21.15.187:0/299999940 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (crc 0 0 0) 0x7f387c015c50 con 0x7f388c112110 2024-09-16T07:54:14.020 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.019+0000 7f38827fc640 1 -- 172.21.15.187:0/299999940 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 747+0+0 (crc 0 0 0) 0x7f387c0116d0 con 0x7f388c112110 2024-09-16T07:54:14.021 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.020+0000 7f3893d91640 1 -- 172.21.15.187:0/299999940 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f388c11e220 con 0x7f388c112110 2024-09-16T07:54:14.021 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.023+0000 7f38827fc640 1 -- 172.21.15.187:0/299999940 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (crc 0 0 0) 0x7f387c011890 con 0x7f388c112110 2024-09-16T07:54:14.109 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.110+0000 7f3893d91640 1 -- 172.21.15.187:0/299999940 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd new", "uuid": "9a91e4ff-5797-4c44-a7e4-c088c97e1063", "id": 0} v 0) -- 0x7f388c11e400 con 0x7f388c112110 2024-09-16T07:54:14.110 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.112+0000 7f38827fc640 1 -- 172.21.15.187:0/299999940 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "9a91e4ff-5797-4c44-a7e4-c088c97e1063", "id": 0}]=0 v4) ==== 112+0+2 (crc 0 0 0) 0x7f387c011ab0 con 0x7f388c112110 2024-09-16T07:54:14.110 INFO:teuthology.orchestra.run.smithi187.stdout:0 2024-09-16T07:54:14.111 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.113+0000 7f3893d91640 1 -- 172.21.15.187:0/299999940 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f388c112110 msgr2=0x7f388c1a6f80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:54:14.112 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.113+0000 7f3893d91640 1 --2- 172.21.15.187:0/299999940 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f388c112110 0x7f388c1a6f80 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:54:14.112 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.113+0000 7f3893d91640 1 -- 172.21.15.187:0/299999940 shutdown_connections 2024-09-16T07:54:14.112 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.113+0000 7f3893d91640 1 --2- 172.21.15.187:0/299999940 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f388c112110 0x7f388c1a6f80 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:54:14.112 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.113+0000 7f3893d91640 1 -- 172.21.15.187:0/299999940 >> 172.21.15.187:0/299999940 conn(0x7f388c0726d0 msgr2=0x7f388c115bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:54:14.112 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.113+0000 7f3893d91640 1 -- 172.21.15.187:0/299999940 shutdown_connections 2024-09-16T07:54:14.112 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.113+0000 7f3893d91640 1 -- 172.21.15.187:0/299999940 wait complete. 2024-09-16T07:54:14.122 DEBUG:teuthology.orchestra.run.smithi187:> sudo ceph --cluster ceph osd new a1b27d22-cc01-4cc9-88c0-c28e8c60a99e 1 2024-09-16T07:54:14.230 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.225+0000 7f7d36951640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T07:54:14.230 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.232+0000 7f7d36951640 1 Processor -- start 2024-09-16T07:54:14.231 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.232+0000 7f7d36951640 1 -- start start 2024-09-16T07:54:14.232 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.233+0000 7f7d36951640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f7d3011d110 0x7f7d3011f530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:54:14.232 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.233+0000 7f7d36951640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7d3011fb00 con 0x7f7d301191d0 2024-09-16T07:54:14.232 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.233+0000 7f7d36951640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 -- 0x7f7d30120cc0 con 0x7f7d3011d110 2024-09-16T07:54:14.232 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.233+0000 7f7d2f7fe640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f7d3011d110 0x7f7d3011f530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:54:14.232 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.233+0000 7f7d2f7fe640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f7d3011d110 0x7f7d3011f530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.187:48442/0 (socket says 172.21.15.187:48442) 2024-09-16T07:54:14.232 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.233+0000 7f7d2f7fe640 1 -- 172.21.15.187:0/2299269358 learned_addr learned my addr 172.21.15.187:0/2299269358 (peer_addr_for_me v2:172.21.15.187:0/0) 2024-09-16T07:54:14.232 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.233+0000 7f7d2f7fe640 1 -- 172.21.15.187:0/2299269358 >> v1:172.21.15.27:6789/0 conn(0x7f7d301191d0 legacy=0x7f7d301195e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:54:14.233 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.233+0000 7f7d2f7fe640 1 -- 172.21.15.187:0/2299269358 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d30120e30 con 0x7f7d3011d110 2024-09-16T07:54:14.233 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.234+0000 7f7d2f7fe640 1 --2- 172.21.15.187:0/2299269358 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7d3011d110 0x7f7d3011f530 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7834e10f4e75c087 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:54:14.233 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.234+0000 7f7d2effd640 1 -- 172.21.15.187:0/2299269358 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f7d20009e50 con 0x7f7d3011d110 2024-09-16T07:54:14.233 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.234+0000 7f7d2effd640 1 -- 172.21.15.187:0/2299269358 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f7d2000e230 con 0x7f7d3011d110 2024-09-16T07:54:14.233 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.234+0000 7f7d2effd640 1 -- 172.21.15.187:0/2299269358 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f7d2000e4d0 con 0x7f7d3011d110 2024-09-16T07:54:14.234 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.235+0000 7f7d36951640 1 -- 172.21.15.187:0/2299269358 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7d3011d110 msgr2=0x7f7d3011f530 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:54:14.234 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.235+0000 7f7d36951640 1 --2- 172.21.15.187:0/2299269358 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7d3011d110 0x7f7d3011f530 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:54:14.234 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.235+0000 7f7d36951640 1 -- 172.21.15.187:0/2299269358 shutdown_connections 2024-09-16T07:54:14.235 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.235+0000 7f7d36951640 1 --2- 172.21.15.187:0/2299269358 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7d3011d110 0x7f7d3011f530 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:54:14.235 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.235+0000 7f7d36951640 1 -- 172.21.15.187:0/2299269358 >> 172.21.15.187:0/2299269358 conn(0x7f7d3006ff60 msgr2=0x7f7d30112db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:54:14.235 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.235+0000 7f7d36951640 1 -- 172.21.15.187:0/2299269358 shutdown_connections 2024-09-16T07:54:14.235 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.235+0000 7f7d36951640 1 -- 172.21.15.187:0/2299269358 wait complete. 2024-09-16T07:54:14.235 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.235+0000 7f7d36951640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T07:54:14.235 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.236+0000 7f7d36951640 1 Processor -- start 2024-09-16T07:54:14.235 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.236+0000 7f7d36951640 1 -- start start 2024-09-16T07:54:14.235 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.236+0000 7f7d36951640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7d301191d0 0x7f7d301a91c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:54:14.235 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.236+0000 7f7d36951640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f7d301a9790 con 0x7f7d301191d0 2024-09-16T07:54:14.237 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.236+0000 7f7d2ffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7d301191d0 0x7f7d301a91c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:54:14.237 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.236+0000 7f7d2ffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7d301191d0 0x7f7d301a91c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.187:48454/0 (socket says 172.21.15.187:48454) 2024-09-16T07:54:14.237 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.237+0000 7f7d2ffff640 1 -- 172.21.15.187:0/2855745650 learned_addr learned my addr 172.21.15.187:0/2855745650 (peer_addr_for_me v2:172.21.15.187:0/0) 2024-09-16T07:54:14.237 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.237+0000 7f7d2ffff640 1 -- 172.21.15.187:0/2855745650 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d301a98d0 con 0x7f7d301191d0 2024-09-16T07:54:14.237 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.237+0000 7f7d2ffff640 1 --2- 172.21.15.187:0/2855745650 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7d301191d0 0x7f7d301a91c0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:54:14.237 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.237+0000 7f7d2cff9640 1 -- 172.21.15.187:0/2855745650 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f7d1800c950 con 0x7f7d301191d0 2024-09-16T07:54:14.237 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.237+0000 7f7d2cff9640 1 -- 172.21.15.187:0/2855745650 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f7d1800cab0 con 0x7f7d301191d0 2024-09-16T07:54:14.237 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.237+0000 7f7d36951640 1 -- 172.21.15.187:0/2855745650 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d301a9b80 con 0x7f7d301191d0 2024-09-16T07:54:14.237 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.237+0000 7f7d2cff9640 1 -- 172.21.15.187:0/2855745650 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f7d1800cd50 con 0x7f7d301191d0 2024-09-16T07:54:14.237 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.237+0000 7f7d36951640 1 -- 172.21.15.187:0/2855745650 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d301aea20 con 0x7f7d301191d0 2024-09-16T07:54:14.238 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.238+0000 7f7d2cff9640 1 -- 172.21.15.187:0/2855745650 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (crc 0 0 0) 0x7f7d180116a0 con 0x7f7d301191d0 2024-09-16T07:54:14.238 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.238+0000 7f7d2cff9640 1 -- 172.21.15.187:0/2855745650 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 862+0+0 (crc 0 0 0) 0x7f7d1800f230 con 0x7f7d301191d0 2024-09-16T07:54:14.238 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.239+0000 7f7d36951640 1 -- 172.21.15.187:0/2855745650 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7d30120750 con 0x7f7d301191d0 2024-09-16T07:54:14.239 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.241+0000 7f7d2cff9640 1 -- 172.21.15.187:0/2855745650 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (crc 0 0 0) 0x7f7d18010e80 con 0x7f7d301191d0 2024-09-16T07:54:14.328 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.330+0000 7f7d36951640 1 -- 172.21.15.187:0/2855745650 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd new", "uuid": "a1b27d22-cc01-4cc9-88c0-c28e8c60a99e", "id": 1} v 0) -- 0x7f7d3011d0a0 con 0x7f7d301191d0 2024-09-16T07:54:14.329 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.331+0000 7f7d2cff9640 1 -- 172.21.15.187:0/2855745650 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "a1b27d22-cc01-4cc9-88c0-c28e8c60a99e", "id": 1}]=0 v5) ==== 112+0+2 (crc 0 0 0) 0x7f7d18011c80 con 0x7f7d301191d0 2024-09-16T07:54:14.330 INFO:teuthology.orchestra.run.smithi187.stdout:1 2024-09-16T07:54:14.331 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.332+0000 7f7d36951640 1 -- 172.21.15.187:0/2855745650 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7d301191d0 msgr2=0x7f7d301a91c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:54:14.331 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.332+0000 7f7d36951640 1 --2- 172.21.15.187:0/2855745650 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7d301191d0 0x7f7d301a91c0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:54:14.331 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.332+0000 7f7d36951640 1 -- 172.21.15.187:0/2855745650 shutdown_connections 2024-09-16T07:54:14.331 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.332+0000 7f7d36951640 1 --2- 172.21.15.187:0/2855745650 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7d301191d0 0x7f7d301a91c0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:54:14.331 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.332+0000 7f7d36951640 1 -- 172.21.15.187:0/2855745650 >> 172.21.15.187:0/2855745650 conn(0x7f7d3006ff60 msgr2=0x7f7d3011ea80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:54:14.331 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.332+0000 7f7d36951640 1 -- 172.21.15.187:0/2855745650 shutdown_connections 2024-09-16T07:54:14.331 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.332+0000 7f7d36951640 1 -- 172.21.15.187:0/2855745650 wait complete. 2024-09-16T07:54:14.342 DEBUG:teuthology.orchestra.run.smithi187:> sudo ceph --cluster ceph osd new 04f4b772-d2c6-4d73-894b-d5635c4d2dda 2 2024-09-16T07:54:14.450 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.445+0000 7fbdfcd51640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T07:54:14.451 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.452+0000 7fbdfcd51640 1 Processor -- start 2024-09-16T07:54:14.452 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.452+0000 7fbdfcd51640 1 -- start start 2024-09-16T07:54:14.452 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.453+0000 7fbdfcd51640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fbdf81191d0 0x7fbdf81195e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:54:14.452 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.453+0000 7fbdfcd51640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbdf8114ff0 con 0x7fbdf81144d0 2024-09-16T07:54:14.452 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.453+0000 7fbdfcd51640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 -- 0x7fbdf8115180 con 0x7fbdf81191d0 2024-09-16T07:54:14.452 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.453+0000 7fbdf659c640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fbdf81191d0 0x7fbdf81195e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:54:14.452 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.453+0000 7fbdf659c640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fbdf81191d0 0x7fbdf81195e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.187:48460/0 (socket says 172.21.15.187:48460) 2024-09-16T07:54:14.452 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.453+0000 7fbdf659c640 1 -- 172.21.15.187:0/309777014 learned_addr learned my addr 172.21.15.187:0/309777014 (peer_addr_for_me v2:172.21.15.187:0/0) 2024-09-16T07:54:14.452 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.453+0000 7fbdeffff640 1 --1- 172.21.15.187:0/309777014 >> v1:172.21.15.27:6789/0 conn(0x7fbdf81144d0 0x7fbdf81148e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.187:50878/0 (socket says 172.21.15.187:50878) 2024-09-16T07:54:14.452 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.453+0000 7fbdf659c640 1 -- 172.21.15.187:0/309777014 >> v1:172.21.15.27:6789/0 conn(0x7fbdf81144d0 legacy=0x7fbdf81148e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:54:14.452 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.453+0000 7fbdf659c640 1 -- 172.21.15.187:0/309777014 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbdf81152f0 con 0x7fbdf81191d0 2024-09-16T07:54:14.454 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.454+0000 7fbdf659c640 1 --2- 172.21.15.187:0/309777014 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbdf81191d0 0x7fbdf81195e0 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f7394ce3c8f111b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:54:14.454 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.454+0000 7fbdf5d9b640 1 -- 172.21.15.187:0/309777014 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fbde8009e50 con 0x7fbdf81191d0 2024-09-16T07:54:14.454 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.454+0000 7fbdf5d9b640 1 -- 172.21.15.187:0/309777014 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7fbde800e230 con 0x7fbdf81191d0 2024-09-16T07:54:14.454 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.454+0000 7fbdf5d9b640 1 -- 172.21.15.187:0/309777014 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fbde800e4d0 con 0x7fbdf81191d0 2024-09-16T07:54:14.454 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.455+0000 7fbdfcd51640 1 -- 172.21.15.187:0/309777014 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbdf81191d0 msgr2=0x7fbdf81195e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:54:14.454 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.455+0000 7fbdfcd51640 1 --2- 172.21.15.187:0/309777014 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbdf81191d0 0x7fbdf81195e0 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:54:14.454 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.455+0000 7fbdfcd51640 1 -- 172.21.15.187:0/309777014 shutdown_connections 2024-09-16T07:54:14.454 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.455+0000 7fbdfcd51640 1 --2- 172.21.15.187:0/309777014 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbdf81191d0 0x7fbdf81195e0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:54:14.454 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.455+0000 7fbdfcd51640 1 -- 172.21.15.187:0/309777014 >> 172.21.15.187:0/309777014 conn(0x7fbdf806ff60 msgr2=0x7fbdf8112db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:54:14.454 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.455+0000 7fbdfcd51640 1 -- 172.21.15.187:0/309777014 shutdown_connections 2024-09-16T07:54:14.454 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.455+0000 7fbdfcd51640 1 -- 172.21.15.187:0/309777014 wait complete. 2024-09-16T07:54:14.454 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.455+0000 7fbdfcd51640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T07:54:14.457 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.456+0000 7fbdfcd51640 1 Processor -- start 2024-09-16T07:54:14.457 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.456+0000 7fbdfcd51640 1 -- start start 2024-09-16T07:54:14.457 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.456+0000 7fbdfcd51640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbdf81144d0 0x7fbdf81a90d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:54:14.457 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.456+0000 7fbdfcd51640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fbdf81a96a0 con 0x7fbdf81144d0 2024-09-16T07:54:14.457 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.456+0000 7fbdf659c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbdf81144d0 0x7fbdf81a90d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:54:14.458 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.457+0000 7fbdf659c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbdf81144d0 0x7fbdf81a90d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.187:48464/0 (socket says 172.21.15.187:48464) 2024-09-16T07:54:14.458 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.457+0000 7fbdf659c640 1 -- 172.21.15.187:0/4114647603 learned_addr learned my addr 172.21.15.187:0/4114647603 (peer_addr_for_me v2:172.21.15.187:0/0) 2024-09-16T07:54:14.458 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.457+0000 7fbdf659c640 1 -- 172.21.15.187:0/4114647603 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbdf81a97e0 con 0x7fbdf81144d0 2024-09-16T07:54:14.459 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.457+0000 7fbdf659c640 1 --2- 172.21.15.187:0/4114647603 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbdf81144d0 0x7fbdf81a90d0 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:54:14.459 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.457+0000 7fbdeeffd640 1 -- 172.21.15.187:0/4114647603 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fbde800eea0 con 0x7fbdf81144d0 2024-09-16T07:54:14.459 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.457+0000 7fbdeeffd640 1 -- 172.21.15.187:0/4114647603 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7fbde8015420 con 0x7fbdf81144d0 2024-09-16T07:54:14.459 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.457+0000 7fbdfcd51640 1 -- 172.21.15.187:0/4114647603 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbdf81a9a30 con 0x7fbdf81144d0 2024-09-16T07:54:14.459 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.457+0000 7fbdeeffd640 1 -- 172.21.15.187:0/4114647603 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fbde80156c0 con 0x7fbdf81144d0 2024-09-16T07:54:14.459 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.457+0000 7fbdfcd51640 1 -- 172.21.15.187:0/4114647603 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbdf81ae9b0 con 0x7fbdf81144d0 2024-09-16T07:54:14.459 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.458+0000 7fbdeeffd640 1 -- 172.21.15.187:0/4114647603 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (crc 0 0 0) 0x7fbde8015c00 con 0x7fbdf81144d0 2024-09-16T07:54:14.459 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.458+0000 7fbdeeffd640 1 -- 172.21.15.187:0/4114647603 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 977+0+0 (crc 0 0 0) 0x7fbde80147f0 con 0x7fbdf81144d0 2024-09-16T07:54:14.459 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.458+0000 7fbdfcd51640 1 -- 172.21.15.187:0/4114647603 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbdf81203d0 con 0x7fbdf81144d0 2024-09-16T07:54:14.459 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.461+0000 7fbdeeffd640 1 -- 172.21.15.187:0/4114647603 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (crc 0 0 0) 0x7fbde80149b0 con 0x7fbdf81144d0 2024-09-16T07:54:14.530 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.532+0000 7fbdfcd51640 1 -- 172.21.15.187:0/4114647603 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd new", "uuid": "04f4b772-d2c6-4d73-894b-d5635c4d2dda", "id": 2} v 0) -- 0x7fbdf8120560 con 0x7fbdf81144d0 2024-09-16T07:54:14.534 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.536+0000 7fbdeeffd640 1 -- 172.21.15.187:0/4114647603 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "04f4b772-d2c6-4d73-894b-d5635c4d2dda", "id": 2}]=0 v6) ==== 112+0+2 (crc 0 0 0) 0x7fbde80143b0 con 0x7fbdf81144d0 2024-09-16T07:54:14.535 INFO:teuthology.orchestra.run.smithi187.stdout:2 2024-09-16T07:54:14.536 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.537+0000 7fbdfcd51640 1 -- 172.21.15.187:0/4114647603 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbdf81144d0 msgr2=0x7fbdf81a90d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:54:14.536 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.537+0000 7fbdfcd51640 1 --2- 172.21.15.187:0/4114647603 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbdf81144d0 0x7fbdf81a90d0 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:54:14.536 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.537+0000 7fbdfcd51640 1 -- 172.21.15.187:0/4114647603 shutdown_connections 2024-09-16T07:54:14.536 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.537+0000 7fbdfcd51640 1 --2- 172.21.15.187:0/4114647603 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbdf81144d0 0x7fbdf81a90d0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:54:14.536 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.537+0000 7fbdfcd51640 1 -- 172.21.15.187:0/4114647603 >> 172.21.15.187:0/4114647603 conn(0x7fbdf806ff60 msgr2=0x7fbdf8117960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:54:14.536 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.537+0000 7fbdfcd51640 1 -- 172.21.15.187:0/4114647603 shutdown_connections 2024-09-16T07:54:14.536 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.537+0000 7fbdfcd51640 1 -- 172.21.15.187:0/4114647603 wait complete. 2024-09-16T07:54:14.546 DEBUG:teuthology.orchestra.run.smithi187:> sudo ceph --cluster ceph osd new 611aad1c-1d66-400c-a54d-b66c9579b6d9 3 2024-09-16T07:54:14.653 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.649+0000 7f26c2d91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T07:54:14.654 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.656+0000 7f26c2d91640 1 Processor -- start 2024-09-16T07:54:14.654 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.656+0000 7f26c2d91640 1 -- start start 2024-09-16T07:54:14.655 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.656+0000 7f26c2d91640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f26bc1191d0 0x7f26bc1195e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:54:14.655 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.656+0000 7f26c2d91640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f26bc114ff0 con 0x7f26bc1144d0 2024-09-16T07:54:14.655 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.656+0000 7f26c2d91640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 -- 0x7f26bc115180 con 0x7f26bc1191d0 2024-09-16T07:54:14.655 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.657+0000 7f26c0b2d640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f26bc1191d0 0x7f26bc1195e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:54:14.655 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.657+0000 7f26bbfff640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f26bc1144d0 0x7f26bc1148e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.187:50888/0 (socket says 172.21.15.187:50888) 2024-09-16T07:54:14.655 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.657+0000 7f26c0b2d640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f26bc1191d0 0x7f26bc1195e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.187:48476/0 (socket says 172.21.15.187:48476) 2024-09-16T07:54:14.656 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.657+0000 7f26bbfff640 1 -- 172.21.15.187:0/3015107867 learned_addr learned my addr 172.21.15.187:0/3015107867 (peer_addr_for_me v1:172.21.15.187:0/0) 2024-09-16T07:54:14.657 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.657+0000 7f26c0b2d640 1 -- 172.21.15.187:0/3015107867 >> v1:172.21.15.27:6789/0 conn(0x7f26bc1144d0 legacy=0x7f26bc1148e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:54:14.657 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.657+0000 7f26c0b2d640 1 -- 172.21.15.187:0/3015107867 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26bc1152f0 con 0x7f26bc1191d0 2024-09-16T07:54:14.657 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.658+0000 7f26c0b2d640 1 --2- 172.21.15.187:0/3015107867 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f26bc1191d0 0x7f26bc1195e0 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dffcdc3c23ce153c server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:54:14.657 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.658+0000 7f26bb7fe640 1 -- 172.21.15.187:0/3015107867 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f26b0009d90 con 0x7f26bc1191d0 2024-09-16T07:54:14.657 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.658+0000 7f26bb7fe640 1 -- 172.21.15.187:0/3015107867 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f26b000e230 con 0x7f26bc1191d0 2024-09-16T07:54:14.658 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.658+0000 7f26bb7fe640 1 -- 172.21.15.187:0/3015107867 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f26b000e4d0 con 0x7f26bc1191d0 2024-09-16T07:54:14.659 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.659+0000 7f26c2d91640 1 -- 172.21.15.187:0/3015107867 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f26bc1191d0 msgr2=0x7f26bc1195e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:54:14.659 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.659+0000 7f26c2d91640 1 --2- 172.21.15.187:0/3015107867 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f26bc1191d0 0x7f26bc1195e0 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:54:14.659 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.659+0000 7f26c2d91640 1 -- 172.21.15.187:0/3015107867 shutdown_connections 2024-09-16T07:54:14.659 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.659+0000 7f26c2d91640 1 --2- 172.21.15.187:0/3015107867 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f26bc1191d0 0x7f26bc1195e0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:54:14.659 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.659+0000 7f26c2d91640 1 -- 172.21.15.187:0/3015107867 >> 172.21.15.187:0/3015107867 conn(0x7f26bc06ff60 msgr2=0x7f26bc112db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:54:14.659 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.659+0000 7f26c2d91640 1 -- 172.21.15.187:0/3015107867 shutdown_connections 2024-09-16T07:54:14.659 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.659+0000 7f26c2d91640 1 -- 172.21.15.187:0/3015107867 wait complete. 2024-09-16T07:54:14.659 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.659+0000 7f26c2d91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T07:54:14.661 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.660+0000 7f26c2d91640 1 Processor -- start 2024-09-16T07:54:14.661 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.660+0000 7f26c2d91640 1 -- start start 2024-09-16T07:54:14.661 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.660+0000 7f26c2d91640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f26bc1144d0 0x7f26bc1a90d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:54:14.661 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.660+0000 7f26c2d91640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f26bc1a96a0 con 0x7f26bc1144d0 2024-09-16T07:54:14.661 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.660+0000 7f26c0b2d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f26bc1144d0 0x7f26bc1a90d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:54:14.662 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.660+0000 7f26c0b2d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f26bc1144d0 0x7f26bc1a90d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.187:48482/0 (socket says 172.21.15.187:48482) 2024-09-16T07:54:14.662 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.660+0000 7f26c0b2d640 1 -- 172.21.15.187:0/2742363123 learned_addr learned my addr 172.21.15.187:0/2742363123 (peer_addr_for_me v2:172.21.15.187:0/0) 2024-09-16T07:54:14.662 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.661+0000 7f26c0b2d640 1 -- 172.21.15.187:0/2742363123 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26bc1a97e0 con 0x7f26bc1144d0 2024-09-16T07:54:14.662 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.661+0000 7f26c0b2d640 1 --2- 172.21.15.187:0/2742363123 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f26bc1144d0 0x7f26bc1a90d0 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:54:14.662 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.661+0000 7f26b97fa640 1 -- 172.21.15.187:0/2742363123 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f26b0014620 con 0x7f26bc1144d0 2024-09-16T07:54:14.662 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.661+0000 7f26b97fa640 1 -- 172.21.15.187:0/2742363123 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f26b0014780 con 0x7f26bc1144d0 2024-09-16T07:54:14.662 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.661+0000 7f26c2d91640 1 -- 172.21.15.187:0/2742363123 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f26bc1a9a30 con 0x7f26bc1144d0 2024-09-16T07:54:14.662 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.661+0000 7f26b97fa640 1 -- 172.21.15.187:0/2742363123 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f26b0014a20 con 0x7f26bc1144d0 2024-09-16T07:54:14.662 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.661+0000 7f26c2d91640 1 -- 172.21.15.187:0/2742363123 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26bc1ae9b0 con 0x7f26bc1144d0 2024-09-16T07:54:14.662 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.661+0000 7f26b97fa640 1 -- 172.21.15.187:0/2742363123 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (crc 0 0 0) 0x7f26b0013400 con 0x7f26bc1144d0 2024-09-16T07:54:14.662 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.661+0000 7f26b97fa640 1 -- 172.21.15.187:0/2742363123 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(6..6 src has 1..6) ==== 1092+0+0 (crc 0 0 0) 0x7f26b0019040 con 0x7f26bc1144d0 2024-09-16T07:54:14.662 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.662+0000 7f26c2d91640 1 -- 172.21.15.187:0/2742363123 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f26bc1203d0 con 0x7f26bc1144d0 2024-09-16T07:54:14.663 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.665+0000 7f26b97fa640 1 -- 172.21.15.187:0/2742363123 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (crc 0 0 0) 0x7f26b00126b0 con 0x7f26bc1144d0 2024-09-16T07:54:14.734 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.735+0000 7f26c2d91640 1 -- 172.21.15.187:0/2742363123 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd new", "uuid": "611aad1c-1d66-400c-a54d-b66c9579b6d9", "id": 3} v 0) -- 0x7f26bc1205b0 con 0x7f26bc1144d0 2024-09-16T07:54:14.736 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.737+0000 7f26b97fa640 1 -- 172.21.15.187:0/2742363123 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "611aad1c-1d66-400c-a54d-b66c9579b6d9", "id": 3}]=0 v7) ==== 112+0+2 (crc 0 0 0) 0x7f26b00129a0 con 0x7f26bc1144d0 2024-09-16T07:54:14.736 INFO:teuthology.orchestra.run.smithi187.stdout:3 2024-09-16T07:54:14.737 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.738+0000 7f26c2d91640 1 -- 172.21.15.187:0/2742363123 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f26bc1144d0 msgr2=0x7f26bc1a90d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:54:14.737 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.738+0000 7f26c2d91640 1 --2- 172.21.15.187:0/2742363123 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f26bc1144d0 0x7f26bc1a90d0 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:54:14.737 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.739+0000 7f26c2d91640 1 -- 172.21.15.187:0/2742363123 shutdown_connections 2024-09-16T07:54:14.737 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.739+0000 7f26c2d91640 1 --2- 172.21.15.187:0/2742363123 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f26bc1144d0 0x7f26bc1a90d0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:54:14.737 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.739+0000 7f26c2d91640 1 -- 172.21.15.187:0/2742363123 >> 172.21.15.187:0/2742363123 conn(0x7f26bc06ff60 msgr2=0x7f26bc117960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:54:14.738 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.739+0000 7f26c2d91640 1 -- 172.21.15.187:0/2742363123 shutdown_connections 2024-09-16T07:54:14.738 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-16T07:54:14.739+0000 7f26c2d91640 1 -- 172.21.15.187:0/2742363123 wait complete. 2024-09-16T07:54:14.748 DEBUG:teuthology.orchestra.run.smithi027:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.0.log 2024-09-16T07:54:14.785 INFO:tasks.ceph.osd.0:Restarting daemon 2024-09-16T07:54:14.785 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2>> /var/log/ceph/ceph-osd.0.log 2024-09-16T07:54:14.827 INFO:tasks.ceph.osd.0:Started 2024-09-16T07:54:14.828 DEBUG:teuthology.orchestra.run.smithi027:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.1.log 2024-09-16T07:54:14.861 INFO:tasks.ceph.osd.1:Restarting daemon 2024-09-16T07:54:14.861 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2>> /var/log/ceph/ceph-osd.1.log 2024-09-16T07:54:14.903 INFO:tasks.ceph.osd.1:Started 2024-09-16T07:54:14.903 DEBUG:teuthology.orchestra.run.smithi187:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.2.log 2024-09-16T07:54:14.934 INFO:tasks.ceph.osd.2:Restarting daemon 2024-09-16T07:54:14.934 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2>> /var/log/ceph/ceph-osd.2.log 2024-09-16T07:54:14.976 INFO:tasks.ceph.osd.2:Started 2024-09-16T07:54:14.976 DEBUG:teuthology.orchestra.run.smithi187:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.3.log 2024-09-16T07:54:15.006 INFO:tasks.ceph.osd.3:Restarting daemon 2024-09-16T07:54:15.006 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2>> /var/log/ceph/ceph-osd.3.log 2024-09-16T07:54:15.049 INFO:tasks.ceph.osd.3:Started 2024-09-16T07:54:15.049 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T07:54:15.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:15.165+0000 7f50940d1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T07:54:15.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:15.170+0000 7f50940d1640 1 Processor -- start 2024-09-16T07:54:15.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:15.171+0000 7f50940d1640 1 -- start start 2024-09-16T07:54:15.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:15.172+0000 7f50940d1640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f508c138720 0x7f508c13ab40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:54:15.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:15.172+0000 7f50940d1640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f508c133040 con 0x7f508c132520 2024-09-16T07:54:15.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:15.172+0000 7f50940d1640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 -- 0x7f508c06f6d0 con 0x7f508c138720 2024-09-16T07:54:15.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:15.172+0000 7f509166c640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f508c138720 0x7f508c13ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:54:15.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:15.172+0000 7f5091e6d640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f508c132520 0x7f508c132930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:55904/0 (socket says 172.21.15.27:55904) 2024-09-16T07:54:15.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:15.172+0000 7f5091e6d640 1 -- 172.21.15.27:0/866762943 learned_addr learned my addr 172.21.15.27:0/866762943 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-09-16T07:54:15.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:15.172+0000 7f509166c640 1 --2- 172.21.15.27:0/866762943 >> v2:172.21.15.27:3300/0 conn(0x7f508c138720 0x7f508c13ab40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54360/0 (socket says 172.21.15.27:54360) 2024-09-16T07:54:15.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:15.173+0000 7f509166c640 1 -- 172.21.15.27:0/866762943 >> v1:172.21.15.27:6789/0 conn(0x7f508c132520 legacy=0x7f508c132930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:54:15.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:15.173+0000 7f509166c640 1 -- 172.21.15.27:0/866762943 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f508c06f3d0 con 0x7f508c138720 2024-09-16T07:54:15.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:15.173+0000 7f5090e6b640 1 -- 172.21.15.27:0/866762943 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3135406837 0 0) 0x7f5088001110 con 0x7f508c132520 2024-09-16T07:54:15.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:15.173+0000 7f509166c640 1 --2- 172.21.15.27:0/866762943 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f508c138720 0x7f508c13ab40 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ef8a5f230cb7105 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:54:15.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:15.173+0000 7f5090e6b640 1 -- 172.21.15.27:0/866762943 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f5080009d90 con 0x7f508c138720 2024-09-16T07:54:15.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:15.173+0000 7f5090e6b640 1 -- 172.21.15.27:0/866762943 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f508000e230 con 0x7f508c138720 2024-09-16T07:54:15.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:15.173+0000 7f5090e6b640 1 -- 172.21.15.27:0/866762943 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f508000e4d0 con 0x7f508c138720 2024-09-16T07:54:15.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:15.174+0000 7f50940d1640 1 -- 172.21.15.27:0/866762943 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f508c138720 msgr2=0x7f508c13ab40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:54:15.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:15.174+0000 7f50940d1640 1 --2- 172.21.15.27:0/866762943 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f508c138720 0x7f508c13ab40 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:54:15.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:15.174+0000 7f50940d1640 1 -- 172.21.15.27:0/866762943 shutdown_connections 2024-09-16T07:54:15.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:15.174+0000 7f50940d1640 1 --2- 172.21.15.27:0/866762943 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f508c138720 0x7f508c13ab40 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:54:15.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:15.174+0000 7f50940d1640 1 -- 172.21.15.27:0/866762943 >> 172.21.15.27:0/866762943 conn(0x7f508c06e490 msgr2=0x7f508c06e8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:54:15.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:15.174+0000 7f50940d1640 1 -- 172.21.15.27:0/866762943 shutdown_connections 2024-09-16T07:54:15.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:15.174+0000 7f50940d1640 1 -- 172.21.15.27:0/866762943 wait complete. 2024-09-16T07:54:15.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:15.174+0000 7f50940d1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T07:54:15.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:15.176+0000 7f50940d1640 1 Processor -- start 2024-09-16T07:54:15.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:15.176+0000 7f50940d1640 1 -- start start 2024-09-16T07:54:15.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:15.178+0000 7f50940d1640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f508c132520 0x7f508c11f580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:54:15.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:15.178+0000 7f50940d1640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f508c122c10 con 0x7f508c132520 2024-09-16T07:54:15.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:15.178+0000 7f5091e6d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f508c132520 0x7f508c11f580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:54:15.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:15.178+0000 7f5091e6d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f508c132520 0x7f508c11f580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54362/0 (socket says 172.21.15.27:54362) 2024-09-16T07:54:15.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:15.178+0000 7f5091e6d640 1 -- 172.21.15.27:0/4272120290 learned_addr learned my addr 172.21.15.27:0/4272120290 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-09-16T07:54:15.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:15.178+0000 7f5091e6d640 1 -- 172.21.15.27:0/4272120290 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f508c11f1c0 con 0x7f508c132520 2024-09-16T07:54:15.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:15.178+0000 7f5091e6d640 1 --2- 172.21.15.27:0/4272120290 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f508c132520 0x7f508c11f580 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:54:15.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:15.178+0000 7f507e7fc640 1 -- 172.21.15.27:0/4272120290 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f5088004b80 con 0x7f508c132520 2024-09-16T07:54:15.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:15.178+0000 7f507e7fc640 1 -- 172.21.15.27:0/4272120290 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f5088004ce0 con 0x7f508c132520 2024-09-16T07:54:15.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:15.178+0000 7f50940d1640 1 -- 172.21.15.27:0/4272120290 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f508c120790 con 0x7f508c132520 2024-09-16T07:54:15.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:15.178+0000 7f507e7fc640 1 -- 172.21.15.27:0/4272120290 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f5088007120 con 0x7f508c132520 2024-09-16T07:54:15.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:15.178+0000 7f50940d1640 1 -- 172.21.15.27:0/4272120290 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f508c1220c0 con 0x7f508c132520 2024-09-16T07:54:15.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:15.179+0000 7f507e7fc640 1 -- 172.21.15.27:0/4272120290 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (crc 0 0 0) 0x7f508800c520 con 0x7f508c132520 2024-09-16T07:54:15.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:15.179+0000 7f507e7fc640 1 -- 172.21.15.27:0/4272120290 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(7..7 src has 1..7) ==== 1207+0+0 (crc 0 0 0) 0x7f508800fc00 con 0x7f508c132520 2024-09-16T07:54:15.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:15.179+0000 7f50940d1640 1 -- 172.21.15.27:0/4272120290 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f508c120f90 con 0x7f508c132520 2024-09-16T07:54:15.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:15.182+0000 7f507e7fc640 1 -- 172.21.15.27:0/4272120290 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (crc 0 0 0) 0x7f508c133040 con 0x7f508c132520 2024-09-16T07:54:15.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:15.243+0000 7f50940d1640 1 -- 172.21.15.27:0/4272120290 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f508c06a300 con 0x7f508c132520 2024-09-16T07:54:15.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:15.244+0000 7f507e7fc640 1 -- 172.21.15.27:0/4272120290 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v7) ==== 74+0+4152 (crc 0 0 0) 0x7f5088007660 con 0x7f508c132520 2024-09-16T07:54:15.242 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-16T07:54:15.242 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":7,"fsid":"16b22dd1-2c19-4258-b5fa-641ce257086f","created":"2024-09-16T07:54:12.578772+0000","modified":"2024-09-16T07:54:14.737256+0000","last_up_change":"0.000000","last_in_change":"2024-09-16T07:54:14.737256+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"9a91e4ff-5797-4c44-a7e4-c088c97e1063","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"a1b27d22-cc01-4cc9-88c0-c28e8c60a99e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"04f4b772-d2c6-4d73-894b-d5635c4d2dda","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"611aad1c-1d66-400c-a54d-b66c9579b6d9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T07:54:15.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:15.245+0000 7f50940d1640 1 -- 172.21.15.27:0/4272120290 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f508c132520 msgr2=0x7f508c11f580 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:54:15.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:15.245+0000 7f50940d1640 1 --2- 172.21.15.27:0/4272120290 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f508c132520 0x7f508c11f580 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:54:15.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:15.245+0000 7f50940d1640 1 -- 172.21.15.27:0/4272120290 shutdown_connections 2024-09-16T07:54:15.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:15.245+0000 7f50940d1640 1 --2- 172.21.15.27:0/4272120290 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f508c132520 0x7f508c11f580 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:54:15.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:15.245+0000 7f50940d1640 1 -- 172.21.15.27:0/4272120290 >> 172.21.15.27:0/4272120290 conn(0x7f508c06e490 msgr2=0x7f508c0a3190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:54:15.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:15.245+0000 7f50940d1640 1 -- 172.21.15.27:0/4272120290 shutdown_connections 2024-09-16T07:54:15.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:15.245+0000 7f50940d1640 1 -- 172.21.15.27:0/4272120290 wait complete. 2024-09-16T07:54:15.254 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-09-16T07:54:15.254 INFO:tasks.ceph:Waiting for OSDs to come up 2024-09-16T07:54:15.656 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-16T07:54:15.764 INFO:teuthology.misc.health.smithi027.stderr:2024-09-16T07:54:15.761+0000 7f63b0cfd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T07:54:15.765 INFO:teuthology.misc.health.smithi027.stderr:2024-09-16T07:54:15.767+0000 7f63b0cfd640 1 Processor -- start 2024-09-16T07:54:15.765 INFO:teuthology.misc.health.smithi027.stderr:2024-09-16T07:54:15.767+0000 7f63b0cfd640 1 -- start start 2024-09-16T07:54:15.768 INFO:teuthology.misc.health.smithi027.stderr:2024-09-16T07:54:15.769+0000 7f63b0cfd640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f63ac135d80 0x7f63ac13ac50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:54:15.769 INFO:teuthology.misc.health.smithi027.stderr:2024-09-16T07:54:15.769+0000 7f63b0cfd640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f63ac06a7b0 con 0x7f63ac0a0c70 2024-09-16T07:54:15.769 INFO:teuthology.misc.health.smithi027.stderr:2024-09-16T07:54:15.769+0000 7f63b0cfd640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 -- 0x7f63ac09f9f0 con 0x7f63ac135d80 2024-09-16T07:54:15.769 INFO:teuthology.misc.health.smithi027.stderr:2024-09-16T07:54:15.769+0000 7f63aa59c640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f63ac0a0c70 0x7f63ac135670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:55910/0 (socket says 172.21.15.27:55910) 2024-09-16T07:54:15.769 INFO:teuthology.misc.health.smithi027.stderr:2024-09-16T07:54:15.769+0000 7f63aa59c640 1 -- 172.21.15.27:0/3385691519 learned_addr learned my addr 172.21.15.27:0/3385691519 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-09-16T07:54:15.769 INFO:teuthology.misc.health.smithi027.stderr:2024-09-16T07:54:15.769+0000 7f63a9d9b640 1 --2- 172.21.15.27:0/3385691519 >> v2:172.21.15.27:3300/0 conn(0x7f63ac135d80 0x7f63ac13ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:54:15.769 INFO:teuthology.misc.health.smithi027.stderr:2024-09-16T07:54:15.770+0000 7f63a959a640 1 -- 172.21.15.27:0/3385691519 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3573020554 0 0) 0x7f63ac06a7b0 con 0x7f63ac0a0c70 2024-09-16T07:54:15.769 INFO:teuthology.misc.health.smithi027.stderr:2024-09-16T07:54:15.770+0000 7f63a959a640 1 -- 172.21.15.27:0/3385691519 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6398003580 con 0x7f63ac0a0c70 2024-09-16T07:54:15.769 INFO:teuthology.misc.health.smithi027.stderr:2024-09-16T07:54:15.770+0000 7f63a9d9b640 1 -- 172.21.15.27:0/3385691519 >> v1:172.21.15.27:6789/0 conn(0x7f63ac0a0c70 legacy=0x7f63ac135670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:54:15.769 INFO:teuthology.misc.health.smithi027.stderr:2024-09-16T07:54:15.770+0000 7f63a9d9b640 1 -- 172.21.15.27:0/3385691519 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63ac09f6e0 con 0x7f63ac135d80 2024-09-16T07:54:15.769 INFO:teuthology.misc.health.smithi027.stderr:2024-09-16T07:54:15.770+0000 7f63a9d9b640 1 --2- 172.21.15.27:0/3385691519 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63ac135d80 0x7f63ac13ac50 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=999c9e4f8c4ada97 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:54:15.769 INFO:teuthology.misc.health.smithi027.stderr:2024-09-16T07:54:15.770+0000 7f63a959a640 1 -- 172.21.15.27:0/3385691519 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f6394009d80 con 0x7f63ac135d80 2024-09-16T07:54:15.770 INFO:teuthology.misc.health.smithi027.stderr:2024-09-16T07:54:15.770+0000 7f63a959a640 1 -- 172.21.15.27:0/3385691519 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f639400e230 con 0x7f63ac135d80 2024-09-16T07:54:15.770 INFO:teuthology.misc.health.smithi027.stderr:2024-09-16T07:54:15.770+0000 7f63a959a640 1 -- 172.21.15.27:0/3385691519 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f639400e4d0 con 0x7f63ac135d80 2024-09-16T07:54:15.771 INFO:teuthology.misc.health.smithi027.stderr:2024-09-16T07:54:15.771+0000 7f63b0cfd640 1 -- 172.21.15.27:0/3385691519 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63ac135d80 msgr2=0x7f63ac13ac50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:54:15.771 INFO:teuthology.misc.health.smithi027.stderr:2024-09-16T07:54:15.771+0000 7f63b0cfd640 1 --2- 172.21.15.27:0/3385691519 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63ac135d80 0x7f63ac13ac50 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:54:15.771 INFO:teuthology.misc.health.smithi027.stderr:2024-09-16T07:54:15.771+0000 7f63b0cfd640 1 -- 172.21.15.27:0/3385691519 shutdown_connections 2024-09-16T07:54:15.771 INFO:teuthology.misc.health.smithi027.stderr:2024-09-16T07:54:15.771+0000 7f63b0cfd640 1 --2- 172.21.15.27:0/3385691519 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63ac135d80 0x7f63ac13ac50 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:54:15.771 INFO:teuthology.misc.health.smithi027.stderr:2024-09-16T07:54:15.771+0000 7f63b0cfd640 1 -- 172.21.15.27:0/3385691519 >> 172.21.15.27:0/3385691519 conn(0x7f63ac0a5f60 msgr2=0x7f63ac0a6370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:54:15.771 INFO:teuthology.misc.health.smithi027.stderr:2024-09-16T07:54:15.771+0000 7f63b0cfd640 1 -- 172.21.15.27:0/3385691519 shutdown_connections 2024-09-16T07:54:15.771 INFO:teuthology.misc.health.smithi027.stderr:2024-09-16T07:54:15.771+0000 7f63b0cfd640 1 -- 172.21.15.27:0/3385691519 wait complete. 2024-09-16T07:54:15.771 INFO:teuthology.misc.health.smithi027.stderr:2024-09-16T07:54:15.771+0000 7f63b0cfd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T07:54:15.772 INFO:teuthology.misc.health.smithi027.stderr:2024-09-16T07:54:15.773+0000 7f63b0cfd640 1 Processor -- start 2024-09-16T07:54:15.772 INFO:teuthology.misc.health.smithi027.stderr:2024-09-16T07:54:15.773+0000 7f63b0cfd640 1 -- start start 2024-09-16T07:54:15.775 INFO:teuthology.misc.health.smithi027.stderr:2024-09-16T07:54:15.775+0000 7f63b0cfd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63ac135d80 0x7f63ac091a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:54:15.775 INFO:teuthology.misc.health.smithi027.stderr:2024-09-16T07:54:15.775+0000 7f63b0cfd640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f63ac09f340 con 0x7f63ac135d80 2024-09-16T07:54:15.775 INFO:teuthology.misc.health.smithi027.stderr:2024-09-16T07:54:15.775+0000 7f63aa59c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63ac135d80 0x7f63ac091a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:54:15.775 INFO:teuthology.misc.health.smithi027.stderr:2024-09-16T07:54:15.775+0000 7f63aa59c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63ac135d80 0x7f63ac091a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54380/0 (socket says 172.21.15.27:54380) 2024-09-16T07:54:15.775 INFO:teuthology.misc.health.smithi027.stderr:2024-09-16T07:54:15.775+0000 7f63aa59c640 1 -- 172.21.15.27:0/2243186641 learned_addr learned my addr 172.21.15.27:0/2243186641 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-09-16T07:54:15.775 INFO:teuthology.misc.health.smithi027.stderr:2024-09-16T07:54:15.775+0000 7f63aa59c640 1 -- 172.21.15.27:0/2243186641 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63ac132340 con 0x7f63ac135d80 2024-09-16T07:54:15.775 INFO:teuthology.misc.health.smithi027.stderr:2024-09-16T07:54:15.776+0000 7f63aa59c640 1 --2- 172.21.15.27:0/2243186641 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63ac135d80 0x7f63ac091a40 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:54:15.775 INFO:teuthology.misc.health.smithi027.stderr:2024-09-16T07:54:15.776+0000 7f63a2ffd640 1 -- 172.21.15.27:0/2243186641 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f639c005a60 con 0x7f63ac135d80 2024-09-16T07:54:15.775 INFO:teuthology.misc.health.smithi027.stderr:2024-09-16T07:54:15.776+0000 7f63a2ffd640 1 -- 172.21.15.27:0/2243186641 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f639c005bc0 con 0x7f63ac135d80 2024-09-16T07:54:15.776 INFO:teuthology.misc.health.smithi027.stderr:2024-09-16T07:54:15.776+0000 7f63b0cfd640 1 -- 172.21.15.27:0/2243186641 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63ac091490 con 0x7f63ac135d80 2024-09-16T07:54:15.776 INFO:teuthology.misc.health.smithi027.stderr:2024-09-16T07:54:15.776+0000 7f63a2ffd640 1 -- 172.21.15.27:0/2243186641 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f639c005e60 con 0x7f63ac135d80 2024-09-16T07:54:15.776 INFO:teuthology.misc.health.smithi027.stderr:2024-09-16T07:54:15.776+0000 7f63b0cfd640 1 -- 172.21.15.27:0/2243186641 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63ac134a00 con 0x7f63ac135d80 2024-09-16T07:54:15.776 INFO:teuthology.misc.health.smithi027.stderr:2024-09-16T07:54:15.776+0000 7f63a2ffd640 1 -- 172.21.15.27:0/2243186641 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (crc 0 0 0) 0x7f639c007440 con 0x7f63ac135d80 2024-09-16T07:54:15.776 INFO:teuthology.misc.health.smithi027.stderr:2024-09-16T07:54:15.776+0000 7f63a2ffd640 1 -- 172.21.15.27:0/2243186641 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(7..7 src has 1..7) ==== 1207+0+0 (crc 0 0 0) 0x7f639c00fac0 con 0x7f63ac135d80 2024-09-16T07:54:15.776 INFO:teuthology.misc.health.smithi027.stderr:2024-09-16T07:54:15.777+0000 7f63b0cfd640 1 -- 172.21.15.27:0/2243186641 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f63ac0a0c70 con 0x7f63ac135d80 2024-09-16T07:54:15.777 INFO:teuthology.misc.health.smithi027.stderr:2024-09-16T07:54:15.779+0000 7f63a2ffd640 1 -- 172.21.15.27:0/2243186641 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (crc 0 0 0) 0x7f639c00fd20 con 0x7f63ac135d80 2024-09-16T07:54:15.840 INFO:teuthology.misc.health.smithi027.stderr:2024-09-16T07:54:15.842+0000 7f63b0cfd640 1 -- 172.21.15.27:0/2243186641 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f63ac131990 con 0x7f63ac135d80 2024-09-16T07:54:15.841 INFO:teuthology.misc.health.smithi027.stderr:2024-09-16T07:54:15.842+0000 7f63a2ffd640 1 -- 172.21.15.27:0/2243186641 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v7) ==== 74+0+4152 (crc 0 0 0) 0x7f639c00f5a0 con 0x7f63ac135d80 2024-09-16T07:54:15.841 INFO:teuthology.misc.health.smithi027.stdout: 2024-09-16T07:54:15.841 INFO:teuthology.misc.health.smithi027.stdout:{"epoch":7,"fsid":"16b22dd1-2c19-4258-b5fa-641ce257086f","created":"2024-09-16T07:54:12.578772+0000","modified":"2024-09-16T07:54:14.737256+0000","last_up_change":"0.000000","last_in_change":"2024-09-16T07:54:14.737256+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"9a91e4ff-5797-4c44-a7e4-c088c97e1063","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"a1b27d22-cc01-4cc9-88c0-c28e8c60a99e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"04f4b772-d2c6-4d73-894b-d5635c4d2dda","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"611aad1c-1d66-400c-a54d-b66c9579b6d9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T07:54:15.842 INFO:teuthology.misc.health.smithi027.stderr:2024-09-16T07:54:15.843+0000 7f63b0cfd640 1 -- 172.21.15.27:0/2243186641 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63ac135d80 msgr2=0x7f63ac091a40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:54:15.842 INFO:teuthology.misc.health.smithi027.stderr:2024-09-16T07:54:15.843+0000 7f63b0cfd640 1 --2- 172.21.15.27:0/2243186641 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63ac135d80 0x7f63ac091a40 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:54:15.842 INFO:teuthology.misc.health.smithi027.stderr:2024-09-16T07:54:15.843+0000 7f63b0cfd640 1 -- 172.21.15.27:0/2243186641 shutdown_connections 2024-09-16T07:54:15.842 INFO:teuthology.misc.health.smithi027.stderr:2024-09-16T07:54:15.843+0000 7f63b0cfd640 1 --2- 172.21.15.27:0/2243186641 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63ac135d80 0x7f63ac091a40 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:54:15.842 INFO:teuthology.misc.health.smithi027.stderr:2024-09-16T07:54:15.843+0000 7f63b0cfd640 1 -- 172.21.15.27:0/2243186641 >> 172.21.15.27:0/2243186641 conn(0x7f63ac0a5f60 msgr2=0x7f63ac0a3280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:54:15.842 INFO:teuthology.misc.health.smithi027.stderr:2024-09-16T07:54:15.843+0000 7f63b0cfd640 1 -- 172.21.15.27:0/2243186641 shutdown_connections 2024-09-16T07:54:15.842 INFO:teuthology.misc.health.smithi027.stderr:2024-09-16T07:54:15.843+0000 7f63b0cfd640 1 -- 172.21.15.27:0/2243186641 wait complete. 2024-09-16T07:54:15.851 DEBUG:teuthology.misc:0 of 4 OSDs are up 2024-09-16T07:54:18.087 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-09-16T07:54:18.088+0000 7f5ee768e640 -1 mgr.server handle_report got status from non-daemon mon.a 2024-09-16T07:54:22.256 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-16T07:54:22.373 INFO:teuthology.misc.health.smithi027.stderr:2024-09-16T07:54:22.371+0000 7efc4a908640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T07:54:22.374 INFO:teuthology.misc.health.smithi027.stderr:2024-09-16T07:54:22.377+0000 7efc4a908640 1 Processor -- start 2024-09-16T07:54:22.375 INFO:teuthology.misc.health.smithi027.stderr:2024-09-16T07:54:22.377+0000 7efc4a908640 1 -- start start 2024-09-16T07:54:22.378 INFO:teuthology.misc.health.smithi027.stderr:2024-09-16T07:54:22.378+0000 7efc4a908640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7efc44135d80 0x7efc4413ac50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:54:22.378 INFO:teuthology.misc.health.smithi027.stderr:2024-09-16T07:54:22.378+0000 7efc4a908640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efc4406a7b0 con 0x7efc440a0c70 2024-09-16T07:54:22.378 INFO:teuthology.misc.health.smithi027.stderr:2024-09-16T07:54:22.378+0000 7efc4a908640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 -- 0x7efc4409f9f0 con 0x7efc44135d80 2024-09-16T07:54:22.378 INFO:teuthology.misc.health.smithi027.stderr:2024-09-16T07:54:22.378+0000 7efc43fff640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7efc440a0c70 0x7efc44135670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:52574/0 (socket says 172.21.15.27:52574) 2024-09-16T07:54:22.378 INFO:teuthology.misc.health.smithi027.stderr:2024-09-16T07:54:22.378+0000 7efc43fff640 1 -- 172.21.15.27:0/1850735755 learned_addr learned my addr 172.21.15.27:0/1850735755 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-09-16T07:54:22.379 INFO:teuthology.misc.health.smithi027.stderr:2024-09-16T07:54:22.378+0000 7efc437fe640 1 --2- 172.21.15.27:0/1850735755 >> v2:172.21.15.27:3300/0 conn(0x7efc44135d80 0x7efc4413ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:54:22.379 INFO:teuthology.misc.health.smithi027.stderr:2024-09-16T07:54:22.379+0000 7efc42ffd640 1 -- 172.21.15.27:0/1850735755 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2135883995 0 0) 0x7efc4406a7b0 con 0x7efc440a0c70 2024-09-16T07:54:22.379 INFO:teuthology.misc.health.smithi027.stderr:2024-09-16T07:54:22.379+0000 7efc42ffd640 1 -- 172.21.15.27:0/1850735755 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efc2c003580 con 0x7efc440a0c70 2024-09-16T07:54:22.379 INFO:teuthology.misc.health.smithi027.stderr:2024-09-16T07:54:22.379+0000 7efc437fe640 1 -- 172.21.15.27:0/1850735755 >> v1:172.21.15.27:6789/0 conn(0x7efc440a0c70 legacy=0x7efc44135670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:54:22.379 INFO:teuthology.misc.health.smithi027.stderr:2024-09-16T07:54:22.379+0000 7efc437fe640 1 -- 172.21.15.27:0/1850735755 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc4409f6e0 con 0x7efc44135d80 2024-09-16T07:54:22.379 INFO:teuthology.misc.health.smithi027.stderr:2024-09-16T07:54:22.379+0000 7efc437fe640 1 --2- 172.21.15.27:0/1850735755 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efc44135d80 0x7efc4413ac50 crc :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5961e94ff0b2c368 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:54:22.379 INFO:teuthology.misc.health.smithi027.stderr:2024-09-16T07:54:22.379+0000 7efc42ffd640 1 -- 172.21.15.27:0/1850735755 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7efc38009d80 con 0x7efc44135d80 2024-09-16T07:54:22.379 INFO:teuthology.misc.health.smithi027.stderr:2024-09-16T07:54:22.379+0000 7efc42ffd640 1 -- 172.21.15.27:0/1850735755 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7efc3800e230 con 0x7efc44135d80 2024-09-16T07:54:22.379 INFO:teuthology.misc.health.smithi027.stderr:2024-09-16T07:54:22.379+0000 7efc42ffd640 1 -- 172.21.15.27:0/1850735755 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7efc3800e4d0 con 0x7efc44135d80 2024-09-16T07:54:22.379 INFO:teuthology.misc.health.smithi027.stderr:2024-09-16T07:54:22.380+0000 7efc4a908640 1 -- 172.21.15.27:0/1850735755 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efc44135d80 msgr2=0x7efc4413ac50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:54:22.379 INFO:teuthology.misc.health.smithi027.stderr:2024-09-16T07:54:22.380+0000 7efc4a908640 1 --2- 172.21.15.27:0/1850735755 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efc44135d80 0x7efc4413ac50 crc :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:54:22.380 INFO:teuthology.misc.health.smithi027.stderr:2024-09-16T07:54:22.380+0000 7efc4a908640 1 -- 172.21.15.27:0/1850735755 shutdown_connections 2024-09-16T07:54:22.380 INFO:teuthology.misc.health.smithi027.stderr:2024-09-16T07:54:22.380+0000 7efc4a908640 1 --2- 172.21.15.27:0/1850735755 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efc44135d80 0x7efc4413ac50 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:54:22.380 INFO:teuthology.misc.health.smithi027.stderr:2024-09-16T07:54:22.380+0000 7efc4a908640 1 -- 172.21.15.27:0/1850735755 >> 172.21.15.27:0/1850735755 conn(0x7efc440a5f60 msgr2=0x7efc440a6370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:54:22.380 INFO:teuthology.misc.health.smithi027.stderr:2024-09-16T07:54:22.380+0000 7efc4a908640 1 -- 172.21.15.27:0/1850735755 shutdown_connections 2024-09-16T07:54:22.380 INFO:teuthology.misc.health.smithi027.stderr:2024-09-16T07:54:22.380+0000 7efc4a908640 1 -- 172.21.15.27:0/1850735755 wait complete. 2024-09-16T07:54:22.380 INFO:teuthology.misc.health.smithi027.stderr:2024-09-16T07:54:22.380+0000 7efc4a908640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T07:54:22.380 INFO:teuthology.misc.health.smithi027.stderr:2024-09-16T07:54:22.382+0000 7efc4a908640 1 Processor -- start 2024-09-16T07:54:22.381 INFO:teuthology.misc.health.smithi027.stderr:2024-09-16T07:54:22.382+0000 7efc4a908640 1 -- start start 2024-09-16T07:54:22.383 INFO:teuthology.misc.health.smithi027.stderr:2024-09-16T07:54:22.384+0000 7efc4a908640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efc44135d80 0x7efc44091a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:54:22.383 INFO:teuthology.misc.health.smithi027.stderr:2024-09-16T07:54:22.384+0000 7efc4a908640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7efc4409f340 con 0x7efc44135d80 2024-09-16T07:54:22.383 INFO:teuthology.misc.health.smithi027.stderr:2024-09-16T07:54:22.384+0000 7efc43fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efc44135d80 0x7efc44091a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:54:22.383 INFO:teuthology.misc.health.smithi027.stderr:2024-09-16T07:54:22.384+0000 7efc43fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efc44135d80 0x7efc44091a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56218/0 (socket says 172.21.15.27:56218) 2024-09-16T07:54:22.383 INFO:teuthology.misc.health.smithi027.stderr:2024-09-16T07:54:22.384+0000 7efc43fff640 1 -- 172.21.15.27:0/1977856376 learned_addr learned my addr 172.21.15.27:0/1977856376 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-09-16T07:54:22.383 INFO:teuthology.misc.health.smithi027.stderr:2024-09-16T07:54:22.384+0000 7efc43fff640 1 -- 172.21.15.27:0/1977856376 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc44132340 con 0x7efc44135d80 2024-09-16T07:54:22.383 INFO:teuthology.misc.health.smithi027.stderr:2024-09-16T07:54:22.384+0000 7efc43fff640 1 --2- 172.21.15.27:0/1977856376 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efc44135d80 0x7efc44091a40 crc :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:54:22.383 INFO:teuthology.misc.health.smithi027.stderr:2024-09-16T07:54:22.384+0000 7efc40ff9640 1 -- 172.21.15.27:0/1977856376 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7efc34005a60 con 0x7efc44135d80 2024-09-16T07:54:22.383 INFO:teuthology.misc.health.smithi027.stderr:2024-09-16T07:54:22.384+0000 7efc40ff9640 1 -- 172.21.15.27:0/1977856376 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7efc34005bc0 con 0x7efc44135d80 2024-09-16T07:54:22.384 INFO:teuthology.misc.health.smithi027.stderr:2024-09-16T07:54:22.385+0000 7efc4a908640 1 -- 172.21.15.27:0/1977856376 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efc44091490 con 0x7efc44135d80 2024-09-16T07:54:22.384 INFO:teuthology.misc.health.smithi027.stderr:2024-09-16T07:54:22.385+0000 7efc40ff9640 1 -- 172.21.15.27:0/1977856376 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7efc34005e60 con 0x7efc44135d80 2024-09-16T07:54:22.384 INFO:teuthology.misc.health.smithi027.stderr:2024-09-16T07:54:22.385+0000 7efc4a908640 1 -- 172.21.15.27:0/1977856376 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efc44134a00 con 0x7efc44135d80 2024-09-16T07:54:22.384 INFO:teuthology.misc.health.smithi027.stderr:2024-09-16T07:54:22.385+0000 7efc4a908640 1 -- 172.21.15.27:0/1977856376 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efc440a0c70 con 0x7efc44135d80 2024-09-16T07:54:22.385 INFO:teuthology.misc.health.smithi027.stderr:2024-09-16T07:54:22.386+0000 7efc40ff9640 1 -- 172.21.15.27:0/1977856376 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 4) ==== 50898+0+0 (crc 0 0 0) 0x7efc340070b0 con 0x7efc44135d80 2024-09-16T07:54:22.385 INFO:teuthology.misc.health.smithi027.stderr:2024-09-16T07:54:22.386+0000 7efc40ff9640 1 --2- 172.21.15.27:0/1977856376 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7efc3003e8d0 0x7efc30040de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:54:22.385 INFO:teuthology.misc.health.smithi027.stderr:2024-09-16T07:54:22.386+0000 7efc40ff9640 1 -- 172.21.15.27:0/1977856376 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(10..10 src has 1..10) ==== 2308+0+0 (crc 0 0 0) 0x7efc3404f3a0 con 0x7efc44135d80 2024-09-16T07:54:22.385 INFO:teuthology.misc.health.smithi027.stderr:2024-09-16T07:54:22.386+0000 7efc437fe640 1 --2- 172.21.15.27:0/1977856376 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7efc3003e8d0 0x7efc30040de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:54:22.385 INFO:teuthology.misc.health.smithi027.stderr:2024-09-16T07:54:22.387+0000 7efc437fe640 1 --2- 172.21.15.27:0/1977856376 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7efc3003e8d0 0x7efc30040de0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:54:22.390 INFO:teuthology.misc.health.smithi027.stderr:2024-09-16T07:54:22.392+0000 7efc40ff9640 1 -- 172.21.15.27:0/1977856376 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (crc 0 0 0) 0x7efc3400dc60 con 0x7efc44135d80 2024-09-16T07:54:22.580 INFO:teuthology.misc.health.smithi027.stderr:2024-09-16T07:54:22.582+0000 7efc4a908640 1 -- 172.21.15.27:0/1977856376 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7efc44131990 con 0x7efc44135d80 2024-09-16T07:54:22.580 INFO:teuthology.misc.health.smithi027.stderr:2024-09-16T07:54:22.582+0000 7efc40ff9640 1 -- 172.21.15.27:0/1977856376 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) ==== 74+0+6946 (crc 0 0 0) 0x7efc3400d660 con 0x7efc44135d80 2024-09-16T07:54:22.581 INFO:teuthology.misc.health.smithi027.stdout: 2024-09-16T07:54:22.581 INFO:teuthology.misc.health.smithi027.stdout:{"epoch":10,"fsid":"16b22dd1-2c19-4258-b5fa-641ce257086f","created":"2024-09-16T07:54:12.578772+0000","modified":"2024-09-16T07:54:21.678374+0000","last_up_change":"2024-09-16T07:54:19.672678+0000","last_in_change":"2024-09-16T07:54:14.737256+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T07:54:21.103580+0000","flags":294945,"flags_names":"hashpspool,nopgchange,creating,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9a91e4ff-5797-4c44-a7e4-c088c97e1063","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":250146213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6801","nonce":250146213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":250146213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6805","nonce":250146213}]},"public_addr":"172.21.15.27:6802/250146213","cluster_addr":"172.21.15.27:6801/250146213","heartbeat_back_addr":"172.21.15.27:6804/250146213","heartbeat_front_addr":"172.21.15.27:6805/250146213","state":["exists","up"]},{"osd":1,"uuid":"a1b27d22-cc01-4cc9-88c0-c28e8c60a99e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":584480168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6803","nonce":584480168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":584480168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6807","nonce":584480168}]},"public_addr":"172.21.15.27:6800/584480168","cluster_addr":"172.21.15.27:6803/584480168","heartbeat_back_addr":"172.21.15.27:6808/584480168","heartbeat_front_addr":"172.21.15.27:6807/584480168","state":["exists","up"]},{"osd":2,"uuid":"04f4b772-d2c6-4d73-894b-d5635c4d2dda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6809","nonce":4110688419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":4110688419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6807","nonce":4110688419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":4110688419}]},"public_addr":"172.21.15.187:6809/4110688419","cluster_addr":"172.21.15.187:6808/4110688419","heartbeat_back_addr":"172.21.15.187:6807/4110688419","heartbeat_front_addr":"172.21.15.187:6812/4110688419","state":["exists","up"]},{"osd":3,"uuid":"611aad1c-1d66-400c-a54d-b66c9579b6d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6811","nonce":1136904278}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":1136904278}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":1136904278}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6803","nonce":1136904278}]},"public_addr":"172.21.15.187:6811/1136904278","cluster_addr":"172.21.15.187:6810/1136904278","heartbeat_back_addr":"172.21.15.187:6804/1136904278","heartbeat_front_addr":"172.21.15.187:6803/1136904278","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T07:54:22.583 INFO:teuthology.misc.health.smithi027.stderr:2024-09-16T07:54:22.585+0000 7efc4a908640 1 -- 172.21.15.27:0/1977856376 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7efc3003e8d0 msgr2=0x7efc30040de0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:54:22.583 INFO:teuthology.misc.health.smithi027.stderr:2024-09-16T07:54:22.585+0000 7efc4a908640 1 --2- 172.21.15.27:0/1977856376 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7efc3003e8d0 0x7efc30040de0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:54:22.583 INFO:teuthology.misc.health.smithi027.stderr:2024-09-16T07:54:22.585+0000 7efc4a908640 1 -- 172.21.15.27:0/1977856376 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efc44135d80 msgr2=0x7efc44091a40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:54:22.584 INFO:teuthology.misc.health.smithi027.stderr:2024-09-16T07:54:22.585+0000 7efc4a908640 1 --2- 172.21.15.27:0/1977856376 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efc44135d80 0x7efc44091a40 crc :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:54:22.584 INFO:teuthology.misc.health.smithi027.stderr:2024-09-16T07:54:22.585+0000 7efc4a908640 1 -- 172.21.15.27:0/1977856376 shutdown_connections 2024-09-16T07:54:22.584 INFO:teuthology.misc.health.smithi027.stderr:2024-09-16T07:54:22.585+0000 7efc4a908640 1 --2- 172.21.15.27:0/1977856376 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7efc3003e8d0 0x7efc30040de0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:54:22.584 INFO:teuthology.misc.health.smithi027.stderr:2024-09-16T07:54:22.585+0000 7efc4a908640 1 --2- 172.21.15.27:0/1977856376 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efc44135d80 0x7efc44091a40 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:54:22.584 INFO:teuthology.misc.health.smithi027.stderr:2024-09-16T07:54:22.585+0000 7efc4a908640 1 -- 172.21.15.27:0/1977856376 >> 172.21.15.27:0/1977856376 conn(0x7efc440a5f60 msgr2=0x7efc440a3280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:54:22.584 INFO:teuthology.misc.health.smithi027.stderr:2024-09-16T07:54:22.585+0000 7efc4a908640 1 -- 172.21.15.27:0/1977856376 shutdown_connections 2024-09-16T07:54:22.584 INFO:teuthology.misc.health.smithi027.stderr:2024-09-16T07:54:22.585+0000 7efc4a908640 1 -- 172.21.15.27:0/1977856376 wait complete. 2024-09-16T07:54:22.593 DEBUG:teuthology.misc:4 of 4 OSDs are up 2024-09-16T07:54:22.593 INFO:tasks.ceph:Creating RBD pool 2024-09-16T07:54:22.593 DEBUG:teuthology.orchestra.run.smithi027:> sudo ceph --cluster ceph osd pool create rbd 8 2024-09-16T07:54:22.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:22.704+0000 7efeb5d91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T07:54:22.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:22.714+0000 7efeb5d91640 1 Processor -- start 2024-09-16T07:54:22.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:22.714+0000 7efeb5d91640 1 -- start start 2024-09-16T07:54:22.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:22.716+0000 7efeb5d91640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7efeb01324b0 0x7efeb01328c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:54:22.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:22.716+0000 7efeb5d91640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efeb009cd20 con 0x7efeb00a68e0 2024-09-16T07:54:22.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:22.716+0000 7efeb5d91640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 -- 0x7efeb009bb70 con 0x7efeb01324b0 2024-09-16T07:54:22.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:22.716+0000 7efeaf7fe640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7efeb01324b0 0x7efeb01328c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:54:22.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:22.716+0000 7efeaf7fe640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7efeb01324b0 0x7efeb01328c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56230/0 (socket says 172.21.15.27:56230) 2024-09-16T07:54:22.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:22.716+0000 7efea6ffd640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7efeb00a68e0 0x7efeb00a6cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:52588/0 (socket says 172.21.15.27:52588) 2024-09-16T07:54:22.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:22.716+0000 7efeaf7fe640 1 -- 172.21.15.27:0/2391921286 learned_addr learned my addr 172.21.15.27:0/2391921286 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-09-16T07:54:22.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:22.717+0000 7efeaf7fe640 1 -- 172.21.15.27:0/2391921286 >> v1:172.21.15.27:6789/0 conn(0x7efeb00a68e0 legacy=0x7efeb00a6cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:54:22.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:22.717+0000 7efeaf7fe640 1 -- 172.21.15.27:0/2391921286 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efeb009c970 con 0x7efeb01324b0 2024-09-16T07:54:22.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:22.717+0000 7efeaf7fe640 1 --2- 172.21.15.27:0/2391921286 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efeb01324b0 0x7efeb01328c0 crc :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c212dfd9c20a3ee server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:54:22.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:22.717+0000 7efeaeffd640 1 -- 172.21.15.27:0/2391921286 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7efea0009e20 con 0x7efeb01324b0 2024-09-16T07:54:22.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:22.717+0000 7efeaeffd640 1 -- 172.21.15.27:0/2391921286 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7efea000e230 con 0x7efeb01324b0 2024-09-16T07:54:22.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:22.717+0000 7efeaeffd640 1 -- 172.21.15.27:0/2391921286 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7efea000e4d0 con 0x7efeb01324b0 2024-09-16T07:54:22.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:22.718+0000 7efeb5d91640 1 -- 172.21.15.27:0/2391921286 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efeb01324b0 msgr2=0x7efeb01328c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:54:22.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:22.718+0000 7efeb5d91640 1 --2- 172.21.15.27:0/2391921286 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efeb01324b0 0x7efeb01328c0 crc :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:54:22.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:22.718+0000 7efeb5d91640 1 -- 172.21.15.27:0/2391921286 shutdown_connections 2024-09-16T07:54:22.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:22.718+0000 7efeb5d91640 1 --2- 172.21.15.27:0/2391921286 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efeb01324b0 0x7efeb01328c0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:54:22.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:22.718+0000 7efeb5d91640 1 -- 172.21.15.27:0/2391921286 >> 172.21.15.27:0/2391921286 conn(0x7efeb006e3e0 msgr2=0x7efeb006e7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:54:22.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:22.718+0000 7efeb5d91640 1 -- 172.21.15.27:0/2391921286 shutdown_connections 2024-09-16T07:54:22.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:22.718+0000 7efeb5d91640 1 -- 172.21.15.27:0/2391921286 wait complete. 2024-09-16T07:54:22.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:22.718+0000 7efeb5d91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T07:54:22.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:22.720+0000 7efeb5d91640 1 Processor -- start 2024-09-16T07:54:22.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:22.720+0000 7efeb5d91640 1 -- start start 2024-09-16T07:54:22.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:22.722+0000 7efeb5d91640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efeb00a68e0 0x7efeb0121520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:54:22.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:22.722+0000 7efeb5d91640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7efeb0124510 con 0x7efeb00a68e0 2024-09-16T07:54:22.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:22.722+0000 7efeaf7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efeb00a68e0 0x7efeb0121520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:54:22.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:22.722+0000 7efeaf7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efeb00a68e0 0x7efeb0121520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56246/0 (socket says 172.21.15.27:56246) 2024-09-16T07:54:22.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:22.722+0000 7efeaf7fe640 1 -- 172.21.15.27:0/3616257164 learned_addr learned my addr 172.21.15.27:0/3616257164 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-09-16T07:54:22.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:22.722+0000 7efeaf7fe640 1 -- 172.21.15.27:0/3616257164 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efeb0125670 con 0x7efeb00a68e0 2024-09-16T07:54:22.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:22.722+0000 7efeaf7fe640 1 --2- 172.21.15.27:0/3616257164 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efeb00a68e0 0x7efeb0121520 crc :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:54:22.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:22.722+0000 7efeacff9640 1 -- 172.21.15.27:0/3616257164 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7efea000ee60 con 0x7efeb00a68e0 2024-09-16T07:54:22.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:22.723+0000 7efeb5d91640 1 -- 172.21.15.27:0/3616257164 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efeb0122ee0 con 0x7efeb00a68e0 2024-09-16T07:54:22.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:22.723+0000 7efeacff9640 1 -- 172.21.15.27:0/3616257164 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7efea0015420 con 0x7efeb00a68e0 2024-09-16T07:54:22.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:22.723+0000 7efeacff9640 1 -- 172.21.15.27:0/3616257164 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7efea00156c0 con 0x7efeb00a68e0 2024-09-16T07:54:22.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:22.723+0000 7efeb5d91640 1 -- 172.21.15.27:0/3616257164 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efeb0124700 con 0x7efeb00a68e0 2024-09-16T07:54:22.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:22.724+0000 7efeacff9640 1 -- 172.21.15.27:0/3616257164 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 4) ==== 50898+0+0 (crc 0 0 0) 0x7efea0015870 con 0x7efeb00a68e0 2024-09-16T07:54:22.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:22.724+0000 7efeb5d91640 1 -- 172.21.15.27:0/3616257164 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efeb00a1ce0 con 0x7efeb00a68e0 2024-09-16T07:54:22.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:22.724+0000 7efeacff9640 1 --2- 172.21.15.27:0/3616257164 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7efe9003e8a0 0x7efe90040db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:54:22.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:22.724+0000 7efeacff9640 1 -- 172.21.15.27:0/3616257164 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(11..11 src has 1..11) ==== 2319+0+0 (crc 0 0 0) 0x7efea0050ad0 con 0x7efeb00a68e0 2024-09-16T07:54:22.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:22.725+0000 7efea6ffd640 1 --2- 172.21.15.27:0/3616257164 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7efe9003e8a0 0x7efe90040db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:54:22.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:22.725+0000 7efea6ffd640 1 --2- 172.21.15.27:0/3616257164 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7efe9003e8a0 0x7efe90040db0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:54:22.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:22.730+0000 7efeacff9640 1 -- 172.21.15.27:0/3616257164 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (crc 0 0 0) 0x7efea0014c10 con 0x7efeb00a68e0 2024-09-16T07:54:22.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:22.982+0000 7efeb5d91640 1 -- 172.21.15.27:0/3616257164 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) -- 0x7efeb006a270 con 0x7efeb00a68e0 2024-09-16T07:54:23.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:23.690+0000 7efeacff9640 1 -- 172.21.15.27:0/3616257164 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]=0 pool 'rbd' created v12) ==== 109+0+0 (crc 0 0 0) 0x7efea00129e0 con 0x7efeb00a68e0 2024-09-16T07:54:23.688 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'rbd' created 2024-09-16T07:54:23.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:23.694+0000 7efeb5d91640 1 -- 172.21.15.27:0/3616257164 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7efe9003e8a0 msgr2=0x7efe90040db0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:54:23.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:23.694+0000 7efeb5d91640 1 --2- 172.21.15.27:0/3616257164 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7efe9003e8a0 0x7efe90040db0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:54:23.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:23.694+0000 7efeb5d91640 1 -- 172.21.15.27:0/3616257164 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efeb00a68e0 msgr2=0x7efeb0121520 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:54:23.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:23.694+0000 7efeb5d91640 1 --2- 172.21.15.27:0/3616257164 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efeb00a68e0 0x7efeb0121520 crc :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:54:23.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:23.694+0000 7efeb5d91640 1 -- 172.21.15.27:0/3616257164 shutdown_connections 2024-09-16T07:54:23.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:23.694+0000 7efeb5d91640 1 --2- 172.21.15.27:0/3616257164 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7efe9003e8a0 0x7efe90040db0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:54:23.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:23.694+0000 7efeb5d91640 1 --2- 172.21.15.27:0/3616257164 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efeb00a68e0 0x7efeb0121520 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:54:23.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:23.694+0000 7efeb5d91640 1 -- 172.21.15.27:0/3616257164 >> 172.21.15.27:0/3616257164 conn(0x7efeb006e3e0 msgr2=0x7efeb00a3220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:54:23.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:23.694+0000 7efeb5d91640 1 -- 172.21.15.27:0/3616257164 shutdown_connections 2024-09-16T07:54:23.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:23.694+0000 7efeb5d91640 1 -- 172.21.15.27:0/3616257164 wait complete. 2024-09-16T07:54:23.709 DEBUG:teuthology.orchestra.run.smithi027:> rbd --cluster ceph pool init rbd 2024-09-16T07:54:23.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:23.825+0000 7f77fb289ac0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T07:54:23.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:23.825+0000 7f77fb289ac0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T07:54:23.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:23.826+0000 7f77fb289ac0 1 Processor -- start 2024-09-16T07:54:23.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:23.826+0000 7f77fb289ac0 1 -- start start 2024-09-16T07:54:23.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:23.828+0000 7f77fb289ac0 1 --2- >> v2:172.21.15.27:3300/0 conn(0x110a1c0 0x110a5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:54:23.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:23.828+0000 7f77fb289ac0 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0xf6f180 con 0x110ab10 2024-09-16T07:54:23.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:23.828+0000 7f77fb289ac0 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 -- 0xfaeb90 con 0x110a1c0 2024-09-16T07:54:23.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:23.828+0000 7f77f548b640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x110a1c0 0x110a5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:54:23.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:23.828+0000 7f77f4c8a640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x110ab10 0x11a6cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:52592/0 (socket says 172.21.15.27:52592) 2024-09-16T07:54:23.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:23.828+0000 7f77f4c8a640 1 -- 172.21.15.27:0/3072426189 learned_addr learned my addr 172.21.15.27:0/3072426189 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-09-16T07:54:23.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:23.828+0000 7f77f548b640 1 --2- 172.21.15.27:0/3072426189 >> v2:172.21.15.27:3300/0 conn(0x110a1c0 0x110a5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56254/0 (socket says 172.21.15.27:56254) 2024-09-16T07:54:23.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:23.828+0000 7f77f548b640 1 -- 172.21.15.27:0/3072426189 >> v1:172.21.15.27:6789/0 conn(0x110ab10 legacy=0x11a6cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:54:23.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:23.828+0000 7f77f548b640 1 -- 172.21.15.27:0/3072426189 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0xf44f80 con 0x110a1c0 2024-09-16T07:54:23.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:23.828+0000 7f77e7fff640 1 -- 172.21.15.27:0/3072426189 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 26450574 0 0) 0xf6f180 con 0x110ab10 2024-09-16T07:54:23.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:23.830+0000 7f77f548b640 1 --2- 172.21.15.27:0/3072426189 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x110a1c0 0x110a5d0 crc :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c45139e5d01e86b3 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:54:23.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:23.830+0000 7f77e7fff640 1 -- 172.21.15.27:0/3072426189 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f77ec02d7f0 con 0x110a1c0 2024-09-16T07:54:23.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:23.830+0000 7f77e7fff640 1 -- 172.21.15.27:0/3072426189 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f77ec02d950 con 0x110a1c0 2024-09-16T07:54:23.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:23.830+0000 7f77e7fff640 1 -- 172.21.15.27:0/3072426189 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f77ec02dbf0 con 0x110a1c0 2024-09-16T07:54:23.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:23.830+0000 7f77fb289ac0 1 -- 172.21.15.27:0/3072426189 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x110a1c0 msgr2=0x110a5d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:54:23.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:23.830+0000 7f77fb289ac0 1 --2- 172.21.15.27:0/3072426189 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x110a1c0 0x110a5d0 crc :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:54:23.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:23.830+0000 7f77fb289ac0 1 -- 172.21.15.27:0/3072426189 shutdown_connections 2024-09-16T07:54:23.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:23.830+0000 7f77fb289ac0 1 --2- 172.21.15.27:0/3072426189 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x110a1c0 0x110a5d0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:54:23.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:23.830+0000 7f77fb289ac0 1 -- 172.21.15.27:0/3072426189 >> 172.21.15.27:0/3072426189 conn(0xf17f70 msgr2=0x1108780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:54:23.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:23.831+0000 7f77fb289ac0 1 -- 172.21.15.27:0/3072426189 shutdown_connections 2024-09-16T07:54:23.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:23.831+0000 7f77fb289ac0 1 -- 172.21.15.27:0/3072426189 wait complete. 2024-09-16T07:54:23.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:23.831+0000 7f77fb289ac0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T07:54:23.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:23.832+0000 7f77fb289ac0 1 Processor -- start 2024-09-16T07:54:23.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:23.832+0000 7f77fb289ac0 1 -- start start 2024-09-16T07:54:23.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:23.834+0000 7f77fb289ac0 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x110a1c0 0x11b4430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:54:23.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:23.834+0000 7f77fb289ac0 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0xfb98f0 con 0x110a1c0 2024-09-16T07:54:23.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:23.834+0000 7f77f548b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x110a1c0 0x11b4430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:54:23.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:23.834+0000 7f77f548b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x110a1c0 0x11b4430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56260/0 (socket says 172.21.15.27:56260) 2024-09-16T07:54:23.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:23.834+0000 7f77f548b640 1 -- 172.21.15.27:0/1578959591 learned_addr learned my addr 172.21.15.27:0/1578959591 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-09-16T07:54:23.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:23.834+0000 7f77f548b640 1 -- 172.21.15.27:0/1578959591 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0xf53800 con 0x110a1c0 2024-09-16T07:54:23.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:23.834+0000 7f77f548b640 1 --2- 172.21.15.27:0/1578959591 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x110a1c0 0x11b4430 crc :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:54:23.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:23.835+0000 7f77e57fa640 1 -- 172.21.15.27:0/1578959591 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f77ec009540 con 0x110a1c0 2024-09-16T07:54:23.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:23.835+0000 7f77e57fa640 1 -- 172.21.15.27:0/1578959591 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f77ec0096a0 con 0x110a1c0 2024-09-16T07:54:23.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:23.835+0000 7f77fb289ac0 1 -- 172.21.15.27:0/1578959591 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0xf45310 con 0x110a1c0 2024-09-16T07:54:23.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:23.835+0000 7f77e57fa640 1 -- 172.21.15.27:0/1578959591 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f77ec009940 con 0x110a1c0 2024-09-16T07:54:23.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:23.835+0000 7f77fb289ac0 1 -- 172.21.15.27:0/1578959591 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0xf44d70 con 0x110a1c0 2024-09-16T07:54:23.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:23.836+0000 7f77e57fa640 1 -- 172.21.15.27:0/1578959591 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 4) ==== 50898+0+0 (crc 0 0 0) 0x7f77ec009af0 con 0x110a1c0 2024-09-16T07:54:23.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:23.836+0000 7f77e57fa640 1 --2- 172.21.15.27:0/1578959591 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7f77d403d130 0x7f77d403f640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:54:23.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:23.836+0000 7f77e57fa640 1 -- 172.21.15.27:0/1578959591 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(12..12 src has 1..12) ==== 2661+0+0 (crc 0 0 0) 0x7f77ec032040 con 0x110a1c0 2024-09-16T07:54:23.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:23.836+0000 7f77fb289ac0 1 -- 172.21.15.27:0/1578959591 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool application enable","pool": "rbd","app": "rbd"} v 0) -- 0x1192980 con 0x110a1c0 2024-09-16T07:54:23.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:23.836+0000 7f77f4c8a640 1 --2- 172.21.15.27:0/1578959591 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7f77d403d130 0x7f77d403f640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:54:23.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:23.837+0000 7f77f4c8a640 1 --2- 172.21.15.27:0/1578959591 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7f77d403d130 0x7f77d403f640 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:54:24.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:24.694+0000 7f77e57fa640 1 -- 172.21.15.27:0/1578959591 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}]=0 enabled application 'rbd' on pool 'rbd' v13) ==== 141+0+0 (crc 0 0 0) 0x7f77ec037370 con 0x110a1c0 2024-09-16T07:54:24.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:24.694+0000 7f77fb289ac0 1 -- 172.21.15.27:0/1578959591 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0xf6c520 con 0x110a1c0 2024-09-16T07:54:24.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:24.697+0000 7f77e57fa640 1 -- 172.21.15.27:0/1578959591 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_get_version_reply(handle=1 version=13) ==== 24+0+0 (crc 0 0 0) 0x7f77ec037370 con 0x110a1c0 2024-09-16T07:54:24.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:24.697+0000 7f77e77fe640 1 -- 172.21.15.27:0/1578959591 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=13}) -- 0x7f77cc002ae0 con 0x110a1c0 2024-09-16T07:54:24.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:24.697+0000 7f77e57fa640 1 -- 172.21.15.27:0/1578959591 <== mon.0 v2:172.21.15.27:3300/0 8 ==== osd_map(13..13 src has 1..13) ==== 658+0+0 (crc 0 0 0) 0x7f77ec034280 con 0x110a1c0 2024-09-16T07:54:24.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:24.698+0000 7f77fb289ac0 1 --2- 172.21.15.27:0/1578959591 >> v2:172.21.15.27:6800/584480168 conn(0x110ab10 0x11b0e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:54:24.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:24.698+0000 7f77fb289ac0 1 -- 172.21.15.27:0/1578959591 --> v2:172.21.15.27:6800/584480168 -- osd_op(unknown.0.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13) -- 0x11b13a0 con 0x110ab10 2024-09-16T07:54:24.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:24.702+0000 7f77f66ed640 1 --2- 172.21.15.27:0/1578959591 >> v2:172.21.15.27:6800/584480168 conn(0x110ab10 0x11b0e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-09-16T07:54:24.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:24.727+0000 7f77f66ed640 1 --2- 172.21.15.27:0/1578959591 >> v2:172.21.15.27:6800/584480168 conn(0x110ab10 0x11b0e60 crc :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:54:24.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:24.835+0000 7f77f66ed640 1 -- 172.21.15.27:0/1578959591 <== osd.1 v2:172.21.15.27:6800/584480168 1 ==== osd_op_reply(1 rbd_trash [create] v13'1 uv1 ondisk = 0) ==== 153+0+0 (crc 0 0 0) 0x7f77e800a4b0 con 0x110ab10 2024-09-16T07:54:24.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:24.838+0000 7f77fb289ac0 1 --2- 172.21.15.27:0/1578959591 >> v2:172.21.15.27:6802/250146213 conn(0x120bbc0 0x120e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:54:24.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:24.838+0000 7f77fb289ac0 1 -- 172.21.15.27:0/1578959591 --> v2:172.21.15.27:6802/250146213 -- osd_op(unknown.0.0:2 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=33b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e13) -- 0x120e670 con 0x120bbc0 2024-09-16T07:54:24.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:24.839+0000 7f77f548b640 1 --2- 172.21.15.27:0/1578959591 >> v2:172.21.15.27:6802/250146213 conn(0x120bbc0 0x120e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-09-16T07:54:24.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:24.842+0000 7f77f548b640 1 --2- 172.21.15.27:0/1578959591 >> v2:172.21.15.27:6802/250146213 conn(0x120bbc0 0x120e0a0 crc :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:54:24.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:24.847+0000 7f77f548b640 1 -- 172.21.15.27:0/1578959591 <== osd.0 v2:172.21.15.27:6802/250146213 1 ==== osd_op_reply(2 rbd_info [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 152+0+0 (crc 0 0 0) 0x7f77ec035040 con 0x120bbc0 2024-09-16T07:54:24.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:24.847+0000 7f77fb289ac0 1 -- 172.21.15.27:0/1578959591 --> v2:172.21.15.27:6802/250146213 -- osd_op(unknown.0.0:3 2.2 2:5cea7035:::rbd_info:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e13) -- 0x120b080 con 0x120bbc0 2024-09-16T07:54:24.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:24.850+0000 7f77f548b640 1 -- 172.21.15.27:0/1578959591 <== osd.0 v2:172.21.15.27:6802/250146213 2 ==== osd_op_reply(3 rbd_info [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 152+0+0 (crc 0 0 0) 0x7f77ec035040 con 0x120bbc0 2024-09-16T07:54:24.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:24.850+0000 7f77e6ffd640 1 -- 172.21.15.27:0/1578959591 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- pool_op(create unmanaged snap pool 2 tid 4 name v0) -- 0x7f77d0001f60 con 0x110a1c0 2024-09-16T07:54:25.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:25.699+0000 7f77e57fa640 1 -- 172.21.15.27:0/1578959591 <== mon.0 v2:172.21.15.27:3300/0 9 ==== mgrmap(e 5) ==== 50954+0+0 (crc 0 0 0) 0x7f77ec0357f0 con 0x110a1c0 2024-09-16T07:54:25.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:25.704+0000 7f77e57fa640 1 -- 172.21.15.27:0/1578959591 <== mon.0 v2:172.21.15.27:3300/0 10 ==== pool_op_reply(tid 4 (0) Success v14) ==== 55+0+0 (crc 0 0 0) 0x7f77ec007b00 con 0x110a1c0 2024-09-16T07:54:25.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:25.704+0000 7f77e57fa640 1 -- 172.21.15.27:0/1578959591 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=14}) -- 0x7f77d40427c0 con 0x110a1c0 2024-09-16T07:54:25.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:25.705+0000 7f77e57fa640 1 -- 172.21.15.27:0/1578959591 <== mon.0 v2:172.21.15.27:3300/0 11 ==== osd_map(14..14 src has 1..14) ==== 626+0+0 (crc 0 0 0) 0x7f77ec02f0b0 con 0x110a1c0 2024-09-16T07:54:25.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:25.705+0000 7f77e77fe640 1 -- 172.21.15.27:0/1578959591 --> v2:172.21.15.27:6802/250146213 -- osd_op(unknown.0.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14) -- 0x7f77cc005640 con 0x120bbc0 2024-09-16T07:54:25.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:25.748+0000 7f77f548b640 1 -- 172.21.15.27:0/1578959591 <== osd.0 v2:172.21.15.27:6802/250146213 3 ==== osd_op_reply(5 rbd_info [create,write 0~8] v14'1 uv1 ondisk = 0) ==== 194+0+0 (crc 0 0 0) 0x7f77ec035040 con 0x120bbc0 2024-09-16T07:54:25.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:25.748+0000 7f77e6ffd640 1 -- 172.21.15.27:0/1578959591 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- pool_op(delete unmanaged snap pool 2 tid 6 name v14) -- 0x7f77d0003370 con 0x110a1c0 2024-09-16T07:54:26.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:26.707+0000 7f77e57fa640 1 -- 172.21.15.27:0/1578959591 <== mon.0 v2:172.21.15.27:3300/0 12 ==== pool_op_reply(tid 6 (0) Success v15) ==== 43+0+0 (crc 0 0 0) 0x7f77ec006550 con 0x110a1c0 2024-09-16T07:54:26.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:26.707+0000 7f77e57fa640 1 -- 172.21.15.27:0/1578959591 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=15}) -- 0x7f77d4042950 con 0x110a1c0 2024-09-16T07:54:26.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:26.708+0000 7f77e57fa640 1 -- 172.21.15.27:0/1578959591 <== mon.0 v2:172.21.15.27:3300/0 13 ==== osd_map(15..15 src has 1..15) ==== 654+0+0 (crc 0 0 0) 0x7f77ec0726a0 con 0x110a1c0 2024-09-16T07:54:26.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:26.708+0000 7f77e77fe640 1 -- 172.21.15.27:0/1578959591 --> v2:172.21.15.27:6802/250146213 -- osd_op(unknown.0.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19 in=19b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15) -- 0x7f77cc0059e0 con 0x120bbc0 2024-09-16T07:54:26.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:26.761+0000 7f77f548b640 1 -- 172.21.15.27:0/1578959591 <== osd.0 v2:172.21.15.27:6802/250146213 4 ==== osd_op_reply(7 rbd_info [write 0~19] v15'2 uv2 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f77ec035040 con 0x120bbc0 2024-09-16T07:54:26.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:26.762+0000 7f77fb289ac0 1 -- 172.21.15.27:0/1578959591 >> v2:172.21.15.27:6802/250146213 conn(0x120bbc0 msgr2=0x120e0a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:54:26.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:26.762+0000 7f77fb289ac0 1 --2- 172.21.15.27:0/1578959591 >> v2:172.21.15.27:6802/250146213 conn(0x120bbc0 0x120e0a0 crc :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:54:26.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:26.762+0000 7f77fb289ac0 1 -- 172.21.15.27:0/1578959591 >> v2:172.21.15.27:6800/584480168 conn(0x110ab10 msgr2=0x11b0e60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:54:26.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:26.762+0000 7f77fb289ac0 1 --2- 172.21.15.27:0/1578959591 >> v2:172.21.15.27:6800/584480168 conn(0x110ab10 0x11b0e60 crc :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:54:26.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:26.762+0000 7f77fb289ac0 1 -- 172.21.15.27:0/1578959591 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7f77d403d130 msgr2=0x7f77d403f640 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:54:26.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:26.762+0000 7f77fb289ac0 1 --2- 172.21.15.27:0/1578959591 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7f77d403d130 0x7f77d403f640 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:54:26.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:26.762+0000 7f77fb289ac0 1 -- 172.21.15.27:0/1578959591 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x110a1c0 msgr2=0x11b4430 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:54:26.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:26.762+0000 7f77fb289ac0 1 --2- 172.21.15.27:0/1578959591 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x110a1c0 0x11b4430 crc :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:54:26.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:26.762+0000 7f77fb289ac0 1 -- 172.21.15.27:0/1578959591 shutdown_connections 2024-09-16T07:54:26.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:26.762+0000 7f77fb289ac0 1 --2- 172.21.15.27:0/1578959591 >> v2:172.21.15.27:6800/584480168 conn(0x110ab10 0x11b0e60 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:54:26.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:26.762+0000 7f77fb289ac0 1 --2- 172.21.15.27:0/1578959591 >> v2:172.21.15.27:6802/250146213 conn(0x120bbc0 0x120e0a0 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:54:26.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:26.762+0000 7f77fb289ac0 1 --2- 172.21.15.27:0/1578959591 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7f77d403d130 0x7f77d403f640 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:54:26.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:26.762+0000 7f77fb289ac0 1 --2- 172.21.15.27:0/1578959591 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x110a1c0 0x11b4430 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:54:26.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:26.762+0000 7f77fb289ac0 1 -- 172.21.15.27:0/1578959591 >> 172.21.15.27:0/1578959591 conn(0xf17f70 msgr2=0x1113fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:54:26.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:26.762+0000 7f77fb289ac0 1 -- 172.21.15.27:0/1578959591 shutdown_connections 2024-09-16T07:54:26.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:26.762+0000 7f77fb289ac0 1 -- 172.21.15.27:0/1578959591 wait complete. 2024-09-16T07:54:26.765 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-09-16T07:54:26.766 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config log 1 --format=json 2024-09-16T07:54:26.767 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-09-16T07:54:26.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:26.880+0000 7f6261d91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T07:54:26.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:26.887+0000 7f6261d91640 1 Processor -- start 2024-09-16T07:54:26.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:26.887+0000 7f6261d91640 1 -- start start 2024-09-16T07:54:26.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:26.888+0000 7f6261d91640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f625c1386c0 0x7f625c13aae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:54:26.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:26.888+0000 7f6261d91640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f625c132e30 con 0x7f625c132310 2024-09-16T07:54:26.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:26.888+0000 7f6261d91640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 -- 0x7f625c0a0f40 con 0x7f625c1386c0 2024-09-16T07:54:26.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:26.888+0000 7f6252ffd640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f625c1386c0 0x7f625c13aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:54:26.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:26.888+0000 7f6252ffd640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f625c1386c0 0x7f625c13aae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56264/0 (socket says 172.21.15.27:56264) 2024-09-16T07:54:26.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:26.888+0000 7f625b7fe640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f625c132310 0x7f625c132720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:52608/0 (socket says 172.21.15.27:52608) 2024-09-16T07:54:26.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:26.888+0000 7f6252ffd640 1 -- 172.21.15.27:0/137920488 learned_addr learned my addr 172.21.15.27:0/137920488 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-09-16T07:54:26.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:26.889+0000 7f6252ffd640 1 -- 172.21.15.27:0/137920488 >> v1:172.21.15.27:6789/0 conn(0x7f625c132310 legacy=0x7f625c132720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:54:26.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:26.889+0000 7f6252ffd640 1 -- 172.21.15.27:0/137920488 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f625c09ed40 con 0x7f625c1386c0 2024-09-16T07:54:26.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:26.889+0000 7f625affd640 1 -- 172.21.15.27:0/137920488 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1869091019 0 0) 0x7f624c001110 con 0x7f625c132310 2024-09-16T07:54:26.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:26.889+0000 7f6252ffd640 1 --2- 172.21.15.27:0/137920488 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f625c1386c0 0x7f625c13aae0 crc :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db9550eae19516a8 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:54:26.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:26.889+0000 7f625affd640 1 -- 172.21.15.27:0/137920488 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f6244009e50 con 0x7f625c1386c0 2024-09-16T07:54:26.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:26.889+0000 7f625affd640 1 -- 172.21.15.27:0/137920488 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f624400e230 con 0x7f625c1386c0 2024-09-16T07:54:26.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:26.889+0000 7f625affd640 1 -- 172.21.15.27:0/137920488 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f624400e4d0 con 0x7f625c1386c0 2024-09-16T07:54:26.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:26.890+0000 7f6261d91640 1 -- 172.21.15.27:0/137920488 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f625c1386c0 msgr2=0x7f625c13aae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:54:26.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:26.890+0000 7f6261d91640 1 --2- 172.21.15.27:0/137920488 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f625c1386c0 0x7f625c13aae0 crc :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:54:26.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:26.890+0000 7f6261d91640 1 -- 172.21.15.27:0/137920488 shutdown_connections 2024-09-16T07:54:26.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:26.890+0000 7f6261d91640 1 --2- 172.21.15.27:0/137920488 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f625c1386c0 0x7f625c13aae0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:54:26.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:26.890+0000 7f6261d91640 1 -- 172.21.15.27:0/137920488 >> 172.21.15.27:0/137920488 conn(0x7f625c069d10 msgr2=0x7f625c09f200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:54:26.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:26.890+0000 7f6261d91640 1 -- 172.21.15.27:0/137920488 shutdown_connections 2024-09-16T07:54:26.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:26.890+0000 7f6261d91640 1 -- 172.21.15.27:0/137920488 wait complete. 2024-09-16T07:54:26.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:26.890+0000 7f6261d91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T07:54:26.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:26.892+0000 7f6261d91640 1 Processor -- start 2024-09-16T07:54:26.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:26.892+0000 7f6261d91640 1 -- start start 2024-09-16T07:54:26.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:26.893+0000 7f6261d91640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f625c132310 0x7f625c11deb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:54:26.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:26.893+0000 7f6261d91640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f625c131ea0 con 0x7f625c132310 2024-09-16T07:54:26.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:26.894+0000 7f625b7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f625c132310 0x7f625c11deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:54:26.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:26.894+0000 7f625b7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f625c132310 0x7f625c11deb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56280/0 (socket says 172.21.15.27:56280) 2024-09-16T07:54:26.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:26.894+0000 7f625b7fe640 1 -- 172.21.15.27:0/2039394414 learned_addr learned my addr 172.21.15.27:0/2039394414 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-09-16T07:54:26.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:26.894+0000 7f625b7fe640 1 -- 172.21.15.27:0/2039394414 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f625c121df0 con 0x7f625c132310 2024-09-16T07:54:26.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:26.894+0000 7f625b7fe640 1 --2- 172.21.15.27:0/2039394414 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f625c132310 0x7f625c11deb0 crc :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:54:26.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:26.894+0000 7f6258ff9640 1 -- 172.21.15.27:0/2039394414 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f624c004b80 con 0x7f625c132310 2024-09-16T07:54:26.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:26.894+0000 7f6258ff9640 1 -- 172.21.15.27:0/2039394414 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f624c004ce0 con 0x7f625c132310 2024-09-16T07:54:26.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:26.894+0000 7f6261d91640 1 -- 172.21.15.27:0/2039394414 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f625c131b70 con 0x7f625c132310 2024-09-16T07:54:26.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:26.894+0000 7f6258ff9640 1 -- 172.21.15.27:0/2039394414 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f624c007120 con 0x7f625c132310 2024-09-16T07:54:26.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:26.894+0000 7f6261d91640 1 -- 172.21.15.27:0/2039394414 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f625c137820 con 0x7f625c132310 2024-09-16T07:54:26.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:26.895+0000 7f6258ff9640 1 -- 172.21.15.27:0/2039394414 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) ==== 50954+0+0 (crc 0 0 0) 0x7f624c0072d0 con 0x7f625c132310 2024-09-16T07:54:26.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:26.895+0000 7f6258ff9640 1 --2- 172.21.15.27:0/2039394414 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7f623c03e930 0x7f623c040e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:54:26.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:26.895+0000 7f6261d91640 1 -- 172.21.15.27:0/2039394414 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f625c131d00 con 0x7f625c132310 2024-09-16T07:54:26.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:26.895+0000 7f6258ff9640 1 -- 172.21.15.27:0/2039394414 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(15..15 src has 1..15) ==== 2728+0+0 (crc 0 0 0) 0x7f624c04f670 con 0x7f625c132310 2024-09-16T07:54:26.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:26.896+0000 7f6252ffd640 1 --2- 172.21.15.27:0/2039394414 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7f623c03e930 0x7f623c040e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:54:26.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:26.896+0000 7f6252ffd640 1 --2- 172.21.15.27:0/2039394414 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7f623c03e930 0x7f623c040e40 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:54:26.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:26.901+0000 7f6258ff9640 1 -- 172.21.15.27:0/2039394414 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (crc 0 0 0) 0x7f625c131d00 con 0x7f625c132310 2024-09-16T07:54:27.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:27.086+0000 7f6261d91640 1 -- 172.21.15.27:0/2039394414 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "config log", "num": 1, "format": "json"} v 0) -- 0x7f625c131d00 con 0x7f625c132310 2024-09-16T07:54:27.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:27.087+0000 7f6258ff9640 1 -- 172.21.15.27:0/2039394414 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "config log", "num": 1, "format": "json"}]=0 v1) ==== 86+0+61 (crc 0 0 0) 0x7f625c131d00 con 0x7f625c132310 2024-09-16T07:54:27.085 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-16T07:54:27.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:27.089+0000 7f6261d91640 1 -- 172.21.15.27:0/2039394414 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7f623c03e930 msgr2=0x7f623c040e40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:54:27.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:27.089+0000 7f6261d91640 1 --2- 172.21.15.27:0/2039394414 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7f623c03e930 0x7f623c040e40 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:54:27.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:27.089+0000 7f6261d91640 1 -- 172.21.15.27:0/2039394414 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f625c132310 msgr2=0x7f625c11deb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:54:27.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:27.089+0000 7f6261d91640 1 --2- 172.21.15.27:0/2039394414 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f625c132310 0x7f625c11deb0 crc :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:54:27.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:27.089+0000 7f6261d91640 1 -- 172.21.15.27:0/2039394414 shutdown_connections 2024-09-16T07:54:27.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:27.089+0000 7f6261d91640 1 --2- 172.21.15.27:0/2039394414 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7f623c03e930 0x7f623c040e40 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:54:27.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:27.089+0000 7f6261d91640 1 --2- 172.21.15.27:0/2039394414 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f625c132310 0x7f625c11deb0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:54:27.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:27.089+0000 7f6261d91640 1 -- 172.21.15.27:0/2039394414 >> 172.21.15.27:0/2039394414 conn(0x7f625c069d10 msgr2=0x7f625c1bd7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:54:27.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:27.089+0000 7f6261d91640 1 -- 172.21.15.27:0/2039394414 shutdown_connections 2024-09-16T07:54:27.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:27.089+0000 7f6261d91640 1 -- 172.21.15.27:0/2039394414 wait complete. 2024-09-16T07:54:27.098 INFO:teuthology.orchestra.run.smithi027.stdout:[{"version":1,"timestamp":"0.000000","name":"","changes":[]}] 2024-09-16T07:54:27.098 INFO:tasks.ceph_manager:config epoch is 1 2024-09-16T07:54:27.099 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-16T07:54:27.099 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-09-16T07:54:27.099 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-09-16T07:54:27.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:27.209+0000 7fc90c751640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T07:54:27.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:27.214+0000 7fc90c751640 1 Processor -- start 2024-09-16T07:54:27.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:27.214+0000 7fc90c751640 1 -- start start 2024-09-16T07:54:27.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:27.215+0000 7fc90c751640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc904132520 0x7fc904132930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:54:27.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:27.215+0000 7fc90c751640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc90406f6d0 con 0x7fc9040a3450 2024-09-16T07:54:27.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:27.216+0000 7fc90c751640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 -- 0x7fc90406f3d0 con 0x7fc904132520 2024-09-16T07:54:27.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:27.216+0000 7fc90a4ed640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc904132520 0x7fc904132930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:54:27.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:27.216+0000 7fc909cec640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fc9040a3450 0x7fc9040a3860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:52618/0 (socket says 172.21.15.27:52618) 2024-09-16T07:54:27.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:27.216+0000 7fc909cec640 1 -- 172.21.15.27:0/2635159216 learned_addr learned my addr 172.21.15.27:0/2635159216 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-09-16T07:54:27.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:27.216+0000 7fc90a4ed640 1 -- 172.21.15.27:0/2635159216 >> v1:172.21.15.27:6789/0 conn(0x7fc9040a3450 legacy=0x7fc9040a3860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:54:27.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:27.216+0000 7fc90a4ed640 1 -- 172.21.15.27:0/2635159216 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc90406f0c0 con 0x7fc904132520 2024-09-16T07:54:27.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:27.216+0000 7fc90a4ed640 1 --2- 172.21.15.27:0/2635159216 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc904132520 0x7fc904132930 crc :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14b93c56476c0b6a server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:54:27.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:27.216+0000 7fc9094eb640 1 -- 172.21.15.27:0/2635159216 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fc900009d90 con 0x7fc904132520 2024-09-16T07:54:27.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:27.216+0000 7fc9094eb640 1 -- 172.21.15.27:0/2635159216 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7fc90000e230 con 0x7fc904132520 2024-09-16T07:54:27.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:27.216+0000 7fc9094eb640 1 -- 172.21.15.27:0/2635159216 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fc90000e4d0 con 0x7fc904132520 2024-09-16T07:54:27.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:27.217+0000 7fc90c751640 1 -- 172.21.15.27:0/2635159216 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc904132520 msgr2=0x7fc904132930 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:54:27.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:27.217+0000 7fc90c751640 1 --2- 172.21.15.27:0/2635159216 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc904132520 0x7fc904132930 crc :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:54:27.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:27.217+0000 7fc90c751640 1 -- 172.21.15.27:0/2635159216 shutdown_connections 2024-09-16T07:54:27.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:27.217+0000 7fc90c751640 1 --2- 172.21.15.27:0/2635159216 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc904132520 0x7fc904132930 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:54:27.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:27.217+0000 7fc90c751640 1 -- 172.21.15.27:0/2635159216 >> 172.21.15.27:0/2635159216 conn(0x7fc90406e490 msgr2=0x7fc90406e8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:54:27.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:27.217+0000 7fc90c751640 1 -- 172.21.15.27:0/2635159216 shutdown_connections 2024-09-16T07:54:27.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:27.217+0000 7fc90c751640 1 -- 172.21.15.27:0/2635159216 wait complete. 2024-09-16T07:54:27.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:27.218+0000 7fc90c751640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T07:54:27.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:27.219+0000 7fc90c751640 1 Processor -- start 2024-09-16T07:54:27.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:27.219+0000 7fc90c751640 1 -- start start 2024-09-16T07:54:27.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:27.221+0000 7fc90c751640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc9040a3450 0x7fc90411e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:54:27.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:27.221+0000 7fc90c751640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fc904121ed0 con 0x7fc9040a3450 2024-09-16T07:54:27.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:27.221+0000 7fc90a4ed640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc9040a3450 0x7fc90411e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:54:27.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:27.221+0000 7fc90a4ed640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc9040a3450 0x7fc90411e090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56284/0 (socket says 172.21.15.27:56284) 2024-09-16T07:54:27.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:27.221+0000 7fc90a4ed640 1 -- 172.21.15.27:0/3401621059 learned_addr learned my addr 172.21.15.27:0/3401621059 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-09-16T07:54:27.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:27.221+0000 7fc90a4ed640 1 -- 172.21.15.27:0/3401621059 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc904121aa0 con 0x7fc9040a3450 2024-09-16T07:54:27.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:27.221+0000 7fc90a4ed640 1 --2- 172.21.15.27:0/3401621059 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc9040a3450 0x7fc90411e090 crc :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:54:27.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:27.221+0000 7fc8faffd640 1 -- 172.21.15.27:0/3401621059 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fc900014620 con 0x7fc9040a3450 2024-09-16T07:54:27.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:27.222+0000 7fc8faffd640 1 -- 172.21.15.27:0/3401621059 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7fc900014780 con 0x7fc9040a3450 2024-09-16T07:54:27.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:27.222+0000 7fc8faffd640 1 -- 172.21.15.27:0/3401621059 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fc900014a20 con 0x7fc9040a3450 2024-09-16T07:54:27.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:27.222+0000 7fc90c751640 1 -- 172.21.15.27:0/3401621059 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9041358f0 con 0x7fc9040a3450 2024-09-16T07:54:27.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:27.222+0000 7fc90c751640 1 -- 172.21.15.27:0/3401621059 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc904121790 con 0x7fc9040a3450 2024-09-16T07:54:27.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:27.223+0000 7fc90c751640 1 -- 172.21.15.27:0/3401621059 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc90406a300 con 0x7fc9040a3450 2024-09-16T07:54:27.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:27.223+0000 7fc8faffd640 1 -- 172.21.15.27:0/3401621059 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) ==== 50954+0+0 (crc 0 0 0) 0x7fc900014bd0 con 0x7fc9040a3450 2024-09-16T07:54:27.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:27.223+0000 7fc8faffd640 1 --2- 172.21.15.27:0/3401621059 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7fc8e803e930 0x7fc8e8040e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:54:27.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:27.223+0000 7fc8faffd640 1 -- 172.21.15.27:0/3401621059 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(15..15 src has 1..15) ==== 2728+0+0 (crc 0 0 0) 0x7fc900019040 con 0x7fc9040a3450 2024-09-16T07:54:27.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:27.223+0000 7fc909cec640 1 --2- 172.21.15.27:0/3401621059 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7fc8e803e930 0x7fc8e8040e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:54:27.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:27.224+0000 7fc909cec640 1 --2- 172.21.15.27:0/3401621059 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7fc8e803e930 0x7fc8e8040e40 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:54:27.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:27.228+0000 7fc8faffd640 1 -- 172.21.15.27:0/3401621059 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (crc 0 0 0) 0x7fc900012580 con 0x7fc9040a3450 2024-09-16T07:54:27.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:27.460+0000 7fc90c751640 1 -- 172.21.15.27:0/3401621059 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7fc904120120 con 0x7fc9040a3450 2024-09-16T07:54:27.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:27.462+0000 7fc8faffd640 1 -- 172.21.15.27:0/3401621059 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) ==== 74+0+97907 (crc 0 0 0) 0x7fc900018190 con 0x7fc9040a3450 2024-09-16T07:54:27.463 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-16T07:54:27.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:27.464+0000 7fc90c751640 1 -- 172.21.15.27:0/3401621059 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7fc8e803e930 msgr2=0x7fc8e8040e40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:54:27.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:27.464+0000 7fc90c751640 1 --2- 172.21.15.27:0/3401621059 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7fc8e803e930 0x7fc8e8040e40 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:54:27.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:27.464+0000 7fc90c751640 1 -- 172.21.15.27:0/3401621059 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc9040a3450 msgr2=0x7fc90411e090 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:54:27.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:27.464+0000 7fc90c751640 1 --2- 172.21.15.27:0/3401621059 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc9040a3450 0x7fc90411e090 crc :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:54:27.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:27.464+0000 7fc90c751640 1 -- 172.21.15.27:0/3401621059 shutdown_connections 2024-09-16T07:54:27.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:27.464+0000 7fc90c751640 1 --2- 172.21.15.27:0/3401621059 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7fc8e803e930 0x7fc8e8040e40 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:54:27.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:27.464+0000 7fc90c751640 1 --2- 172.21.15.27:0/3401621059 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc9040a3450 0x7fc90411e090 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:54:27.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:27.464+0000 7fc90c751640 1 -- 172.21.15.27:0/3401621059 >> 172.21.15.27:0/3401621059 conn(0x7fc90406e490 msgr2=0x7fc9040a5840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:54:27.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:27.464+0000 7fc90c751640 1 -- 172.21.15.27:0/3401621059 shutdown_connections 2024-09-16T07:54:27.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:27.464+0000 7fc90c751640 1 -- 172.21.15.27:0/3401621059 wait complete. 2024-09-16T07:54:27.474 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":5,"flags":0,"active_gid":4102,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":4038049593},{"type":"v1","addr":"172.21.15.187:6801","nonce":4038049593}]},"active_addr":"172.21.15.187:6801/4038049593","active_change":"2024-09-16T07:54:17.075979+0000","active_mgr_features":4540719139924082687,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:sclorg-nginx-126","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.17","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"oauth2-proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba_metrics":{"name":"container_image_samba_metrics","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-metrics:latest","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB metrics exporter container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_count_max":{"name":"ssh_keepalive_count_max","type":"int","level":"advanced","flags":0,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"How many times ssh connections can fail liveness checks before the host is marked offline","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_interval":{"name":"ssh_keepalive_interval","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"How often ssh connections are checked for liveness","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.187:0","nonce":1501127669}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.187:0","nonce":772921577}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.187:0","nonce":1516721041}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.187:0","nonce":1588421439}]}]} 2024-09-16T07:54:27.476 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-09-16T07:54:27.476 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-09-16T07:54:27.476 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T07:54:27.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:27.586+0000 7f132769c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T07:54:27.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:27.591+0000 7f132769c640 1 Processor -- start 2024-09-16T07:54:27.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:27.592+0000 7f132769c640 1 -- start start 2024-09-16T07:54:27.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:27.593+0000 7f132769c640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f13201365c0 0x7f13201389e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:54:27.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:27.593+0000 7f132769c640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f13200745e0 con 0x7f1320130470 2024-09-16T07:54:27.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:27.593+0000 7f132769c640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 -- 0x7f13200740f0 con 0x7f13201365c0 2024-09-16T07:54:27.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:27.593+0000 7f1324c37640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f13201365c0 0x7f13201389e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:54:27.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:27.593+0000 7f1325438640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f1320130470 0x7f1320130880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:52624/0 (socket says 172.21.15.27:52624) 2024-09-16T07:54:27.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:27.593+0000 7f1324c37640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f13201365c0 0x7f13201389e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56288/0 (socket says 172.21.15.27:56288) 2024-09-16T07:54:27.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:27.593+0000 7f1325438640 1 -- 172.21.15.27:0/3611926083 learned_addr learned my addr 172.21.15.27:0/3611926083 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-09-16T07:54:27.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:27.594+0000 7f1324c37640 1 -- 172.21.15.27:0/3611926083 >> v1:172.21.15.27:6789/0 conn(0x7f1320130470 legacy=0x7f1320130880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:54:27.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:27.594+0000 7f1317fff640 1 -- 172.21.15.27:0/3611926083 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2022958874 0 0) 0x7f13200745e0 con 0x7f1320130470 2024-09-16T07:54:27.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:27.594+0000 7f1324c37640 1 -- 172.21.15.27:0/3611926083 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13200749a0 con 0x7f13201365c0 2024-09-16T07:54:27.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:27.594+0000 7f1324c37640 1 --2- 172.21.15.27:0/3611926083 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f13201365c0 0x7f13201389e0 crc :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d4bdb9a894e4c55 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:54:27.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:27.594+0000 7f1317fff640 1 -- 172.21.15.27:0/3611926083 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f1310009d90 con 0x7f13201365c0 2024-09-16T07:54:27.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:27.594+0000 7f1317fff640 1 -- 172.21.15.27:0/3611926083 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f131000e230 con 0x7f13201365c0 2024-09-16T07:54:27.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:27.594+0000 7f1317fff640 1 -- 172.21.15.27:0/3611926083 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f131000e4d0 con 0x7f13201365c0 2024-09-16T07:54:27.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:27.595+0000 7f132769c640 1 -- 172.21.15.27:0/3611926083 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f13201365c0 msgr2=0x7f13201389e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:54:27.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:27.595+0000 7f132769c640 1 --2- 172.21.15.27:0/3611926083 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f13201365c0 0x7f13201389e0 crc :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:54:27.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:27.595+0000 7f132769c640 1 -- 172.21.15.27:0/3611926083 shutdown_connections 2024-09-16T07:54:27.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:27.595+0000 7f132769c640 1 --2- 172.21.15.27:0/3611926083 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f13201365c0 0x7f13201389e0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:54:27.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:27.595+0000 7f132769c640 1 -- 172.21.15.27:0/3611926083 >> 172.21.15.27:0/3611926083 conn(0x7f13200690e0 msgr2=0x7f13200694f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:54:27.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:27.595+0000 7f132769c640 1 -- 172.21.15.27:0/3611926083 shutdown_connections 2024-09-16T07:54:27.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:27.595+0000 7f132769c640 1 -- 172.21.15.27:0/3611926083 wait complete. 2024-09-16T07:54:27.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:27.595+0000 7f132769c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T07:54:27.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:27.597+0000 7f132769c640 1 Processor -- start 2024-09-16T07:54:27.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:27.597+0000 7f132769c640 1 -- start start 2024-09-16T07:54:27.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:27.598+0000 7f132769c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1320130470 0x7f132011d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:54:27.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:27.598+0000 7f132769c640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f1320120a10 con 0x7f1320130470 2024-09-16T07:54:27.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:27.599+0000 7f1325438640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1320130470 0x7f132011d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:54:27.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:27.599+0000 7f1325438640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1320130470 0x7f132011d190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56290/0 (socket says 172.21.15.27:56290) 2024-09-16T07:54:27.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:27.599+0000 7f1325438640 1 -- 172.21.15.27:0/908478711 learned_addr learned my addr 172.21.15.27:0/908478711 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-09-16T07:54:27.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:27.599+0000 7f1325438640 1 -- 172.21.15.27:0/908478711 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f132011f2d0 con 0x7f1320130470 2024-09-16T07:54:27.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:27.599+0000 7f1325438640 1 --2- 172.21.15.27:0/908478711 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1320130470 0x7f132011d190 crc :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:54:27.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:27.599+0000 7f1315ffb640 1 -- 172.21.15.27:0/908478711 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f131c004bc0 con 0x7f1320130470 2024-09-16T07:54:27.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:27.599+0000 7f1315ffb640 1 -- 172.21.15.27:0/908478711 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f131c004d20 con 0x7f1320130470 2024-09-16T07:54:27.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:27.599+0000 7f132769c640 1 -- 172.21.15.27:0/908478711 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f132011cdf0 con 0x7f1320130470 2024-09-16T07:54:27.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:27.599+0000 7f1315ffb640 1 -- 172.21.15.27:0/908478711 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f131c007120 con 0x7f1320130470 2024-09-16T07:54:27.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:27.599+0000 7f132769c640 1 -- 172.21.15.27:0/908478711 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f132011ca00 con 0x7f1320130470 2024-09-16T07:54:27.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:27.600+0000 7f1315ffb640 1 -- 172.21.15.27:0/908478711 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) ==== 50954+0+0 (crc 0 0 0) 0x7f131c0072d0 con 0x7f1320130470 2024-09-16T07:54:27.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:27.600+0000 7f132769c640 1 -- 172.21.15.27:0/908478711 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f132006a300 con 0x7f1320130470 2024-09-16T07:54:27.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:27.600+0000 7f1315ffb640 1 --2- 172.21.15.27:0/908478711 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7f130403e930 0x7f1304040e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:54:27.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:27.601+0000 7f1315ffb640 1 -- 172.21.15.27:0/908478711 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(15..15 src has 1..15) ==== 2728+0+0 (crc 0 0 0) 0x7f131c04ffc0 con 0x7f1320130470 2024-09-16T07:54:27.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:27.601+0000 7f1324c37640 1 --2- 172.21.15.27:0/908478711 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7f130403e930 0x7f1304040e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:54:27.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:27.601+0000 7f1324c37640 1 --2- 172.21.15.27:0/908478711 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7f130403e930 0x7f1304040e40 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:54:27.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:27.606+0000 7f1315ffb640 1 -- 172.21.15.27:0/908478711 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (crc 0 0 0) 0x7f131c0066b0 con 0x7f1320130470 2024-09-16T07:54:27.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:27.793+0000 7f132769c640 1 -- 172.21.15.27:0/908478711 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f1320130880 con 0x7f1320130470 2024-09-16T07:54:27.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:27.793+0000 7f1315ffb640 1 -- 172.21.15.27:0/908478711 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v15) ==== 74+0+8867 (crc 0 0 0) 0x7f131c0060b0 con 0x7f1320130470 2024-09-16T07:54:27.791 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-16T07:54:27.792 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":15,"fsid":"16b22dd1-2c19-4258-b5fa-641ce257086f","created":"2024-09-16T07:54:12.578772+0000","modified":"2024-09-16T07:54:26.702664+0000","last_up_change":"2024-09-16T07:54:19.672678+0000","last_in_change":"2024-09-16T07:54:14.737256+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T07:54:21.103580+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-16T07:54:22.986244+0000","flags":270369,"flags_names":"hashpspool,nopgchange,selfmanaged_snaps,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":15,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9a91e4ff-5797-4c44-a7e4-c088c97e1063","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":250146213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6801","nonce":250146213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":250146213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6805","nonce":250146213}]},"public_addr":"172.21.15.27:6802/250146213","cluster_addr":"172.21.15.27:6801/250146213","heartbeat_back_addr":"172.21.15.27:6804/250146213","heartbeat_front_addr":"172.21.15.27:6805/250146213","state":["exists","up"]},{"osd":1,"uuid":"a1b27d22-cc01-4cc9-88c0-c28e8c60a99e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":584480168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6803","nonce":584480168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":584480168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6807","nonce":584480168}]},"public_addr":"172.21.15.27:6800/584480168","cluster_addr":"172.21.15.27:6803/584480168","heartbeat_back_addr":"172.21.15.27:6808/584480168","heartbeat_front_addr":"172.21.15.27:6807/584480168","state":["exists","up"]},{"osd":2,"uuid":"04f4b772-d2c6-4d73-894b-d5635c4d2dda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6809","nonce":4110688419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":4110688419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6807","nonce":4110688419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":4110688419}]},"public_addr":"172.21.15.187:6809/4110688419","cluster_addr":"172.21.15.187:6808/4110688419","heartbeat_back_addr":"172.21.15.187:6807/4110688419","heartbeat_front_addr":"172.21.15.187:6812/4110688419","state":["exists","up"]},{"osd":3,"uuid":"611aad1c-1d66-400c-a54d-b66c9579b6d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6811","nonce":1136904278}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":1136904278}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":1136904278}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6803","nonce":1136904278}]},"public_addr":"172.21.15.187:6811/1136904278","cluster_addr":"172.21.15.187:6810/1136904278","heartbeat_back_addr":"172.21.15.187:6804/1136904278","heartbeat_front_addr":"172.21.15.187:6803/1136904278","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T07:54:27.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:27.795+0000 7f132769c640 1 -- 172.21.15.27:0/908478711 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7f130403e930 msgr2=0x7f1304040e40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:54:27.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:27.795+0000 7f132769c640 1 --2- 172.21.15.27:0/908478711 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7f130403e930 0x7f1304040e40 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:54:27.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:27.795+0000 7f132769c640 1 -- 172.21.15.27:0/908478711 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1320130470 msgr2=0x7f132011d190 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:54:27.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:27.796+0000 7f132769c640 1 --2- 172.21.15.27:0/908478711 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1320130470 0x7f132011d190 crc :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:54:27.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:27.796+0000 7f132769c640 1 -- 172.21.15.27:0/908478711 shutdown_connections 2024-09-16T07:54:27.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:27.796+0000 7f132769c640 1 --2- 172.21.15.27:0/908478711 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7f130403e930 0x7f1304040e40 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:54:27.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:27.796+0000 7f132769c640 1 --2- 172.21.15.27:0/908478711 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1320130470 0x7f132011d190 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:54:27.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:27.796+0000 7f132769c640 1 -- 172.21.15.27:0/908478711 >> 172.21.15.27:0/908478711 conn(0x7f13200690e0 msgr2=0x7f13201343a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:54:27.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:27.796+0000 7f132769c640 1 -- 172.21.15.27:0/908478711 shutdown_connections 2024-09-16T07:54:27.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:27.796+0000 7f132769c640 1 -- 172.21.15.27:0/908478711 wait complete. 2024-09-16T07:54:27.805 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-09-16T07:54:27.805 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T07:54:27.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:27.915+0000 7fbce946c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T07:54:27.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:27.920+0000 7fbce946c640 1 Processor -- start 2024-09-16T07:54:27.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:27.920+0000 7fbce946c640 1 -- start start 2024-09-16T07:54:27.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:27.922+0000 7fbce946c640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fbce41365a0 0x7fbce41389c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:54:27.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:27.922+0000 7fbce946c640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbce4074a00 con 0x7fbce4130420 2024-09-16T07:54:27.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:27.922+0000 7fbce946c640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 -- 0x7fbce4074c30 con 0x7fbce41365a0 2024-09-16T07:54:27.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:27.922+0000 7fbce2ffd640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fbce4130420 0x7fbce4130830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:52634/0 (socket says 172.21.15.27:52634) 2024-09-16T07:54:27.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:27.922+0000 7fbce2ffd640 1 -- 172.21.15.27:0/2812599395 learned_addr learned my addr 172.21.15.27:0/2812599395 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-09-16T07:54:27.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:27.922+0000 7fbce27fc640 1 --2- 172.21.15.27:0/2812599395 >> v2:172.21.15.27:3300/0 conn(0x7fbce41365a0 0x7fbce41389c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:54:27.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:27.922+0000 7fbce27fc640 1 -- 172.21.15.27:0/2812599395 >> v1:172.21.15.27:6789/0 conn(0x7fbce4130420 legacy=0x7fbce4130830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:54:27.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:27.922+0000 7fbce27fc640 1 -- 172.21.15.27:0/2812599395 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbce4071930 con 0x7fbce41365a0 2024-09-16T07:54:27.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:27.922+0000 7fbce1ffb640 1 -- 172.21.15.27:0/2812599395 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2658576729 0 0) 0x7fbce4074a00 con 0x7fbce4130420 2024-09-16T07:54:27.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:27.923+0000 7fbce27fc640 1 --2- 172.21.15.27:0/2812599395 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbce41365a0 0x7fbce41389c0 crc :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f0ad2b882f9cec server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:54:27.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:27.923+0000 7fbce1ffb640 1 -- 172.21.15.27:0/2812599395 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fbccc009d90 con 0x7fbce41365a0 2024-09-16T07:54:27.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:27.923+0000 7fbce1ffb640 1 -- 172.21.15.27:0/2812599395 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7fbccc00e230 con 0x7fbce41365a0 2024-09-16T07:54:27.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:27.923+0000 7fbce1ffb640 1 -- 172.21.15.27:0/2812599395 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fbccc00e4d0 con 0x7fbce41365a0 2024-09-16T07:54:27.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:27.923+0000 7fbce946c640 1 -- 172.21.15.27:0/2812599395 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbce41365a0 msgr2=0x7fbce41389c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:54:27.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:27.923+0000 7fbce946c640 1 --2- 172.21.15.27:0/2812599395 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbce41365a0 0x7fbce41389c0 crc :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:54:27.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:27.923+0000 7fbce946c640 1 -- 172.21.15.27:0/2812599395 shutdown_connections 2024-09-16T07:54:27.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:27.923+0000 7fbce946c640 1 --2- 172.21.15.27:0/2812599395 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbce41365a0 0x7fbce41389c0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:54:27.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:27.923+0000 7fbce946c640 1 -- 172.21.15.27:0/2812599395 >> 172.21.15.27:0/2812599395 conn(0x7fbce40a0320 msgr2=0x7fbce40a0730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:54:27.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:27.924+0000 7fbce946c640 1 -- 172.21.15.27:0/2812599395 shutdown_connections 2024-09-16T07:54:27.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:27.924+0000 7fbce946c640 1 -- 172.21.15.27:0/2812599395 wait complete. 2024-09-16T07:54:27.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:27.924+0000 7fbce946c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T07:54:27.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:27.926+0000 7fbce946c640 1 Processor -- start 2024-09-16T07:54:27.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:27.926+0000 7fbce946c640 1 -- start start 2024-09-16T07:54:27.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:27.927+0000 7fbce946c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbce4130420 0x7fbce4120310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:54:27.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:27.927+0000 7fbce946c640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fbce4136190 con 0x7fbce4130420 2024-09-16T07:54:27.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:27.927+0000 7fbce2ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbce4130420 0x7fbce4120310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:54:27.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:27.927+0000 7fbce2ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbce4130420 0x7fbce4120310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56302/0 (socket says 172.21.15.27:56302) 2024-09-16T07:54:27.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:27.927+0000 7fbce2ffd640 1 -- 172.21.15.27:0/1664929624 learned_addr learned my addr 172.21.15.27:0/1664929624 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-09-16T07:54:27.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:27.928+0000 7fbce2ffd640 1 -- 172.21.15.27:0/1664929624 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbce41219f0 con 0x7fbce4130420 2024-09-16T07:54:27.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:27.928+0000 7fbce2ffd640 1 --2- 172.21.15.27:0/1664929624 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbce4130420 0x7fbce4120310 crc :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:54:27.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:27.928+0000 7fbcdb7fe640 1 -- 172.21.15.27:0/1664929624 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fbcd4004bc0 con 0x7fbce4130420 2024-09-16T07:54:27.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:27.928+0000 7fbcdb7fe640 1 -- 172.21.15.27:0/1664929624 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7fbcd4004d20 con 0x7fbce4130420 2024-09-16T07:54:27.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:27.928+0000 7fbce946c640 1 -- 172.21.15.27:0/1664929624 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbce4074260 con 0x7fbce4130420 2024-09-16T07:54:27.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:27.928+0000 7fbcdb7fe640 1 -- 172.21.15.27:0/1664929624 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fbcd4007120 con 0x7fbce4130420 2024-09-16T07:54:27.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:27.928+0000 7fbce946c640 1 -- 172.21.15.27:0/1664929624 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbce4122e80 con 0x7fbce4130420 2024-09-16T07:54:27.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:27.929+0000 7fbcdb7fe640 1 -- 172.21.15.27:0/1664929624 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) ==== 50954+0+0 (crc 0 0 0) 0x7fbcd40072d0 con 0x7fbce4130420 2024-09-16T07:54:27.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:27.929+0000 7fbce946c640 1 -- 172.21.15.27:0/1664929624 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbce406a300 con 0x7fbce4130420 2024-09-16T07:54:27.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:27.929+0000 7fbcdb7fe640 1 --2- 172.21.15.27:0/1664929624 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7fbcc403e960 0x7fbcc4040e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:54:27.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:27.929+0000 7fbcdb7fe640 1 -- 172.21.15.27:0/1664929624 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(15..15 src has 1..15) ==== 2728+0+0 (crc 0 0 0) 0x7fbcd404ffb0 con 0x7fbce4130420 2024-09-16T07:54:27.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:27.930+0000 7fbce27fc640 1 --2- 172.21.15.27:0/1664929624 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7fbcc403e960 0x7fbcc4040e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:54:27.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:27.930+0000 7fbce27fc640 1 --2- 172.21.15.27:0/1664929624 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7fbcc403e960 0x7fbcc4040e70 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:54:27.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:27.934+0000 7fbcdb7fe640 1 -- 172.21.15.27:0/1664929624 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (crc 0 0 0) 0x7fbcd40066b0 con 0x7fbce4130420 2024-09-16T07:54:28.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.122+0000 7fbce946c640 1 -- 172.21.15.27:0/1664929624 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fbce4130830 con 0x7fbce4130420 2024-09-16T07:54:28.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.123+0000 7fbcdb7fe640 1 -- 172.21.15.27:0/1664929624 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v15) ==== 74+0+8867 (crc 0 0 0) 0x7fbcd40060b0 con 0x7fbce4130420 2024-09-16T07:54:28.121 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-16T07:54:28.121 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":15,"fsid":"16b22dd1-2c19-4258-b5fa-641ce257086f","created":"2024-09-16T07:54:12.578772+0000","modified":"2024-09-16T07:54:26.702664+0000","last_up_change":"2024-09-16T07:54:19.672678+0000","last_in_change":"2024-09-16T07:54:14.737256+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T07:54:21.103580+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-16T07:54:22.986244+0000","flags":270369,"flags_names":"hashpspool,nopgchange,selfmanaged_snaps,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":15,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9a91e4ff-5797-4c44-a7e4-c088c97e1063","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":250146213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6801","nonce":250146213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":250146213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6805","nonce":250146213}]},"public_addr":"172.21.15.27:6802/250146213","cluster_addr":"172.21.15.27:6801/250146213","heartbeat_back_addr":"172.21.15.27:6804/250146213","heartbeat_front_addr":"172.21.15.27:6805/250146213","state":["exists","up"]},{"osd":1,"uuid":"a1b27d22-cc01-4cc9-88c0-c28e8c60a99e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":584480168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6803","nonce":584480168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":584480168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6807","nonce":584480168}]},"public_addr":"172.21.15.27:6800/584480168","cluster_addr":"172.21.15.27:6803/584480168","heartbeat_back_addr":"172.21.15.27:6808/584480168","heartbeat_front_addr":"172.21.15.27:6807/584480168","state":["exists","up"]},{"osd":2,"uuid":"04f4b772-d2c6-4d73-894b-d5635c4d2dda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6809","nonce":4110688419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":4110688419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6807","nonce":4110688419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":4110688419}]},"public_addr":"172.21.15.187:6809/4110688419","cluster_addr":"172.21.15.187:6808/4110688419","heartbeat_back_addr":"172.21.15.187:6807/4110688419","heartbeat_front_addr":"172.21.15.187:6812/4110688419","state":["exists","up"]},{"osd":3,"uuid":"611aad1c-1d66-400c-a54d-b66c9579b6d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6811","nonce":1136904278}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":1136904278}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":1136904278}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6803","nonce":1136904278}]},"public_addr":"172.21.15.187:6811/1136904278","cluster_addr":"172.21.15.187:6810/1136904278","heartbeat_back_addr":"172.21.15.187:6804/1136904278","heartbeat_front_addr":"172.21.15.187:6803/1136904278","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T07:54:28.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.125+0000 7fbce946c640 1 -- 172.21.15.27:0/1664929624 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7fbcc403e960 msgr2=0x7fbcc4040e70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:54:28.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.125+0000 7fbce946c640 1 --2- 172.21.15.27:0/1664929624 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7fbcc403e960 0x7fbcc4040e70 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:54:28.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.125+0000 7fbce946c640 1 -- 172.21.15.27:0/1664929624 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbce4130420 msgr2=0x7fbce4120310 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:54:28.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.125+0000 7fbce946c640 1 --2- 172.21.15.27:0/1664929624 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbce4130420 0x7fbce4120310 crc :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:54:28.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.125+0000 7fbce946c640 1 -- 172.21.15.27:0/1664929624 shutdown_connections 2024-09-16T07:54:28.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.125+0000 7fbce946c640 1 --2- 172.21.15.27:0/1664929624 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7fbcc403e960 0x7fbcc4040e70 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:54:28.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.125+0000 7fbce946c640 1 --2- 172.21.15.27:0/1664929624 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbce4130420 0x7fbce4120310 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:54:28.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.125+0000 7fbce946c640 1 -- 172.21.15.27:0/1664929624 >> 172.21.15.27:0/1664929624 conn(0x7fbce40a0320 msgr2=0x7fbce406f450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:54:28.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.126+0000 7fbce946c640 1 -- 172.21.15.27:0/1664929624 shutdown_connections 2024-09-16T07:54:28.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.126+0000 7fbce946c640 1 -- 172.21.15.27:0/1664929624 wait complete. 2024-09-16T07:54:28.135 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-16T07:54:28.135 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-09-16T07:54:28.136 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-16T07:54:28.136 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-09-16T07:54:28.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.257+0000 7f06da591640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T07:54:28.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.257+0000 7f4b33846640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T07:54:28.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.257+0000 7f26e3ac8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T07:54:28.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.258+0000 7f2c8ed91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T07:54:28.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.264+0000 7f06da591640 1 Processor -- start 2024-09-16T07:54:28.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.264+0000 7f06da591640 1 -- start start 2024-09-16T07:54:28.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.264+0000 7f26e3ac8640 1 Processor -- start 2024-09-16T07:54:28.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.264+0000 7f4b33846640 1 Processor -- start 2024-09-16T07:54:28.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.264+0000 7f26e3ac8640 1 -- start start 2024-09-16T07:54:28.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.264+0000 7f4b33846640 1 -- start start 2024-09-16T07:54:28.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.264+0000 7f2c8ed91640 1 Processor -- start 2024-09-16T07:54:28.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.265+0000 7f2c8ed91640 1 -- start start 2024-09-16T07:54:28.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.266+0000 7f06da591640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f06d4132520 0x7f06d4132930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:54:28.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.266+0000 7f06da591640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f06d406f6d0 con 0x7f06d40a3450 2024-09-16T07:54:28.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.266+0000 7f06da591640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 -- 0x7f06d406f3d0 con 0x7f06d4132520 2024-09-16T07:54:28.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.266+0000 7f06d3fff640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f06d4132520 0x7f06d4132930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:54:28.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.266+0000 7f06d3fff640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f06d4132520 0x7f06d4132930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56432/0 (socket says 172.21.15.27:56432) 2024-09-16T07:54:28.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.266+0000 7f06d3fff640 1 -- 172.21.15.27:0/2918536596 learned_addr learned my addr 172.21.15.27:0/2918536596 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-09-16T07:54:28.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.266+0000 7f26e3ac8640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f26dc135d80 0x7f26dc13a830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:54:28.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.266+0000 7f26e3ac8640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f26dc09cab0 con 0x7f26dc09bbc0 2024-09-16T07:54:28.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.266+0000 7f26e3ac8640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 -- 0x7f26dc09c140 con 0x7f26dc135d80 2024-09-16T07:54:28.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.266+0000 7f4b33846640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f4b2c138720 0x7f4b2c13ab40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:54:28.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.266+0000 7f26e1864640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f26dc09bbc0 0x7f26dc135670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:46570/0 (socket says 172.21.15.27:46570) 2024-09-16T07:54:28.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.266+0000 7f26e1063640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f26dc135d80 0x7f26dc13a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:54:28.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.266+0000 7f26e1864640 1 -- 172.21.15.27:0/2639167503 learned_addr learned my addr 172.21.15.27:0/2639167503 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-09-16T07:54:28.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.266+0000 7f4b33846640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4b2c133040 con 0x7f4b2c132520 2024-09-16T07:54:28.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.266+0000 7f4b33846640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 -- 0x7f4b2c06f6d0 con 0x7f4b2c138720 2024-09-16T07:54:28.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.266+0000 7f4b315e2640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f4b2c132520 0x7f4b2c132930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:46578/0 (socket says 172.21.15.27:46578) 2024-09-16T07:54:28.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.266+0000 7f4b315e2640 1 -- 172.21.15.27:0/1080053945 learned_addr learned my addr 172.21.15.27:0/1080053945 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-09-16T07:54:28.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.266+0000 7f2c8ed91640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2c880722d0 0x7f2c88138810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:54:28.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.266+0000 7f2c8ed91640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2c88072860 con 0x7f2c880717b0 2024-09-16T07:54:28.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.266+0000 7f2c8ed91640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 -- 0x7f2c880a1f50 con 0x7f2c880722d0 2024-09-16T07:54:28.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.266+0000 7f4b30de1640 1 --2- 172.21.15.27:0/1080053945 >> v2:172.21.15.27:3300/0 conn(0x7f4b2c138720 0x7f4b2c13ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:54:28.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.267+0000 7f26e1063640 1 -- 172.21.15.27:0/2639167503 >> v1:172.21.15.27:6789/0 conn(0x7f26dc09bbc0 legacy=0x7f26dc135670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:54:28.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.267+0000 7f26e1063640 1 -- 172.21.15.27:0/2639167503 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26dc09c750 con 0x7f26dc135d80 2024-09-16T07:54:28.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.267+0000 7f2c87fff640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2c880722d0 0x7f2c88138810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:54:28.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.267+0000 7f2c8cb2d640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f2c880717b0 0x7f2c88071bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:46588/0 (socket says 172.21.15.27:46588) 2024-09-16T07:54:28.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.267+0000 7f2c8cb2d640 1 -- 172.21.15.27:0/1389355035 learned_addr learned my addr 172.21.15.27:0/1389355035 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-09-16T07:54:28.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.267+0000 7f06d3fff640 1 -- 172.21.15.27:0/2918536596 >> v1:172.21.15.27:6789/0 conn(0x7f06d40a3450 legacy=0x7f06d40a3860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:54:28.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.267+0000 7f06d3fff640 1 -- 172.21.15.27:0/2918536596 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06d406f0c0 con 0x7f06d4132520 2024-09-16T07:54:28.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.267+0000 7f26e1063640 1 --2- 172.21.15.27:0/2639167503 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f26dc135d80 0x7f26dc13a830 crc :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a83c905a9f62c36b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:54:28.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.267+0000 7f2c87fff640 1 -- 172.21.15.27:0/1389355035 >> v1:172.21.15.27:6789/0 conn(0x7f2c880717b0 legacy=0x7f2c88071bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:54:28.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.267+0000 7f2c87fff640 1 -- 172.21.15.27:0/1389355035 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c88074760 con 0x7f2c880722d0 2024-09-16T07:54:28.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.267+0000 7f4b23fff640 1 -- 172.21.15.27:0/1080053945 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4171998243 0 0) 0x7f4b28001110 con 0x7f4b2c132520 2024-09-16T07:54:28.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.267+0000 7f4b23fff640 1 -- 172.21.15.27:0/1080053945 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4b18003580 con 0x7f4b2c132520 2024-09-16T07:54:28.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.267+0000 7f4b30de1640 1 -- 172.21.15.27:0/1080053945 >> v1:172.21.15.27:6789/0 conn(0x7f4b2c132520 legacy=0x7f4b2c132930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:54:28.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.267+0000 7f4b30de1640 1 -- 172.21.15.27:0/1080053945 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b2c06f3d0 con 0x7f4b2c138720 2024-09-16T07:54:28.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.267+0000 7f06d3fff640 1 --2- 172.21.15.27:0/2918536596 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f06d4132520 0x7f06d4132930 crc :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1331216db07c29a server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:54:28.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.267+0000 7f26e0862640 1 -- 172.21.15.27:0/2639167503 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f26cc009d90 con 0x7f26dc135d80 2024-09-16T07:54:28.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.267+0000 7f26e0862640 1 -- 172.21.15.27:0/2639167503 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f26cc00e230 con 0x7f26dc135d80 2024-09-16T07:54:28.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.267+0000 7f2c87fff640 1 --2- 172.21.15.27:0/1389355035 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2c880722d0 0x7f2c88138810 crc :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e430ddfcf7bc5e56 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:54:28.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.268+0000 7f26e0862640 1 -- 172.21.15.27:0/2639167503 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f26cc009d90 con 0x7f26dc135d80 2024-09-16T07:54:28.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.268+0000 7f4b30de1640 1 --2- 172.21.15.27:0/1080053945 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4b2c138720 0x7f4b2c13ab40 crc :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f498281890ba4c server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:54:28.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.268+0000 7f2c877fe640 1 -- 172.21.15.27:0/1389355035 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f2c74009d90 con 0x7f2c880722d0 2024-09-16T07:54:28.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.268+0000 7f06d2ffd640 1 -- 172.21.15.27:0/2918536596 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f06c4009e50 con 0x7f06d4132520 2024-09-16T07:54:28.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.268+0000 7f2c877fe640 1 -- 172.21.15.27:0/1389355035 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f2c7400e230 con 0x7f2c880722d0 2024-09-16T07:54:28.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.268+0000 7f2c877fe640 1 -- 172.21.15.27:0/1389355035 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f2c7400e4d0 con 0x7f2c880722d0 2024-09-16T07:54:28.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.268+0000 7f06d2ffd640 1 -- 172.21.15.27:0/2918536596 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f06c400e230 con 0x7f06d4132520 2024-09-16T07:54:28.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.268+0000 7f06d2ffd640 1 -- 172.21.15.27:0/2918536596 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f06c400e4d0 con 0x7f06d4132520 2024-09-16T07:54:28.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.268+0000 7f4b23fff640 1 -- 172.21.15.27:0/1080053945 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f4b1c009d90 con 0x7f4b2c138720 2024-09-16T07:54:28.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.268+0000 7f4b23fff640 1 -- 172.21.15.27:0/1080053945 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f4b1c00e230 con 0x7f4b2c138720 2024-09-16T07:54:28.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.268+0000 7f4b23fff640 1 -- 172.21.15.27:0/1080053945 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f4b1c00e4d0 con 0x7f4b2c138720 2024-09-16T07:54:28.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.269+0000 7f2c8ed91640 1 -- 172.21.15.27:0/1389355035 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2c880722d0 msgr2=0x7f2c88138810 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:54:28.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.269+0000 7f2c8ed91640 1 --2- 172.21.15.27:0/1389355035 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2c880722d0 0x7f2c88138810 crc :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:54:28.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.269+0000 7f06da591640 1 -- 172.21.15.27:0/2918536596 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f06d4132520 msgr2=0x7f06d4132930 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:54:28.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.269+0000 7f06da591640 1 --2- 172.21.15.27:0/2918536596 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f06d4132520 0x7f06d4132930 crc :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:54:28.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.269+0000 7f26e3ac8640 1 -- 172.21.15.27:0/2639167503 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f26dc135d80 msgr2=0x7f26dc13a830 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:54:28.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.269+0000 7f26e3ac8640 1 --2- 172.21.15.27:0/2639167503 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f26dc135d80 0x7f26dc13a830 crc :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:54:28.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.269+0000 7f06da591640 1 -- 172.21.15.27:0/2918536596 shutdown_connections 2024-09-16T07:54:28.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.269+0000 7f2c8ed91640 1 -- 172.21.15.27:0/1389355035 shutdown_connections 2024-09-16T07:54:28.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.269+0000 7f06da591640 1 --2- 172.21.15.27:0/2918536596 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f06d4132520 0x7f06d4132930 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:54:28.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.269+0000 7f2c8ed91640 1 --2- 172.21.15.27:0/1389355035 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2c880722d0 0x7f2c88138810 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:54:28.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.269+0000 7f2c8ed91640 1 -- 172.21.15.27:0/1389355035 >> 172.21.15.27:0/1389355035 conn(0x7f2c880a0320 msgr2=0x7f2c880a0730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:54:28.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.269+0000 7f06da591640 1 -- 172.21.15.27:0/2918536596 >> 172.21.15.27:0/2918536596 conn(0x7f06d406e490 msgr2=0x7f06d406e8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:54:28.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.269+0000 7f26e3ac8640 1 -- 172.21.15.27:0/2639167503 shutdown_connections 2024-09-16T07:54:28.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.269+0000 7f26e3ac8640 1 --2- 172.21.15.27:0/2639167503 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f26dc135d80 0x7f26dc13a830 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:54:28.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.269+0000 7f26e3ac8640 1 -- 172.21.15.27:0/2639167503 >> 172.21.15.27:0/2639167503 conn(0x7f26dc069cd0 msgr2=0x7f26dc0a27a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:54:28.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.269+0000 7f2c8ed91640 1 -- 172.21.15.27:0/1389355035 shutdown_connections 2024-09-16T07:54:28.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.269+0000 7f06da591640 1 -- 172.21.15.27:0/2918536596 shutdown_connections 2024-09-16T07:54:28.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.269+0000 7f2c8ed91640 1 -- 172.21.15.27:0/1389355035 wait complete. 2024-09-16T07:54:28.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.269+0000 7f06da591640 1 -- 172.21.15.27:0/2918536596 wait complete. 2024-09-16T07:54:28.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.269+0000 7f26e3ac8640 1 -- 172.21.15.27:0/2639167503 shutdown_connections 2024-09-16T07:54:28.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.269+0000 7f26e3ac8640 1 -- 172.21.15.27:0/2639167503 wait complete. 2024-09-16T07:54:28.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.269+0000 7f4b33846640 1 -- 172.21.15.27:0/1080053945 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4b2c138720 msgr2=0x7f4b2c13ab40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:54:28.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.269+0000 7f4b33846640 1 --2- 172.21.15.27:0/1080053945 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4b2c138720 0x7f4b2c13ab40 crc :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:54:28.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.269+0000 7f06da591640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T07:54:28.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.269+0000 7f2c8ed91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T07:54:28.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.269+0000 7f4b33846640 1 -- 172.21.15.27:0/1080053945 shutdown_connections 2024-09-16T07:54:28.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.269+0000 7f4b33846640 1 --2- 172.21.15.27:0/1080053945 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4b2c138720 0x7f4b2c13ab40 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:54:28.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.269+0000 7f4b33846640 1 -- 172.21.15.27:0/1080053945 >> 172.21.15.27:0/1080053945 conn(0x7f4b2c06e490 msgr2=0x7f4b2c06e8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:54:28.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.269+0000 7f26e3ac8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T07:54:28.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.270+0000 7f4b33846640 1 -- 172.21.15.27:0/1080053945 shutdown_connections 2024-09-16T07:54:28.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.270+0000 7f4b33846640 1 -- 172.21.15.27:0/1080053945 wait complete. 2024-09-16T07:54:28.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.270+0000 7f4b33846640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T07:54:28.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.271+0000 7f06da591640 1 Processor -- start 2024-09-16T07:54:28.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.271+0000 7f2c8ed91640 1 Processor -- start 2024-09-16T07:54:28.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.271+0000 7f2c8ed91640 1 -- start start 2024-09-16T07:54:28.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.272+0000 7f06da591640 1 -- start start 2024-09-16T07:54:28.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.272+0000 7f26e3ac8640 1 Processor -- start 2024-09-16T07:54:28.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.272+0000 7f26e3ac8640 1 -- start start 2024-09-16T07:54:28.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.273+0000 7f4b33846640 1 Processor -- start 2024-09-16T07:54:28.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.273+0000 7f2c8ed91640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2c880717b0 0x7f2c8811cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:54:28.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.273+0000 7f06da591640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f06d40a3450 0x7f06d411e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:54:28.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.273+0000 7f2c8ed91640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f2c8811dfd0 con 0x7f2c880717b0 2024-09-16T07:54:28.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.273+0000 7f4b33846640 1 -- start start 2024-09-16T07:54:28.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.273+0000 7f2c8cb2d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2c880717b0 0x7f2c8811cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:54:28.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.273+0000 7f06da591640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f06d4121f40 con 0x7f06d40a3450 2024-09-16T07:54:28.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.273+0000 7f2c8cb2d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2c880717b0 0x7f2c8811cc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56468/0 (socket says 172.21.15.27:56468) 2024-09-16T07:54:28.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.273+0000 7f2c8cb2d640 1 -- 172.21.15.27:0/2284962911 learned_addr learned my addr 172.21.15.27:0/2284962911 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-09-16T07:54:28.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.273+0000 7f06d3fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f06d40a3450 0x7f06d411e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:54:28.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.273+0000 7f06d3fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f06d40a3450 0x7f06d411e090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56480/0 (socket says 172.21.15.27:56480) 2024-09-16T07:54:28.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.273+0000 7f06d3fff640 1 -- 172.21.15.27:0/3060043060 learned_addr learned my addr 172.21.15.27:0/3060043060 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-09-16T07:54:28.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.274+0000 7f2c8cb2d640 1 -- 172.21.15.27:0/2284962911 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c88120bd0 con 0x7f2c880717b0 2024-09-16T07:54:28.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.274+0000 7f2c8cb2d640 1 --2- 172.21.15.27:0/2284962911 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2c880717b0 0x7f2c8811cc90 crc :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:54:28.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.274+0000 7f06d3fff640 1 -- 172.21.15.27:0/3060043060 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06d41320a0 con 0x7f06d40a3450 2024-09-16T07:54:28.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.274+0000 7f2c857fa640 1 -- 172.21.15.27:0/2284962911 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f2c7c005a60 con 0x7f2c880717b0 2024-09-16T07:54:28.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.274+0000 7f06d3fff640 1 --2- 172.21.15.27:0/3060043060 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f06d40a3450 0x7f06d411e090 crc :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:54:28.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.274+0000 7f2c857fa640 1 -- 172.21.15.27:0/2284962911 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f2c7c005bc0 con 0x7f2c880717b0 2024-09-16T07:54:28.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.274+0000 7f2c8ed91640 1 -- 172.21.15.27:0/2284962911 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2c8811e670 con 0x7f2c880717b0 2024-09-16T07:54:28.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.274+0000 7f2c8ed91640 1 -- 172.21.15.27:0/2284962911 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2c88130de0 con 0x7f2c880717b0 2024-09-16T07:54:28.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.274+0000 7f2c857fa640 1 -- 172.21.15.27:0/2284962911 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f2c7c005e60 con 0x7f2c880717b0 2024-09-16T07:54:28.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.274+0000 7f06d0ff9640 1 -- 172.21.15.27:0/3060043060 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f06c400edc0 con 0x7f06d40a3450 2024-09-16T07:54:28.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.274+0000 7f06d0ff9640 1 -- 172.21.15.27:0/3060043060 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f06c4015420 con 0x7f06d40a3450 2024-09-16T07:54:28.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.274+0000 7f06d0ff9640 1 -- 172.21.15.27:0/3060043060 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f06c40156c0 con 0x7f06d40a3450 2024-09-16T07:54:28.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.275+0000 7f06da591640 1 -- 172.21.15.27:0/3060043060 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06d4135960 con 0x7f06d40a3450 2024-09-16T07:54:28.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.275+0000 7f06da591640 1 -- 172.21.15.27:0/3060043060 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06d41219f0 con 0x7f06d40a3450 2024-09-16T07:54:28.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.275+0000 7f26e3ac8640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f26dc09bbc0 0x7f26dc11dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:54:28.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.275+0000 7f26e3ac8640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f26dc11eff0 con 0x7f26dc09bbc0 2024-09-16T07:54:28.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.275+0000 7f26e1864640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f26dc09bbc0 0x7f26dc11dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:54:28.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.275+0000 7f2c8ed91640 1 -- 172.21.15.27:0/2284962911 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f2c68000f80 con 0x7f2c880717b0 2024-09-16T07:54:28.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.275+0000 7f26e1864640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f26dc09bbc0 0x7f26dc11dcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56486/0 (socket says 172.21.15.27:56486) 2024-09-16T07:54:28.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.275+0000 7f26e1864640 1 -- 172.21.15.27:0/1676848044 learned_addr learned my addr 172.21.15.27:0/1676848044 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-09-16T07:54:28.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.276+0000 7f2c857fa640 1 -- 172.21.15.27:0/2284962911 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) ==== 50954+0+0 (crc 0 0 0) 0x7f2c7c00c7f0 con 0x7f2c880717b0 2024-09-16T07:54:28.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.276+0000 7f2c857fa640 1 --2- 172.21.15.27:0/2284962911 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7f2c6c03e930 0x7f2c6c040e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:54:28.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.276+0000 7f06da591640 1 -- 172.21.15.27:0/3060043060 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f06d409c830 con 0x7f06d40a3450 2024-09-16T07:54:28.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.276+0000 7f2c857fa640 1 -- 172.21.15.27:0/2284962911 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(15..15 src has 1..15) ==== 2728+0+0 (crc 0 0 0) 0x7f2c7c04f780 con 0x7f2c880717b0 2024-09-16T07:54:28.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.276+0000 7f2c857fa640 1 --2- 172.21.15.27:0/2284962911 >> v2:172.21.15.27:6800/584480168 conn(0x7f2c6c043930 0x7f2c6c045e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:54:28.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.276+0000 7f2c857fa640 1 -- 172.21.15.27:0/2284962911 --> v2:172.21.15.27:6800/584480168 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2c6c046490 con 0x7f2c6c043930 2024-09-16T07:54:28.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.276+0000 7f4b33846640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4b2c132520 0x7f4b2c11e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:54:28.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.276+0000 7f2c857fa640 1 -- 172.21.15.27:0/2284962911 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_get_version_reply(handle=1 version=15) ==== 24+0+0 (crc 0 0 0) 0x7f2c7c00dc10 con 0x7f2c880717b0 2024-09-16T07:54:28.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.276+0000 7f4b33846640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f4b2c121ee0 con 0x7f4b2c132520 2024-09-16T07:54:28.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.276+0000 7f26e1864640 1 -- 172.21.15.27:0/1676848044 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26dc121c50 con 0x7f26dc09bbc0 2024-09-16T07:54:28.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.276+0000 7f2c87fff640 1 --2- 172.21.15.27:0/2284962911 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7f2c6c03e930 0x7f2c6c040e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:54:28.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.277+0000 7f4b315e2640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4b2c132520 0x7f4b2c11e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:54:28.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.277+0000 7f4b315e2640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4b2c132520 0x7f4b2c11e090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56498/0 (socket says 172.21.15.27:56498) 2024-09-16T07:54:28.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.277+0000 7f4b315e2640 1 -- 172.21.15.27:0/3363195109 learned_addr learned my addr 172.21.15.27:0/3363195109 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-09-16T07:54:28.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.277+0000 7f26e1864640 1 --2- 172.21.15.27:0/1676848044 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f26dc09bbc0 0x7f26dc11dcb0 crc :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:54:28.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.277+0000 7f06d0ff9640 1 -- 172.21.15.27:0/3060043060 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) ==== 50954+0+0 (crc 0 0 0) 0x7f06c4015870 con 0x7f06d40a3450 2024-09-16T07:54:28.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.277+0000 7f06d0ff9640 1 --2- 172.21.15.27:0/3060043060 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7f06c003e930 0x7f06c0040e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:54:28.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.277+0000 7f4b315e2640 1 -- 172.21.15.27:0/3363195109 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b2c121ab0 con 0x7f4b2c132520 2024-09-16T07:54:28.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.277+0000 7f26d27fc640 1 -- 172.21.15.27:0/1676848044 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f26d8005a60 con 0x7f26dc09bbc0 2024-09-16T07:54:28.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.277+0000 7f26d27fc640 1 -- 172.21.15.27:0/1676848044 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f26d8005bc0 con 0x7f26dc09bbc0 2024-09-16T07:54:28.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.277+0000 7f26e3ac8640 1 -- 172.21.15.27:0/1676848044 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f26dc11f6f0 con 0x7f26dc09bbc0 2024-09-16T07:54:28.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.277+0000 7f06d0ff9640 1 -- 172.21.15.27:0/3060043060 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(15..15 src has 1..15) ==== 2728+0+0 (crc 0 0 0) 0x7f06c4050aa0 con 0x7f06d40a3450 2024-09-16T07:54:28.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.277+0000 7f26d27fc640 1 -- 172.21.15.27:0/1676848044 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f26d8005e60 con 0x7f26dc09bbc0 2024-09-16T07:54:28.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.277+0000 7f4b315e2640 1 --2- 172.21.15.27:0/3363195109 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4b2c132520 0x7f4b2c11e090 crc :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:54:28.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.277+0000 7f2c87fff640 1 --2- 172.21.15.27:0/2284962911 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7f2c6c03e930 0x7f2c6c040e40 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:54:28.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.278+0000 7f26e3ac8640 1 -- 172.21.15.27:0/1676848044 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26dc132e70 con 0x7f26dc09bbc0 2024-09-16T07:54:28.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.278+0000 7f2c8d32e640 1 --2- 172.21.15.27:0/2284962911 >> v2:172.21.15.27:6800/584480168 conn(0x7f2c6c043930 0x7f2c6c045e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-09-16T07:54:28.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.278+0000 7f4b21ffb640 1 -- 172.21.15.27:0/3363195109 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f4b28004a50 con 0x7f4b2c132520 2024-09-16T07:54:28.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.278+0000 7f4b21ffb640 1 -- 172.21.15.27:0/3363195109 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f4b28004bb0 con 0x7f4b2c132520 2024-09-16T07:54:28.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.278+0000 7f4b33846640 1 -- 172.21.15.27:0/3363195109 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b2c131370 con 0x7f4b2c132520 2024-09-16T07:54:28.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.278+0000 7f4b33846640 1 -- 172.21.15.27:0/3363195109 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b2c130f90 con 0x7f4b2c132520 2024-09-16T07:54:28.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.277+0000 7f06d0ff9640 1 --2- 172.21.15.27:0/3060043060 >> v2:172.21.15.27:6802/250146213 conn(0x7f06c0043930 0x7f06c0045e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:54:28.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.277+0000 7f06d0ff9640 1 -- 172.21.15.27:0/3060043060 --> v2:172.21.15.27:6802/250146213 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f06c0046490 con 0x7f06c0043930 2024-09-16T07:54:28.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.277+0000 7f06d0ff9640 1 -- 172.21.15.27:0/3060043060 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_get_version_reply(handle=1 version=15) ==== 24+0+0 (crc 0 0 0) 0x7f06c4050df0 con 0x7f06d40a3450 2024-09-16T07:54:28.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.277+0000 7f06d37fe640 1 --2- 172.21.15.27:0/3060043060 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7f06c003e930 0x7f06c0040e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:54:28.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.278+0000 7f06d37fe640 1 --2- 172.21.15.27:0/3060043060 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7f06c003e930 0x7f06c0040e40 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:54:28.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.279+0000 7f26e3ac8640 1 -- 172.21.15.27:0/1676848044 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f26dc09ed80 con 0x7f26dc09bbc0 2024-09-16T07:54:28.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.279+0000 7f26d27fc640 1 -- 172.21.15.27:0/1676848044 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) ==== 50954+0+0 (crc 0 0 0) 0x7f26d80070b0 con 0x7f26dc09bbc0 2024-09-16T07:54:28.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.280+0000 7f4b21ffb640 1 -- 172.21.15.27:0/3363195109 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f4b28004e50 con 0x7f4b2c132520 2024-09-16T07:54:28.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.280+0000 7f26d27fc640 1 --2- 172.21.15.27:0/1676848044 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7f26c003e9d0 0x7f26c0040ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:54:28.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.280+0000 7f26d27fc640 1 -- 172.21.15.27:0/1676848044 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(15..15 src has 1..15) ==== 2728+0+0 (crc 0 0 0) 0x7f26d804f640 con 0x7f26dc09bbc0 2024-09-16T07:54:28.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.280+0000 7f4b21ffb640 1 -- 172.21.15.27:0/3363195109 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) ==== 50954+0+0 (crc 0 0 0) 0x7f4b28007120 con 0x7f4b2c132520 2024-09-16T07:54:28.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.280+0000 7f4b33846640 1 -- 172.21.15.27:0/3363195109 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f4b0c000f80 con 0x7f4b2c132520 2024-09-16T07:54:28.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.280+0000 7f26d27fc640 1 --2- 172.21.15.27:0/1676848044 >> v2:172.21.15.187:6811/1136904278 conn(0x7f26c0043a50 0x7f26c0045f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:54:28.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.280+0000 7f4b21ffb640 1 --2- 172.21.15.27:0/3363195109 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7f4b1003e980 0x7f4b10040e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:54:28.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.280+0000 7f26d27fc640 1 -- 172.21.15.27:0/1676848044 --> v2:172.21.15.187:6811/1136904278 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f26c00465b0 con 0x7f26c0043a50 2024-09-16T07:54:28.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.280+0000 7f26d27fc640 1 -- 172.21.15.27:0/1676848044 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_get_version_reply(handle=1 version=15) ==== 24+0+0 (crc 0 0 0) 0x7f26d8051a60 con 0x7f26dc09bbc0 2024-09-16T07:54:28.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.280+0000 7f4b21ffb640 1 -- 172.21.15.27:0/3363195109 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(15..15 src has 1..15) ==== 2728+0+0 (crc 0 0 0) 0x7f4b2804f840 con 0x7f4b2c132520 2024-09-16T07:54:28.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.280+0000 7f26e1063640 1 --2- 172.21.15.27:0/1676848044 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7f26c003e9d0 0x7f26c0040ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:54:28.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.281+0000 7f4b21ffb640 1 --2- 172.21.15.27:0/3363195109 >> v2:172.21.15.187:6809/4110688419 conn(0x7f4b10043980 0x7f4b10045e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:54:28.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.281+0000 7f4b30de1640 1 --2- 172.21.15.27:0/3363195109 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7f4b1003e980 0x7f4b10040e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:54:28.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.281+0000 7f06d8b2e640 1 --2- 172.21.15.27:0/3060043060 >> v2:172.21.15.27:6802/250146213 conn(0x7f06c0043930 0x7f06c0045e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-09-16T07:54:28.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.281+0000 7f4b21ffb640 1 -- 172.21.15.27:0/3363195109 --> v2:172.21.15.187:6809/4110688419 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4b100464e0 con 0x7f4b10043980 2024-09-16T07:54:28.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.281+0000 7f26e1063640 1 --2- 172.21.15.27:0/1676848044 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7f26c003e9d0 0x7f26c0040ee0 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:54:28.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.281+0000 7f4b21ffb640 1 -- 172.21.15.27:0/3363195109 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_get_version_reply(handle=1 version=15) ==== 24+0+0 (crc 0 0 0) 0x7f4b28051bf0 con 0x7f4b2c132520 2024-09-16T07:54:28.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.281+0000 7f4b30de1640 1 --2- 172.21.15.27:0/3363195109 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7f4b1003e980 0x7f4b10040e90 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:54:28.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.282+0000 7f26e2065640 1 --2- 172.21.15.27:0/1676848044 >> v2:172.21.15.187:6811/1136904278 conn(0x7f26c0043a50 0x7f26c0045f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-09-16T07:54:28.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.282+0000 7f4b31de3640 1 --2- 172.21.15.27:0/3363195109 >> v2:172.21.15.187:6809/4110688419 conn(0x7f4b10043980 0x7f4b10045e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-09-16T07:54:28.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.283+0000 7f2c8d32e640 1 --2- 172.21.15.27:0/2284962911 >> v2:172.21.15.27:6800/584480168 conn(0x7f2c6c043930 0x7f2c6c045e00 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:54:28.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.284+0000 7f06d8b2e640 1 --2- 172.21.15.27:0/3060043060 >> v2:172.21.15.27:6802/250146213 conn(0x7f06c0043930 0x7f06c0045e00 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:54:28.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.285+0000 7f2c857fa640 1 -- 172.21.15.27:0/2284962911 <== osd.1 v2:172.21.15.27:6800/584480168 1 ==== command_reply(tid 1: 0 ) ==== 8+0+5790 (crc 0 0 0) 0x7f2c6c046490 con 0x7f2c6c043930 2024-09-16T07:54:28.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.285+0000 7f26e2065640 1 --2- 172.21.15.27:0/1676848044 >> v2:172.21.15.187:6811/1136904278 conn(0x7f26c0043a50 0x7f26c0045f20 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:54:28.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.286+0000 7f06d0ff9640 1 -- 172.21.15.27:0/3060043060 <== osd.0 v2:172.21.15.27:6802/250146213 1 ==== command_reply(tid 1: 0 ) ==== 8+0+5790 (crc 0 0 0) 0x7f06c0046490 con 0x7f06c0043930 2024-09-16T07:54:28.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.286+0000 7f4b31de3640 1 --2- 172.21.15.27:0/3363195109 >> v2:172.21.15.187:6809/4110688419 conn(0x7f4b10043980 0x7f4b10045e50 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:54:28.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.287+0000 7f26d27fc640 1 -- 172.21.15.27:0/1676848044 <== osd.3 v2:172.21.15.187:6811/1136904278 1 ==== command_reply(tid 1: 0 ) ==== 8+0+5790 (crc 0 0 0) 0x7f26c00465b0 con 0x7f26c0043a50 2024-09-16T07:54:28.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.288+0000 7f4b21ffb640 1 -- 172.21.15.27:0/3363195109 <== osd.2 v2:172.21.15.187:6809/4110688419 1 ==== command_reply(tid 1: 0 ) ==== 8+0+5790 (crc 0 0 0) 0x7f4b100464e0 con 0x7f4b10043980 2024-09-16T07:54:28.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.290+0000 7f2c8ed91640 1 -- 172.21.15.27:0/2284962911 --> v2:172.21.15.27:6800/584480168 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f2c68002c30 con 0x7f2c6c043930 2024-09-16T07:54:28.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.290+0000 7f06da591640 1 -- 172.21.15.27:0/3060043060 --> v2:172.21.15.27:6802/250146213 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f06d406a300 con 0x7f06c0043930 2024-09-16T07:54:28.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.292+0000 7f06d0ff9640 1 -- 172.21.15.27:0/3060043060 <== osd.0 v2:172.21.15.27:6802/250146213 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f06d406a300 con 0x7f06c0043930 2024-09-16T07:54:28.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.292+0000 7f2c857fa640 1 -- 172.21.15.27:0/2284962911 <== osd.1 v2:172.21.15.27:6800/584480168 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f2c68002c30 con 0x7f2c6c043930 2024-09-16T07:54:28.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.293+0000 7f2c8ed91640 1 -- 172.21.15.27:0/2284962911 >> v2:172.21.15.27:6800/584480168 conn(0x7f2c6c043930 msgr2=0x7f2c6c045e00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:54:28.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.293+0000 7f06ba7fc640 1 -- 172.21.15.27:0/3060043060 >> v2:172.21.15.27:6802/250146213 conn(0x7f06c0043930 msgr2=0x7f06c0045e00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:54:28.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.293+0000 7f2c8ed91640 1 --2- 172.21.15.27:0/2284962911 >> v2:172.21.15.27:6800/584480168 conn(0x7f2c6c043930 0x7f2c6c045e00 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:54:28.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.293+0000 7f06ba7fc640 1 --2- 172.21.15.27:0/3060043060 >> v2:172.21.15.27:6802/250146213 conn(0x7f06c0043930 0x7f06c0045e00 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:54:28.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.293+0000 7f4b33846640 1 -- 172.21.15.27:0/3363195109 --> v2:172.21.15.187:6809/4110688419 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f4b0c002c30 con 0x7f4b10043980 2024-09-16T07:54:28.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.293+0000 7f06ba7fc640 1 -- 172.21.15.27:0/3060043060 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7f06c003e930 msgr2=0x7f06c0040e40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:54:28.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.293+0000 7f06ba7fc640 1 --2- 172.21.15.27:0/3060043060 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7f06c003e930 0x7f06c0040e40 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:54:28.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.293+0000 7f2c8ed91640 1 -- 172.21.15.27:0/2284962911 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7f2c6c03e930 msgr2=0x7f2c6c040e40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:54:28.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.293+0000 7f2c8ed91640 1 --2- 172.21.15.27:0/2284962911 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7f2c6c03e930 0x7f2c6c040e40 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:54:28.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.293+0000 7f2c8ed91640 1 -- 172.21.15.27:0/2284962911 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2c880717b0 msgr2=0x7f2c8811cc90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:54:28.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.293+0000 7f2c8ed91640 1 --2- 172.21.15.27:0/2284962911 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2c880717b0 0x7f2c8811cc90 crc :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:54:28.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.293+0000 7f06ba7fc640 1 -- 172.21.15.27:0/3060043060 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f06d40a3450 msgr2=0x7f06d411e090 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:54:28.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.293+0000 7f06ba7fc640 1 --2- 172.21.15.27:0/3060043060 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f06d40a3450 0x7f06d411e090 crc :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:54:28.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.293+0000 7f06ba7fc640 1 -- 172.21.15.27:0/3060043060 shutdown_connections 2024-09-16T07:54:28.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.293+0000 7f06ba7fc640 1 --2- 172.21.15.27:0/3060043060 >> v2:172.21.15.27:6802/250146213 conn(0x7f06c0043930 0x7f06c0045e00 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:54:28.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.293+0000 7f06ba7fc640 1 --2- 172.21.15.27:0/3060043060 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7f06c003e930 0x7f06c0040e40 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:54:28.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.293+0000 7f06ba7fc640 1 --2- 172.21.15.27:0/3060043060 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f06d40a3450 0x7f06d411e090 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:54:28.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.293+0000 7f06ba7fc640 1 -- 172.21.15.27:0/3060043060 >> 172.21.15.27:0/3060043060 conn(0x7f06d406e490 msgr2=0x7f06d41bd520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:54:28.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.293+0000 7f2c8ed91640 1 -- 172.21.15.27:0/2284962911 shutdown_connections 2024-09-16T07:54:28.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.293+0000 7f2c8ed91640 1 --2- 172.21.15.27:0/2284962911 >> v2:172.21.15.27:6800/584480168 conn(0x7f2c6c043930 0x7f2c6c045e00 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:54:28.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.293+0000 7f06ba7fc640 1 -- 172.21.15.27:0/3060043060 shutdown_connections 2024-09-16T07:54:28.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.293+0000 7f2c8ed91640 1 --2- 172.21.15.27:0/2284962911 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7f2c6c03e930 0x7f2c6c040e40 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:54:28.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.293+0000 7f2c8ed91640 1 --2- 172.21.15.27:0/2284962911 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2c880717b0 0x7f2c8811cc90 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:54:28.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.293+0000 7f2c8ed91640 1 -- 172.21.15.27:0/2284962911 >> 172.21.15.27:0/2284962911 conn(0x7f2c880a0320 msgr2=0x7f2c880a36b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:54:28.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.293+0000 7f06ba7fc640 1 -- 172.21.15.27:0/3060043060 wait complete. 2024-09-16T07:54:28.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.293+0000 7f2c8ed91640 1 -- 172.21.15.27:0/2284962911 shutdown_connections 2024-09-16T07:54:28.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.294+0000 7f2c8ed91640 1 -- 172.21.15.27:0/2284962911 wait complete. 2024-09-16T07:54:28.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.294+0000 7f26e3ac8640 1 -- 172.21.15.27:0/1676848044 --> v2:172.21.15.187:6811/1136904278 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f26dc09e6e0 con 0x7f26c0043a50 2024-09-16T07:54:28.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.295+0000 7f4b21ffb640 1 -- 172.21.15.27:0/3363195109 <== osd.2 v2:172.21.15.187:6809/4110688419 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f4b0c002c30 con 0x7f4b10043980 2024-09-16T07:54:28.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.295+0000 7f4b33846640 1 -- 172.21.15.27:0/3363195109 >> v2:172.21.15.187:6809/4110688419 conn(0x7f4b10043980 msgr2=0x7f4b10045e50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:54:28.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.295+0000 7f4b33846640 1 --2- 172.21.15.27:0/3363195109 >> v2:172.21.15.187:6809/4110688419 conn(0x7f4b10043980 0x7f4b10045e50 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:54:28.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.295+0000 7f4b33846640 1 -- 172.21.15.27:0/3363195109 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7f4b1003e980 msgr2=0x7f4b10040e90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:54:28.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.295+0000 7f4b33846640 1 --2- 172.21.15.27:0/3363195109 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7f4b1003e980 0x7f4b10040e90 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:54:28.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.295+0000 7f4b33846640 1 -- 172.21.15.27:0/3363195109 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4b2c132520 msgr2=0x7f4b2c11e090 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:54:28.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.295+0000 7f4b33846640 1 --2- 172.21.15.27:0/3363195109 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4b2c132520 0x7f4b2c11e090 crc :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:54:28.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.296+0000 7f4b33846640 1 -- 172.21.15.27:0/3363195109 shutdown_connections 2024-09-16T07:54:28.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.296+0000 7f4b33846640 1 --2- 172.21.15.27:0/3363195109 >> v2:172.21.15.187:6809/4110688419 conn(0x7f4b10043980 0x7f4b10045e50 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:54:28.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.296+0000 7f4b33846640 1 --2- 172.21.15.27:0/3363195109 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7f4b1003e980 0x7f4b10040e90 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:54:28.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.296+0000 7f4b33846640 1 --2- 172.21.15.27:0/3363195109 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4b2c132520 0x7f4b2c11e090 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:54:28.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.296+0000 7f4b33846640 1 -- 172.21.15.27:0/3363195109 >> 172.21.15.27:0/3363195109 conn(0x7f4b2c06e490 msgr2=0x7f4b2c0a5840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:54:28.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.296+0000 7f4b33846640 1 -- 172.21.15.27:0/3363195109 shutdown_connections 2024-09-16T07:54:28.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.296+0000 7f4b33846640 1 -- 172.21.15.27:0/3363195109 wait complete. 2024-09-16T07:54:28.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.296+0000 7f26d27fc640 1 -- 172.21.15.27:0/1676848044 <== osd.3 v2:172.21.15.187:6811/1136904278 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f26dc09e6e0 con 0x7f26c0043a50 2024-09-16T07:54:28.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.296+0000 7f26e3ac8640 1 -- 172.21.15.27:0/1676848044 >> v2:172.21.15.187:6811/1136904278 conn(0x7f26c0043a50 msgr2=0x7f26c0045f20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:54:28.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.296+0000 7f26e3ac8640 1 --2- 172.21.15.27:0/1676848044 >> v2:172.21.15.187:6811/1136904278 conn(0x7f26c0043a50 0x7f26c0045f20 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:54:28.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.296+0000 7f26e3ac8640 1 -- 172.21.15.27:0/1676848044 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7f26c003e9d0 msgr2=0x7f26c0040ee0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:54:28.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.296+0000 7f26e3ac8640 1 --2- 172.21.15.27:0/1676848044 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7f26c003e9d0 0x7f26c0040ee0 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:54:28.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.296+0000 7f26e3ac8640 1 -- 172.21.15.27:0/1676848044 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f26dc09bbc0 msgr2=0x7f26dc11dcb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:54:28.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.296+0000 7f26e3ac8640 1 --2- 172.21.15.27:0/1676848044 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f26dc09bbc0 0x7f26dc11dcb0 crc :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:54:28.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.296+0000 7f26e3ac8640 1 -- 172.21.15.27:0/1676848044 shutdown_connections 2024-09-16T07:54:28.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.296+0000 7f26e3ac8640 1 --2- 172.21.15.27:0/1676848044 >> v2:172.21.15.187:6811/1136904278 conn(0x7f26c0043a50 0x7f26c0045f20 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:54:28.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.296+0000 7f26e3ac8640 1 --2- 172.21.15.27:0/1676848044 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7f26c003e9d0 0x7f26c0040ee0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:54:28.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.296+0000 7f26e3ac8640 1 --2- 172.21.15.27:0/1676848044 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f26dc09bbc0 0x7f26dc11dcb0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:54:28.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.296+0000 7f26e3ac8640 1 -- 172.21.15.27:0/1676848044 >> 172.21.15.27:0/1676848044 conn(0x7f26dc069cd0 msgr2=0x7f26dc1365e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:54:28.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.296+0000 7f26e3ac8640 1 -- 172.21.15.27:0/1676848044 shutdown_connections 2024-09-16T07:54:28.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.296+0000 7f26e3ac8640 1 -- 172.21.15.27:0/1676848044 wait complete. 2024-09-16T07:54:28.322 INFO:teuthology.orchestra.run.smithi027.stdout:38654705672 2024-09-16T07:54:28.322 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-16T07:54:28.323 INFO:teuthology.orchestra.run.smithi027.stdout:38654705672 2024-09-16T07:54:28.323 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-16T07:54:28.324 INFO:teuthology.orchestra.run.smithi027.stdout:38654705672 2024-09-16T07:54:28.325 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-16T07:54:28.325 INFO:teuthology.orchestra.run.smithi027.stdout:38654705672 2024-09-16T07:54:28.326 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-16T07:54:28.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.484+0000 7fdb4f191640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T07:54:28.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.484+0000 7f6021f91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T07:54:28.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.484+0000 7fc0c6791640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T07:54:28.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.485+0000 7f186ef91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T07:54:28.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.491+0000 7fdb4f191640 1 Processor -- start 2024-09-16T07:54:28.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.491+0000 7f6021f91640 1 Processor -- start 2024-09-16T07:54:28.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.491+0000 7fc0c6791640 1 Processor -- start 2024-09-16T07:54:28.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.491+0000 7fdb4f191640 1 -- start start 2024-09-16T07:54:28.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.491+0000 7fc0c6791640 1 -- start start 2024-09-16T07:54:28.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.491+0000 7f6021f91640 1 -- start start 2024-09-16T07:54:28.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.491+0000 7f186ef91640 1 Processor -- start 2024-09-16T07:54:28.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.491+0000 7f186ef91640 1 -- start start 2024-09-16T07:54:28.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.492+0000 7f6021f91640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f601c132520 0x7f601c132930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:54:28.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.492+0000 7f6021f91640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f601c06f6d0 con 0x7f601c0a3450 2024-09-16T07:54:28.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.492+0000 7f6021f91640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 -- 0x7f601c06f3d0 con 0x7f601c132520 2024-09-16T07:54:28.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.492+0000 7fdb4f191640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fdb48138720 0x7fdb4813ab40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:54:28.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.492+0000 7fdb4f191640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdb48133040 con 0x7fdb48132520 2024-09-16T07:54:28.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.492+0000 7fdb4f191640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 -- 0x7fdb4806f6d0 con 0x7fdb48138720 2024-09-16T07:54:28.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.493+0000 7fc0c6791640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc0c009daa0 0x7fc0c009ca60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:54:28.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.493+0000 7fc0c6791640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc0c012f190 con 0x7fc0c009cfa0 2024-09-16T07:54:28.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.493+0000 7fc0c6791640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 -- 0x7fc0c00706a0 con 0x7fc0c009daa0 2024-09-16T07:54:28.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.493+0000 7fdb4cf2d640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fdb48132520 0x7fdb48132930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:46594/0 (socket says 172.21.15.27:46594) 2024-09-16T07:54:28.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.493+0000 7fdb47fff640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fdb48138720 0x7fdb4813ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:54:28.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.493+0000 7f601affd640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f601c0a3450 0x7f601c0a3860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:46608/0 (socket says 172.21.15.27:46608) 2024-09-16T07:54:28.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.493+0000 7fdb4cf2d640 1 -- 172.21.15.27:0/2201642031 learned_addr learned my addr 172.21.15.27:0/2201642031 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-09-16T07:54:28.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.493+0000 7f601affd640 1 -- 172.21.15.27:0/2219228454 learned_addr learned my addr 172.21.15.27:0/2219228454 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-09-16T07:54:28.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.493+0000 7fc0bf7fe640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fc0c009cfa0 0x7fc0c0131770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:46620/0 (socket says 172.21.15.27:46620) 2024-09-16T07:54:28.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.493+0000 7fc0bf7fe640 1 -- 172.21.15.27:0/3019950773 learned_addr learned my addr 172.21.15.27:0/3019950773 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-09-16T07:54:28.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.493+0000 7f601b7fe640 1 --2- 172.21.15.27:0/2219228454 >> v2:172.21.15.27:3300/0 conn(0x7f601c132520 0x7f601c132930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:54:28.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.493+0000 7fc0bffff640 1 --2- 172.21.15.27:0/3019950773 >> v2:172.21.15.27:3300/0 conn(0x7fc0c009daa0 0x7fc0c009ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:54:28.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.493+0000 7f186ef91640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f18680717b0 0x7f1868071bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:54:28.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.493+0000 7f601b7fe640 1 -- 172.21.15.27:0/2219228454 >> v1:172.21.15.27:6789/0 conn(0x7f601c0a3450 legacy=0x7f601c0a3860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:54:28.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.493+0000 7f601b7fe640 1 -- 172.21.15.27:0/2219228454 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f601c06f0c0 con 0x7f601c132520 2024-09-16T07:54:28.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.493+0000 7f186ef91640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1868072860 con 0x7f1868072100 2024-09-16T07:54:28.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.493+0000 7f186ef91640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 -- 0x7f18680a1f50 con 0x7f18680717b0 2024-09-16T07:54:28.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.493+0000 7f186cd2d640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f18680717b0 0x7f1868071bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:54:28.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.493+0000 7f186cd2d640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f18680717b0 0x7f1868071bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56522/0 (socket says 172.21.15.27:56522) 2024-09-16T07:54:28.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.493+0000 7f1867fff640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f1868072100 0x7f1868130040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:46628/0 (socket says 172.21.15.27:46628) 2024-09-16T07:54:28.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.493+0000 7f186cd2d640 1 -- 172.21.15.27:0/3396515529 learned_addr learned my addr 172.21.15.27:0/3396515529 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-09-16T07:54:28.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.493+0000 7f601b7fe640 1 --2- 172.21.15.27:0/2219228454 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f601c132520 0x7f601c132930 crc :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ea26e4cb9671366 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:54:28.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.493+0000 7fdb477fe640 1 -- 172.21.15.27:0/2201642031 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 980086065 0 0) 0x7fdb3c001110 con 0x7fdb48132520 2024-09-16T07:54:28.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.494+0000 7fc0bffff640 1 -- 172.21.15.27:0/3019950773 >> v1:172.21.15.27:6789/0 conn(0x7fc0c009cfa0 legacy=0x7fc0c0131770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:54:28.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.494+0000 7fdb477fe640 1 -- 172.21.15.27:0/2201642031 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdb38003580 con 0x7fdb48132520 2024-09-16T07:54:28.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.494+0000 7fc0bffff640 1 -- 172.21.15.27:0/3019950773 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0c012e970 con 0x7fc0c009daa0 2024-09-16T07:54:28.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.494+0000 7fdb47fff640 1 -- 172.21.15.27:0/2201642031 >> v1:172.21.15.27:6789/0 conn(0x7fdb48132520 legacy=0x7fdb48132930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:54:28.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.494+0000 7fdb47fff640 1 -- 172.21.15.27:0/2201642031 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb4806f3d0 con 0x7fdb48138720 2024-09-16T07:54:28.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.494+0000 7f186cd2d640 1 -- 172.21.15.27:0/3396515529 >> v1:172.21.15.27:6789/0 conn(0x7f1868072100 legacy=0x7f1868130040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:54:28.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.494+0000 7f18677fe640 1 -- 172.21.15.27:0/3396515529 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3528627638 0 0) 0x7f1868072860 con 0x7f1868072100 2024-09-16T07:54:28.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.494+0000 7f186cd2d640 1 -- 172.21.15.27:0/3396515529 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1868074760 con 0x7f18680717b0 2024-09-16T07:54:28.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.494+0000 7f601a7fc640 1 -- 172.21.15.27:0/2219228454 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f600c009d90 con 0x7f601c132520 2024-09-16T07:54:28.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.494+0000 7f601a7fc640 1 -- 172.21.15.27:0/2219228454 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f600c00e230 con 0x7f601c132520 2024-09-16T07:54:28.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.494+0000 7fdb47fff640 1 --2- 172.21.15.27:0/2201642031 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb48138720 0x7fdb4813ab40 crc :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cab999999343534e server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:54:28.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.494+0000 7fc0bffff640 1 --2- 172.21.15.27:0/3019950773 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc0c009daa0 0x7fc0c009ca60 crc :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8118ab14f0af417 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:54:28.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.494+0000 7fc0beffd640 1 -- 172.21.15.27:0/3019950773 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fc0b0009d90 con 0x7fc0c009daa0 2024-09-16T07:54:28.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.494+0000 7fdb477fe640 1 -- 172.21.15.27:0/2201642031 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fdb34009d90 con 0x7fdb48138720 2024-09-16T07:54:28.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.494+0000 7f186cd2d640 1 --2- 172.21.15.27:0/3396515529 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f18680717b0 0x7f1868071bc0 crc :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bdd50f1b6654405c server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:54:28.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.494+0000 7fc0beffd640 1 -- 172.21.15.27:0/3019950773 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7fc0b000e230 con 0x7fc0c009daa0 2024-09-16T07:54:28.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.494+0000 7fdb477fe640 1 -- 172.21.15.27:0/2201642031 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7fdb3400e230 con 0x7fdb48138720 2024-09-16T07:54:28.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.494+0000 7fdb477fe640 1 -- 172.21.15.27:0/2201642031 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fdb34009d90 con 0x7fdb48138720 2024-09-16T07:54:28.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.494+0000 7f18677fe640 1 -- 172.21.15.27:0/3396515529 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f185c009e50 con 0x7f18680717b0 2024-09-16T07:54:28.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.494+0000 7f18677fe640 1 -- 172.21.15.27:0/3396515529 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f185c00e230 con 0x7f18680717b0 2024-09-16T07:54:28.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.494+0000 7f18677fe640 1 -- 172.21.15.27:0/3396515529 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f185c00e4d0 con 0x7f18680717b0 2024-09-16T07:54:28.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.494+0000 7f601a7fc640 1 -- 172.21.15.27:0/2219228454 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f600c00e4d0 con 0x7f601c132520 2024-09-16T07:54:28.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.495+0000 7f6021f91640 1 -- 172.21.15.27:0/2219228454 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f601c132520 msgr2=0x7f601c132930 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:54:28.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.495+0000 7f6021f91640 1 --2- 172.21.15.27:0/2219228454 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f601c132520 0x7f601c132930 crc :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:54:28.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.495+0000 7f6021f91640 1 -- 172.21.15.27:0/2219228454 shutdown_connections 2024-09-16T07:54:28.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.495+0000 7fdb4f191640 1 -- 172.21.15.27:0/2201642031 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb48138720 msgr2=0x7fdb4813ab40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:54:28.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.495+0000 7fdb4f191640 1 --2- 172.21.15.27:0/2201642031 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb48138720 0x7fdb4813ab40 crc :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:54:28.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.495+0000 7f6021f91640 1 --2- 172.21.15.27:0/2219228454 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f601c132520 0x7f601c132930 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:54:28.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.495+0000 7f186ef91640 1 -- 172.21.15.27:0/3396515529 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f18680717b0 msgr2=0x7f1868071bc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:54:28.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.495+0000 7f6021f91640 1 -- 172.21.15.27:0/2219228454 >> 172.21.15.27:0/2219228454 conn(0x7f601c06e490 msgr2=0x7f601c06e8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:54:28.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.495+0000 7f186ef91640 1 --2- 172.21.15.27:0/3396515529 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f18680717b0 0x7f1868071bc0 crc :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:54:28.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.495+0000 7fc0beffd640 1 -- 172.21.15.27:0/3019950773 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fc0b0009d90 con 0x7fc0c009daa0 2024-09-16T07:54:28.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.495+0000 7f6021f91640 1 -- 172.21.15.27:0/2219228454 shutdown_connections 2024-09-16T07:54:28.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.495+0000 7fdb4f191640 1 -- 172.21.15.27:0/2201642031 shutdown_connections 2024-09-16T07:54:28.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.495+0000 7f186ef91640 1 -- 172.21.15.27:0/3396515529 shutdown_connections 2024-09-16T07:54:28.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.496+0000 7fdb4f191640 1 --2- 172.21.15.27:0/2201642031 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb48138720 0x7fdb4813ab40 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:54:28.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.496+0000 7fdb4f191640 1 -- 172.21.15.27:0/2201642031 >> 172.21.15.27:0/2201642031 conn(0x7fdb4806e490 msgr2=0x7fdb4806e8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:54:28.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.496+0000 7f186ef91640 1 --2- 172.21.15.27:0/3396515529 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f18680717b0 0x7f1868071bc0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:54:28.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.495+0000 7fc0c6791640 1 -- 172.21.15.27:0/3019950773 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc0c009daa0 msgr2=0x7fc0c009ca60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:54:28.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.495+0000 7fc0c6791640 1 --2- 172.21.15.27:0/3019950773 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc0c009daa0 0x7fc0c009ca60 crc :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:54:28.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.496+0000 7f186ef91640 1 -- 172.21.15.27:0/3396515529 >> 172.21.15.27:0/3396515529 conn(0x7f18680a0320 msgr2=0x7f18680a0730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:54:28.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.496+0000 7fdb4f191640 1 -- 172.21.15.27:0/2201642031 shutdown_connections 2024-09-16T07:54:28.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.496+0000 7fc0c6791640 1 -- 172.21.15.27:0/3019950773 shutdown_connections 2024-09-16T07:54:28.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.496+0000 7fc0c6791640 1 --2- 172.21.15.27:0/3019950773 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc0c009daa0 0x7fc0c009ca60 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:54:28.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.496+0000 7fdb4f191640 1 -- 172.21.15.27:0/2201642031 wait complete. 2024-09-16T07:54:28.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.496+0000 7fc0c6791640 1 -- 172.21.15.27:0/3019950773 >> 172.21.15.27:0/3019950773 conn(0x7fc0c0069d60 msgr2=0x7fc0c009e1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:54:28.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.496+0000 7f186ef91640 1 -- 172.21.15.27:0/3396515529 shutdown_connections 2024-09-16T07:54:28.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.496+0000 7f186ef91640 1 -- 172.21.15.27:0/3396515529 wait complete. 2024-09-16T07:54:28.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.496+0000 7fc0c6791640 1 -- 172.21.15.27:0/3019950773 shutdown_connections 2024-09-16T07:54:28.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.496+0000 7fc0c6791640 1 -- 172.21.15.27:0/3019950773 wait complete. 2024-09-16T07:54:28.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.496+0000 7f6021f91640 1 -- 172.21.15.27:0/2219228454 wait complete. 2024-09-16T07:54:28.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.496+0000 7fdb4f191640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T07:54:28.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.496+0000 7fc0c6791640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T07:54:28.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.496+0000 7f6021f91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T07:54:28.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.496+0000 7f186ef91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T07:54:28.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.498+0000 7fdb4f191640 1 Processor -- start 2024-09-16T07:54:28.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.498+0000 7fc0c6791640 1 Processor -- start 2024-09-16T07:54:28.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.498+0000 7fdb4f191640 1 -- start start 2024-09-16T07:54:28.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.498+0000 7fc0c6791640 1 -- start start 2024-09-16T07:54:28.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.499+0000 7f6021f91640 1 Processor -- start 2024-09-16T07:54:28.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.499+0000 7f186ef91640 1 Processor -- start 2024-09-16T07:54:28.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.499+0000 7f186ef91640 1 -- start start 2024-09-16T07:54:28.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.499+0000 7f6021f91640 1 -- start start 2024-09-16T07:54:28.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.500+0000 7fc0c6791640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc0c009cfa0 0x7fc0c01358f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:54:28.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.500+0000 7fdb4f191640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb48132520 0x7fdb481c9fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:54:28.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.500+0000 7fc0c6791640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fc0c0138ec0 con 0x7fc0c009cfa0 2024-09-16T07:54:28.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.500+0000 7fdb4f191640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fdb480a1440 con 0x7fdb48132520 2024-09-16T07:54:28.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.500+0000 7fdb4cf2d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb48132520 0x7fdb481c9fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:54:28.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.500+0000 7fc0bffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc0c009cfa0 0x7fc0c01358f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:54:28.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.500+0000 7fdb4cf2d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb48132520 0x7fdb481c9fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56544/0 (socket says 172.21.15.27:56544) 2024-09-16T07:54:28.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.500+0000 7fdb4cf2d640 1 -- 172.21.15.27:0/3561065618 learned_addr learned my addr 172.21.15.27:0/3561065618 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-09-16T07:54:28.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.500+0000 7fc0bffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc0c009cfa0 0x7fc0c01358f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56534/0 (socket says 172.21.15.27:56534) 2024-09-16T07:54:28.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.500+0000 7fc0bffff640 1 -- 172.21.15.27:0/3294553693 learned_addr learned my addr 172.21.15.27:0/3294553693 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-09-16T07:54:28.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.500+0000 7fdb4cf2d640 1 -- 172.21.15.27:0/3561065618 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb4811fce0 con 0x7fdb48132520 2024-09-16T07:54:28.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.500+0000 7fdb4cf2d640 1 --2- 172.21.15.27:0/3561065618 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb48132520 0x7fdb481c9fb0 crc :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:54:28.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.500+0000 7fc0bffff640 1 -- 172.21.15.27:0/3294553693 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0c0136f60 con 0x7fc0c009cfa0 2024-09-16T07:54:28.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.501+0000 7fdb457fa640 1 -- 172.21.15.27:0/3561065618 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fdb3c005930 con 0x7fdb48132520 2024-09-16T07:54:28.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.501+0000 7fdb457fa640 1 -- 172.21.15.27:0/3561065618 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7fdb3c005a90 con 0x7fdb48132520 2024-09-16T07:54:28.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.501+0000 7fdb4f191640 1 -- 172.21.15.27:0/3561065618 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb4809ce40 con 0x7fdb48132520 2024-09-16T07:54:28.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.501+0000 7fc0bffff640 1 --2- 172.21.15.27:0/3294553693 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc0c009cfa0 0x7fc0c01358f0 crc :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:54:28.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.501+0000 7fdb457fa640 1 -- 172.21.15.27:0/3561065618 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fdb3c005d30 con 0x7fdb48132520 2024-09-16T07:54:28.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.501+0000 7fdb4f191640 1 -- 172.21.15.27:0/3561065618 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb4811ff10 con 0x7fdb48132520 2024-09-16T07:54:28.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.502+0000 7f6021f91640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f601c0a3450 0x7f601c11e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:54:28.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.502+0000 7f6021f91640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f601c121ed0 con 0x7f601c0a3450 2024-09-16T07:54:28.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.502+0000 7f186ef91640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f18680717b0 0x7f186811e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:54:28.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.502+0000 7fc0bcff9640 1 -- 172.21.15.27:0/3294553693 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fc0b0013530 con 0x7fc0c009cfa0 2024-09-16T07:54:28.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.502+0000 7f186ef91640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f1868120f60 con 0x7f18680717b0 2024-09-16T07:54:28.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.502+0000 7fc0c6791640 1 -- 172.21.15.27:0/3294553693 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc0c006fc60 con 0x7fc0c009cfa0 2024-09-16T07:54:28.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.502+0000 7fc0c6791640 1 -- 172.21.15.27:0/3294553693 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0c0136b40 con 0x7fc0c009cfa0 2024-09-16T07:54:28.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.502+0000 7f601b7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f601c0a3450 0x7f601c11e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:54:28.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.502+0000 7f601b7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f601c0a3450 0x7f601c11e090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56554/0 (socket says 172.21.15.27:56554) 2024-09-16T07:54:28.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.502+0000 7f601b7fe640 1 -- 172.21.15.27:0/3693281507 learned_addr learned my addr 172.21.15.27:0/3693281507 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-09-16T07:54:28.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.502+0000 7fdb457fa640 1 -- 172.21.15.27:0/3561065618 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) ==== 50954+0+0 (crc 0 0 0) 0x7fdb3c00c800 con 0x7fdb48132520 2024-09-16T07:54:28.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.502+0000 7f186cd2d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f18680717b0 0x7f186811e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:54:28.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.502+0000 7f186cd2d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f18680717b0 0x7f186811e090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56552/0 (socket says 172.21.15.27:56552) 2024-09-16T07:54:28.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.502+0000 7f186cd2d640 1 -- 172.21.15.27:0/1305793401 learned_addr learned my addr 172.21.15.27:0/1305793401 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-09-16T07:54:28.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.502+0000 7fc0bcff9640 1 -- 172.21.15.27:0/3294553693 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7fc0b0013690 con 0x7fc0c009cfa0 2024-09-16T07:54:28.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.502+0000 7fc0bcff9640 1 -- 172.21.15.27:0/3294553693 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fc0b0013930 con 0x7fc0c009cfa0 2024-09-16T07:54:28.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.502+0000 7fdb457fa640 1 --2- 172.21.15.27:0/3561065618 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7fdb2c03e8e0 0x7fdb2c040df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:54:28.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.503+0000 7fdb457fa640 1 -- 172.21.15.27:0/3561065618 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(15..15 src has 1..15) ==== 2728+0+0 (crc 0 0 0) 0x7fdb3c04f620 con 0x7fdb48132520 2024-09-16T07:54:28.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.503+0000 7f186cd2d640 1 -- 172.21.15.27:0/1305793401 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1868121950 con 0x7f18680717b0 2024-09-16T07:54:28.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.503+0000 7f601b7fe640 1 -- 172.21.15.27:0/3693281507 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f601c121aa0 con 0x7f601c0a3450 2024-09-16T07:54:28.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.503+0000 7fdb47fff640 1 --2- 172.21.15.27:0/3561065618 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7fdb2c03e8e0 0x7fdb2c040df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:54:28.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.503+0000 7f601b7fe640 1 --2- 172.21.15.27:0/3693281507 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f601c0a3450 0x7f601c11e090 crc :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:54:28.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.503+0000 7fdb4f191640 1 -- 172.21.15.27:0/3561065618 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdb4806a300 con 0x7fdb48132520 2024-09-16T07:54:28.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.503+0000 7fc0a67fc640 1 -- 172.21.15.27:0/3294553693 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc0980050f0 con 0x7fc0c009cfa0 2024-09-16T07:54:28.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.504+0000 7fdb47fff640 1 --2- 172.21.15.27:0/3561065618 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7fdb2c03e8e0 0x7fdb2c040df0 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:54:28.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.504+0000 7f186cd2d640 1 --2- 172.21.15.27:0/1305793401 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f18680717b0 0x7f186811e090 crc :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:54:28.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.504+0000 7f6003fff640 1 -- 172.21.15.27:0/3693281507 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f600c014620 con 0x7f601c0a3450 2024-09-16T07:54:28.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.504+0000 7f6021f91640 1 -- 172.21.15.27:0/3693281507 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f601c1358f0 con 0x7f601c0a3450 2024-09-16T07:54:28.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.504+0000 7f6003fff640 1 -- 172.21.15.27:0/3693281507 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f600c014780 con 0x7f601c0a3450 2024-09-16T07:54:28.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.504+0000 7f6003fff640 1 -- 172.21.15.27:0/3693281507 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f600c014620 con 0x7f601c0a3450 2024-09-16T07:54:28.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.504+0000 7fc0bcff9640 1 -- 172.21.15.27:0/3294553693 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) ==== 50954+0+0 (crc 0 0 0) 0x7fc0b0013b20 con 0x7fc0c009cfa0 2024-09-16T07:54:28.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.504+0000 7f6021f91640 1 -- 172.21.15.27:0/3693281507 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f601c121790 con 0x7f601c0a3450 2024-09-16T07:54:28.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.504+0000 7fc0bcff9640 1 --2- 172.21.15.27:0/3294553693 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7fc0ac03e8e0 0x7fc0ac040df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:54:28.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.504+0000 7fc0bcff9640 1 -- 172.21.15.27:0/3294553693 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(15..15 src has 1..15) ==== 2728+0+0 (crc 0 0 0) 0x7fc0b0018040 con 0x7fc0c009cfa0 2024-09-16T07:54:28.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.505+0000 7fc0bf7fe640 1 --2- 172.21.15.27:0/3294553693 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7fc0ac03e8e0 0x7fc0ac040df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:54:28.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.505+0000 7fc0bf7fe640 1 --2- 172.21.15.27:0/3294553693 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7fc0ac03e8e0 0x7fc0ac040df0 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:54:28.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.505+0000 7f6021f91640 1 -- 172.21.15.27:0/3693281507 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f601c06a300 con 0x7f601c0a3450 2024-09-16T07:54:28.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.509+0000 7fdb457fa640 1 -- 172.21.15.27:0/3561065618 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (crc 0 0 0) 0x7fdb3c00f230 con 0x7fdb48132520 2024-09-16T07:54:28.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.514+0000 7f18657fa640 1 -- 172.21.15.27:0/1305793401 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f185c00ee50 con 0x7f18680717b0 2024-09-16T07:54:28.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.514+0000 7fc0bcff9640 1 -- 172.21.15.27:0/3294553693 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (crc 0 0 0) 0x7fc0b0017190 con 0x7fc0c009cfa0 2024-09-16T07:54:28.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.514+0000 7f186ef91640 1 -- 172.21.15.27:0/1305793401 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1868120350 con 0x7f18680717b0 2024-09-16T07:54:28.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.514+0000 7f186ef91640 1 -- 172.21.15.27:0/1305793401 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f186811dcf0 con 0x7f18680717b0 2024-09-16T07:54:28.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.515+0000 7f18657fa640 1 -- 172.21.15.27:0/1305793401 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f185c015420 con 0x7f18680717b0 2024-09-16T07:54:28.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.515+0000 7f18657fa640 1 -- 172.21.15.27:0/1305793401 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f185c0156c0 con 0x7f18680717b0 2024-09-16T07:54:28.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.516+0000 7f6003fff640 1 -- 172.21.15.27:0/3693281507 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) ==== 50954+0+0 (crc 0 0 0) 0x7f600c014990 con 0x7f601c0a3450 2024-09-16T07:54:28.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.516+0000 7f18657fa640 1 -- 172.21.15.27:0/1305793401 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) ==== 50954+0+0 (crc 0 0 0) 0x7f185c015820 con 0x7f18680717b0 2024-09-16T07:54:28.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.516+0000 7f6003fff640 1 --2- 172.21.15.27:0/3693281507 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7f600803e9b0 0x7f6008040ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:54:28.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.516+0000 7f6003fff640 1 -- 172.21.15.27:0/3693281507 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(15..15 src has 1..15) ==== 2728+0+0 (crc 0 0 0) 0x7f600c019040 con 0x7f601c0a3450 2024-09-16T07:54:28.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.516+0000 7f18657fa640 1 --2- 172.21.15.27:0/1305793401 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7f184c03e930 0x7f184c040e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:54:28.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.516+0000 7f601affd640 1 --2- 172.21.15.27:0/3693281507 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7f600803e9b0 0x7f6008040ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:54:28.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.516+0000 7f186ef91640 1 -- 172.21.15.27:0/1305793401 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f186806a300 con 0x7f18680717b0 2024-09-16T07:54:28.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.517+0000 7f1867fff640 1 --2- 172.21.15.27:0/1305793401 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7f184c03e930 0x7f184c040e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:54:28.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.517+0000 7f601affd640 1 --2- 172.21.15.27:0/3693281507 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7f600803e9b0 0x7f6008040ec0 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:54:28.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.517+0000 7f1867fff640 1 --2- 172.21.15.27:0/1305793401 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7f184c03e930 0x7f184c040e40 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:54:28.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.520+0000 7f18657fa640 1 -- 172.21.15.27:0/1305793401 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(15..15 src has 1..15) ==== 2728+0+0 (crc 0 0 0) 0x7f185c012d40 con 0x7f18680717b0 2024-09-16T07:54:28.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.520+0000 7f6003fff640 1 -- 172.21.15.27:0/3693281507 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (crc 0 0 0) 0x7f600c013cb0 con 0x7f601c0a3450 2024-09-16T07:54:28.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.526+0000 7f18657fa640 1 -- 172.21.15.27:0/1305793401 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (crc 0 0 0) 0x7f185c014c60 con 0x7f18680717b0 2024-09-16T07:54:28.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.700+0000 7fdb4f191640 1 -- 172.21.15.27:0/3561065618 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7fdb48133040 con 0x7fdb48132520 2024-09-16T07:54:28.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.700+0000 7fdb457fa640 1 -- 172.21.15.27:0/3561065618 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (crc 0 0 0) 0x7fdb48133040 con 0x7fdb48132520 2024-09-16T07:54:28.699 INFO:teuthology.orchestra.run.smithi027.stdout:38654705672 2024-09-16T07:54:28.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.702+0000 7fdb4f191640 1 -- 172.21.15.27:0/3561065618 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7fdb2c03e8e0 msgr2=0x7fdb2c040df0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:54:28.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.702+0000 7fdb4f191640 1 --2- 172.21.15.27:0/3561065618 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7fdb2c03e8e0 0x7fdb2c040df0 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:54:28.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.702+0000 7fc0a67fc640 1 -- 172.21.15.27:0/3294553693 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7fc098005670 con 0x7fc0c009cfa0 2024-09-16T07:54:28.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.702+0000 7fdb4f191640 1 -- 172.21.15.27:0/3561065618 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb48132520 msgr2=0x7fdb481c9fb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:54:28.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.702+0000 7fdb4f191640 1 --2- 172.21.15.27:0/3561065618 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb48132520 0x7fdb481c9fb0 crc :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:54:28.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.702+0000 7fdb4f191640 1 -- 172.21.15.27:0/3561065618 shutdown_connections 2024-09-16T07:54:28.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.702+0000 7fdb4f191640 1 --2- 172.21.15.27:0/3561065618 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7fdb2c03e8e0 0x7fdb2c040df0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:54:28.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.702+0000 7fdb4f191640 1 --2- 172.21.15.27:0/3561065618 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb48132520 0x7fdb481c9fb0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:54:28.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.702+0000 7fdb4f191640 1 -- 172.21.15.27:0/3561065618 >> 172.21.15.27:0/3561065618 conn(0x7fdb4806e490 msgr2=0x7fdb480a5840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:54:28.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.703+0000 7fdb4f191640 1 -- 172.21.15.27:0/3561065618 shutdown_connections 2024-09-16T07:54:28.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.703+0000 7fdb4f191640 1 -- 172.21.15.27:0/3561065618 wait complete. 2024-09-16T07:54:28.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.703+0000 7fc0bcff9640 1 -- 172.21.15.27:0/3294553693 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (crc 0 0 0) 0x7fc0b000f2a0 con 0x7fc0c009cfa0 2024-09-16T07:54:28.703 INFO:teuthology.orchestra.run.smithi027.stdout:38654705672 2024-09-16T07:54:28.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.705+0000 7fc0a67fc640 1 -- 172.21.15.27:0/3294553693 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7fc0ac03e8e0 msgr2=0x7fc0ac040df0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:54:28.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.705+0000 7fc0a67fc640 1 --2- 172.21.15.27:0/3294553693 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7fc0ac03e8e0 0x7fc0ac040df0 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:54:28.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.705+0000 7fc0a67fc640 1 -- 172.21.15.27:0/3294553693 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc0c009cfa0 msgr2=0x7fc0c01358f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:54:28.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.705+0000 7fc0a67fc640 1 --2- 172.21.15.27:0/3294553693 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc0c009cfa0 0x7fc0c01358f0 crc :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:54:28.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.706+0000 7fc0a67fc640 1 -- 172.21.15.27:0/3294553693 shutdown_connections 2024-09-16T07:54:28.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.706+0000 7fc0a67fc640 1 --2- 172.21.15.27:0/3294553693 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7fc0ac03e8e0 0x7fc0ac040df0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:54:28.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.706+0000 7fc0a67fc640 1 --2- 172.21.15.27:0/3294553693 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc0c009cfa0 0x7fc0c01358f0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:54:28.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.706+0000 7fc0a67fc640 1 -- 172.21.15.27:0/3294553693 >> 172.21.15.27:0/3294553693 conn(0x7fc0c0069d60 msgr2=0x7fc0c0131580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:54:28.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.706+0000 7fc0a67fc640 1 -- 172.21.15.27:0/3294553693 shutdown_connections 2024-09-16T07:54:28.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.706+0000 7fc0a67fc640 1 -- 172.21.15.27:0/3294553693 wait complete. 2024-09-16T07:54:28.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.708+0000 7f6021f91640 1 -- 172.21.15.27:0/3693281507 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f601c120120 con 0x7f601c0a3450 2024-09-16T07:54:28.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.710+0000 7f6003fff640 1 -- 172.21.15.27:0/3693281507 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (crc 0 0 0) 0x7f600c013cb0 con 0x7f601c0a3450 2024-09-16T07:54:28.709 INFO:teuthology.orchestra.run.smithi027.stdout:38654705672 2024-09-16T07:54:28.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.713+0000 7f6021f91640 1 -- 172.21.15.27:0/3693281507 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7f600803e9b0 msgr2=0x7f6008040ec0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:54:28.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.713+0000 7f6021f91640 1 --2- 172.21.15.27:0/3693281507 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7f600803e9b0 0x7f6008040ec0 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:54:28.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.713+0000 7f6021f91640 1 -- 172.21.15.27:0/3693281507 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f601c0a3450 msgr2=0x7f601c11e090 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:54:28.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.713+0000 7f6021f91640 1 --2- 172.21.15.27:0/3693281507 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f601c0a3450 0x7f601c11e090 crc :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:54:28.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.713+0000 7f6021f91640 1 -- 172.21.15.27:0/3693281507 shutdown_connections 2024-09-16T07:54:28.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.713+0000 7f6021f91640 1 --2- 172.21.15.27:0/3693281507 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7f600803e9b0 0x7f6008040ec0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:54:28.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.713+0000 7f6021f91640 1 --2- 172.21.15.27:0/3693281507 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f601c0a3450 0x7f601c11e090 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:54:28.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.713+0000 7f6021f91640 1 -- 172.21.15.27:0/3693281507 >> 172.21.15.27:0/3693281507 conn(0x7f601c06e490 msgr2=0x7f601c0a5840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:54:28.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.713+0000 7f6021f91640 1 -- 172.21.15.27:0/3693281507 shutdown_connections 2024-09-16T07:54:28.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.714+0000 7f6021f91640 1 -- 172.21.15.27:0/3693281507 wait complete. 2024-09-16T07:54:28.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.716+0000 7f186ef91640 1 -- 172.21.15.27:0/1305793401 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f186811fd70 con 0x7f18680717b0 2024-09-16T07:54:28.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.716+0000 7f18657fa640 1 -- 172.21.15.27:0/1305793401 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (crc 0 0 0) 0x7f185c012b80 con 0x7f18680717b0 2024-09-16T07:54:28.715 INFO:teuthology.orchestra.run.smithi027.stdout:38654705672 2024-09-16T07:54:28.715 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705672 got 38654705672 for osd.3 2024-09-16T07:54:28.716 DEBUG:teuthology.parallel:result is None 2024-09-16T07:54:28.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.719+0000 7f186ef91640 1 -- 172.21.15.27:0/1305793401 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7f184c03e930 msgr2=0x7f184c040e40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:54:28.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.719+0000 7f186ef91640 1 --2- 172.21.15.27:0/1305793401 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7f184c03e930 0x7f184c040e40 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:54:28.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.719+0000 7f186ef91640 1 -- 172.21.15.27:0/1305793401 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f18680717b0 msgr2=0x7f186811e090 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:54:28.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.719+0000 7f186ef91640 1 --2- 172.21.15.27:0/1305793401 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f18680717b0 0x7f186811e090 crc :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:54:28.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.719+0000 7f186ef91640 1 -- 172.21.15.27:0/1305793401 shutdown_connections 2024-09-16T07:54:28.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.719+0000 7f186ef91640 1 --2- 172.21.15.27:0/1305793401 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7f184c03e930 0x7f184c040e40 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:54:28.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.719+0000 7f186ef91640 1 --2- 172.21.15.27:0/1305793401 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f18680717b0 0x7f186811e090 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:54:28.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.719+0000 7f186ef91640 1 -- 172.21.15.27:0/1305793401 >> 172.21.15.27:0/1305793401 conn(0x7f18680a0320 msgr2=0x7f18680a36b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:54:28.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.719+0000 7f186ef91640 1 -- 172.21.15.27:0/1305793401 shutdown_connections 2024-09-16T07:54:28.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.719+0000 7f186ef91640 1 -- 172.21.15.27:0/1305793401 wait complete. 2024-09-16T07:54:28.719 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705672 got 38654705672 for osd.2 2024-09-16T07:54:28.719 DEBUG:teuthology.parallel:result is None 2024-09-16T07:54:28.726 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705672 got 38654705672 for osd.1 2024-09-16T07:54:28.726 DEBUG:teuthology.parallel:result is None 2024-09-16T07:54:28.728 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705672 got 38654705672 for osd.0 2024-09-16T07:54:28.729 DEBUG:teuthology.parallel:result is None 2024-09-16T07:54:28.729 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-16T07:54:28.729 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T07:54:28.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.880+0000 7f405c311640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T07:54:28.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.885+0000 7f405c311640 1 Processor -- start 2024-09-16T07:54:28.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.885+0000 7f405c311640 1 -- start start 2024-09-16T07:54:28.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.887+0000 7f405c311640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f40541365a0 0x7f40541389c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:54:28.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.887+0000 7f405c311640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4054074a00 con 0x7f4054130420 2024-09-16T07:54:28.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.887+0000 7f405c311640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 -- 0x7f4054074c30 con 0x7f40541365a0 2024-09-16T07:54:28.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.887+0000 7f40598ac640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f40541365a0 0x7f40541389c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:54:28.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.887+0000 7f40598ac640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f40541365a0 0x7f40541389c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56558/0 (socket says 172.21.15.27:56558) 2024-09-16T07:54:28.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.887+0000 7f40598ac640 1 -- 172.21.15.27:0/3484022764 learned_addr learned my addr 172.21.15.27:0/3484022764 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-09-16T07:54:28.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.887+0000 7f405a0ad640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f4054130420 0x7f4054130830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:46630/0 (socket says 172.21.15.27:46630) 2024-09-16T07:54:28.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.887+0000 7f40598ac640 1 -- 172.21.15.27:0/3484022764 >> v1:172.21.15.27:6789/0 conn(0x7f4054130420 legacy=0x7f4054130830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:54:28.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.887+0000 7f40598ac640 1 -- 172.21.15.27:0/3484022764 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4054071930 con 0x7f40541365a0 2024-09-16T07:54:28.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.887+0000 7f40598ac640 1 --2- 172.21.15.27:0/3484022764 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f40541365a0 0x7f40541389c0 crc :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12eb4fde61e446e0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:54:28.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.888+0000 7f40590ab640 1 -- 172.21.15.27:0/3484022764 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f4048009e50 con 0x7f40541365a0 2024-09-16T07:54:28.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.888+0000 7f40590ab640 1 -- 172.21.15.27:0/3484022764 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f404800e230 con 0x7f40541365a0 2024-09-16T07:54:28.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.888+0000 7f40590ab640 1 -- 172.21.15.27:0/3484022764 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f404800e4d0 con 0x7f40541365a0 2024-09-16T07:54:28.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.888+0000 7f405c311640 1 -- 172.21.15.27:0/3484022764 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f40541365a0 msgr2=0x7f40541389c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:54:28.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.888+0000 7f405c311640 1 --2- 172.21.15.27:0/3484022764 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f40541365a0 0x7f40541389c0 crc :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:54:28.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.888+0000 7f405c311640 1 -- 172.21.15.27:0/3484022764 shutdown_connections 2024-09-16T07:54:28.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.888+0000 7f405c311640 1 --2- 172.21.15.27:0/3484022764 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f40541365a0 0x7f40541389c0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:54:28.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.888+0000 7f405c311640 1 -- 172.21.15.27:0/3484022764 >> 172.21.15.27:0/3484022764 conn(0x7f40540a0320 msgr2=0x7f40540a0730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:54:28.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.888+0000 7f405c311640 1 -- 172.21.15.27:0/3484022764 shutdown_connections 2024-09-16T07:54:28.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.888+0000 7f405c311640 1 -- 172.21.15.27:0/3484022764 wait complete. 2024-09-16T07:54:28.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.889+0000 7f405c311640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T07:54:28.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.890+0000 7f405c311640 1 Processor -- start 2024-09-16T07:54:28.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.890+0000 7f405c311640 1 -- start start 2024-09-16T07:54:28.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.892+0000 7f405c311640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4054130420 0x7f405411fe70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:54:28.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.892+0000 7f405c311640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f405411dfc0 con 0x7f4054130420 2024-09-16T07:54:28.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.892+0000 7f405a0ad640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4054130420 0x7f405411fe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:54:28.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.892+0000 7f405a0ad640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4054130420 0x7f405411fe70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56566/0 (socket says 172.21.15.27:56566) 2024-09-16T07:54:28.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.892+0000 7f405a0ad640 1 -- 172.21.15.27:0/3850374573 learned_addr learned my addr 172.21.15.27:0/3850374573 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-09-16T07:54:28.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.892+0000 7f405a0ad640 1 -- 172.21.15.27:0/3850374573 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f405411e730 con 0x7f4054130420 2024-09-16T07:54:28.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.892+0000 7f405a0ad640 1 --2- 172.21.15.27:0/3850374573 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4054130420 0x7f405411fe70 crc :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:54:28.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.893+0000 7f4052ffd640 1 -- 172.21.15.27:0/3850374573 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f40440071e0 con 0x7f4054130420 2024-09-16T07:54:28.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.893+0000 7f4052ffd640 1 -- 172.21.15.27:0/3850374573 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f4044007340 con 0x7f4054130420 2024-09-16T07:54:28.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.893+0000 7f405c311640 1 -- 172.21.15.27:0/3850374573 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f405412fe90 con 0x7f4054130420 2024-09-16T07:54:28.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.893+0000 7f4052ffd640 1 -- 172.21.15.27:0/3850374573 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f40440075e0 con 0x7f4054130420 2024-09-16T07:54:28.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.893+0000 7f405c311640 1 -- 172.21.15.27:0/3850374573 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f405411f550 con 0x7f4054130420 2024-09-16T07:54:28.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.894+0000 7f405c311640 1 -- 172.21.15.27:0/3850374573 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f405406a300 con 0x7f4054130420 2024-09-16T07:54:28.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.894+0000 7f4052ffd640 1 -- 172.21.15.27:0/3850374573 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) ==== 50954+0+0 (crc 0 0 0) 0x7f4044007790 con 0x7f4054130420 2024-09-16T07:54:28.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.894+0000 7f4052ffd640 1 --2- 172.21.15.27:0/3850374573 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7f404003e930 0x7f4040040e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:54:28.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.894+0000 7f4052ffd640 1 -- 172.21.15.27:0/3850374573 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(15..15 src has 1..15) ==== 2728+0+0 (crc 0 0 0) 0x7f40440502d0 con 0x7f4054130420 2024-09-16T07:54:28.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.894+0000 7f40598ac640 1 --2- 172.21.15.27:0/3850374573 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7f404003e930 0x7f4040040e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:54:28.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.895+0000 7f40598ac640 1 --2- 172.21.15.27:0/3850374573 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7f404003e930 0x7f4040040e40 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:54:28.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:28.899+0000 7f4052ffd640 1 -- 172.21.15.27:0/3850374573 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (crc 0 0 0) 0x7f4044009bc0 con 0x7f4054130420 2024-09-16T07:54:29.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:29.080+0000 7f405c311640 1 -- 172.21.15.27:0/3850374573 --> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f4054120dc0 con 0x7f404003e930 2024-09-16T07:54:29.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:29.082+0000 7f4052ffd640 1 -- 172.21.15.27:0/3850374573 <== mgr.4102 v2:172.21.15.187:6800/4038049593 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+31201 (crc 0 0 0) 0x7f4054120dc0 con 0x7f404003e930 2024-09-16T07:54:29.081 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-16T07:54:29.081 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-09-16T07:54:29.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:29.084+0000 7f405c311640 1 -- 172.21.15.27:0/3850374573 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7f404003e930 msgr2=0x7f4040040e40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:54:29.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:29.084+0000 7f405c311640 1 --2- 172.21.15.27:0/3850374573 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7f404003e930 0x7f4040040e40 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:54:29.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:29.084+0000 7f405c311640 1 -- 172.21.15.27:0/3850374573 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4054130420 msgr2=0x7f405411fe70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:54:29.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:29.084+0000 7f405c311640 1 --2- 172.21.15.27:0/3850374573 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4054130420 0x7f405411fe70 crc :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:54:29.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:29.084+0000 7f405c311640 1 -- 172.21.15.27:0/3850374573 shutdown_connections 2024-09-16T07:54:29.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:29.084+0000 7f405c311640 1 --2- 172.21.15.27:0/3850374573 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7f404003e930 0x7f4040040e40 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:54:29.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:29.084+0000 7f405c311640 1 --2- 172.21.15.27:0/3850374573 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4054130420 0x7f405411fe70 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:54:29.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:29.084+0000 7f405c311640 1 -- 172.21.15.27:0/3850374573 >> 172.21.15.27:0/3850374573 conn(0x7f40540a0320 msgr2=0x7f405409f100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:54:29.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:29.084+0000 7f405c311640 1 -- 172.21.15.27:0/3850374573 shutdown_connections 2024-09-16T07:54:29.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:29.085+0000 7f405c311640 1 -- 172.21.15.27:0/3850374573 wait complete. 2024-09-16T07:54:29.097 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":11,"stamp":"2024-09-16T07:54:28.297476+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":209190912,"kb_used":3664,"kb_used_data":3664,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":209187248,"statfs":{"total":214211493888,"available":214207741952,"internally_reserved":0,"allocated":3751936,"data_stored":3751936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-16T07:54:26.736548+0000","last_change":"2024-09-16T07:54:26.740923+0000","last_active":"2024-09-16T07:54:26.736548+0000","last_peered":"2024-09-16T07:54:26.736548+0000","last_clean":"2024-09-16T07:54:26.736548+0000","last_became_active":"2024-09-16T07:54:24.784360+0000","last_became_peered":"2024-09-16T07:54:24.784360+0000","last_unstale":"2024-09-16T07:54:26.736548+0000","last_undegraded":"2024-09-16T07:54:26.736548+0000","last_fullsized":"2024-09-16T07:54:26.736548+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T07:54:23.687651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T07:54:23.687651+0000","last_clean_scrub_stamp":"2024-09-16T07:54:23.687651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-16T07:54:26.736485+0000","last_change":"2024-09-16T07:54:26.739464+0000","last_active":"2024-09-16T07:54:26.736485+0000","last_peered":"2024-09-16T07:54:26.736485+0000","last_clean":"2024-09-16T07:54:26.736485+0000","last_became_active":"2024-09-16T07:54:24.805913+0000","last_became_peered":"2024-09-16T07:54:24.805913+0000","last_unstale":"2024-09-16T07:54:26.736485+0000","last_undegraded":"2024-09-16T07:54:26.736485+0000","last_fullsized":"2024-09-16T07:54:26.736485+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T07:54:23.687651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T07:54:23.687651+0000","last_clean_scrub_stamp":"2024-09-16T07:54:23.687651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-16T07:54:26.732090+0000","last_change":"2024-09-16T07:54:26.735332+0000","last_active":"2024-09-16T07:54:26.732090+0000","last_peered":"2024-09-16T07:54:26.732090+0000","last_clean":"2024-09-16T07:54:26.732090+0000","last_became_active":"2024-09-16T07:54:24.773365+0000","last_became_peered":"2024-09-16T07:54:24.773365+0000","last_unstale":"2024-09-16T07:54:26.732090+0000","last_undegraded":"2024-09-16T07:54:26.732090+0000","last_fullsized":"2024-09-16T07:54:26.732090+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T07:54:23.687651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T07:54:23.687651+0000","last_clean_scrub_stamp":"2024-09-16T07:54:23.687651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-16T07:54:26.730092+0000","last_change":"2024-09-16T07:54:26.734682+0000","last_active":"2024-09-16T07:54:26.730092+0000","last_peered":"2024-09-16T07:54:26.730092+0000","last_clean":"2024-09-16T07:54:26.730092+0000","last_became_active":"2024-09-16T07:54:24.756974+0000","last_became_peered":"2024-09-16T07:54:24.756974+0000","last_unstale":"2024-09-16T07:54:26.730092+0000","last_undegraded":"2024-09-16T07:54:26.730092+0000","last_fullsized":"2024-09-16T07:54:26.730092+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T07:54:23.687651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T07:54:23.687651+0000","last_clean_scrub_stamp":"2024-09-16T07:54:23.687651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"15'2","reported_seq":17,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-16T07:54:26.741927+0000","last_change":"2024-09-16T07:54:26.741927+0000","last_active":"2024-09-16T07:54:26.741927+0000","last_peered":"2024-09-16T07:54:26.741927+0000","last_clean":"2024-09-16T07:54:26.741927+0000","last_became_active":"2024-09-16T07:54:24.794754+0000","last_became_peered":"2024-09-16T07:54:24.794754+0000","last_unstale":"2024-09-16T07:54:26.741927+0000","last_undegraded":"2024-09-16T07:54:26.741927+0000","last_fullsized":"2024-09-16T07:54:26.741927+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T07:54:23.687651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T07:54:23.687651+0000","last_clean_scrub_stamp":"2024-09-16T07:54:23.687651+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-16T07:54:26.730661+0000","last_change":"2024-09-16T07:54:26.734478+0000","last_active":"2024-09-16T07:54:26.730661+0000","last_peered":"2024-09-16T07:54:26.730661+0000","last_clean":"2024-09-16T07:54:26.730661+0000","last_became_active":"2024-09-16T07:54:24.774923+0000","last_became_peered":"2024-09-16T07:54:24.774923+0000","last_unstale":"2024-09-16T07:54:26.730661+0000","last_undegraded":"2024-09-16T07:54:26.730661+0000","last_fullsized":"2024-09-16T07:54:26.730661+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T07:54:23.687651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T07:54:23.687651+0000","last_clean_scrub_stamp":"2024-09-16T07:54:23.687651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-16T07:54:26.733855+0000","last_change":"2024-09-16T07:54:26.742127+0000","last_active":"2024-09-16T07:54:26.733855+0000","last_peered":"2024-09-16T07:54:26.733855+0000","last_clean":"2024-09-16T07:54:26.733855+0000","last_became_active":"2024-09-16T07:54:24.773472+0000","last_became_peered":"2024-09-16T07:54:24.773472+0000","last_unstale":"2024-09-16T07:54:26.733855+0000","last_undegraded":"2024-09-16T07:54:26.733855+0000","last_fullsized":"2024-09-16T07:54:26.733855+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T07:54:23.687651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T07:54:23.687651+0000","last_clean_scrub_stamp":"2024-09-16T07:54:23.687651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"13'1","reported_seq":16,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-16T07:54:26.735960+0000","last_change":"2024-09-16T07:54:26.739091+0000","last_active":"2024-09-16T07:54:26.735960+0000","last_peered":"2024-09-16T07:54:26.735960+0000","last_clean":"2024-09-16T07:54:26.735960+0000","last_became_active":"2024-09-16T07:54:24.801907+0000","last_became_peered":"2024-09-16T07:54:24.801907+0000","last_unstale":"2024-09-16T07:54:26.735960+0000","last_undegraded":"2024-09-16T07:54:26.735960+0000","last_fullsized":"2024-09-16T07:54:26.735960+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T07:54:23.687651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T07:54:23.687651+0000","last_clean_scrub_stamp":"2024-09-16T07:54:23.687651+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'52","reported_seq":87,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-16T07:54:26.730190+0000","last_change":"2024-09-16T07:54:22.755090+0000","last_active":"2024-09-16T07:54:26.730190+0000","last_peered":"2024-09-16T07:54:26.730190+0000","last_clean":"2024-09-16T07:54:26.730190+0000","last_became_active":"2024-09-16T07:54:22.736266+0000","last_became_peered":"2024-09-16T07:54:22.736266+0000","last_unstale":"2024-09-16T07:54:26.730190+0000","last_undegraded":"2024-09-16T07:54:26.730190+0000","last_fullsized":"2024-09-16T07:54:26.730190+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T07:54:21.678374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T07:54:21.678374+0000","last_clean_scrub_stamp":"2024-09-16T07:54:21.678374+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":214211493888,"available":214207741952,"internally_reserved":0,"allocated":3751936,"data_stored":3751936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53552873472,"available":53551673344,"internally_reserved":0,"allocated":1200128,"data_stored":1200128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705672,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":1172,"kb_used_data":1172,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52296556,"statfs":{"total":53552873472,"available":53551673344,"internally_reserved":0,"allocated":1200128,"data_stored":1200128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705672,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":532,"kb_used_data":532,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52297196,"statfs":{"total":53552873472,"available":53552328704,"internally_reserved":0,"allocated":544768,"data_stored":544768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705672,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":528,"kb_used_data":528,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52297200,"statfs":{"total":53552873472,"available":53552332800,"internally_reserved":0,"allocated":540672,"data_stored":540672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705672,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":1432,"kb_used_data":1432,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52296296,"statfs":{"total":53552873472,"available":53551407104,"internally_reserved":0,"allocated":1466368,"data_stored":1466368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":3,"total":53552873472,"available":53551673344,"internally_reserved":0,"allocated":1200128,"data_stored":1200128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53552873472,"available":53551407104,"internally_reserved":0,"allocated":1466368,"data_stored":1466368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53552873472,"available":53552332800,"internally_reserved":0,"allocated":540672,"data_stored":540672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53552873472,"available":53552328704,"internally_reserved":0,"allocated":544768,"data_stored":544768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":53552873472,"available":53551673344,"internally_reserved":0,"allocated":1200128,"data_stored":1200128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T07:54:29.098 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T07:54:29.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:29.207+0000 7f76bea6c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T07:54:29.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:29.213+0000 7f76bea6c640 1 Processor -- start 2024-09-16T07:54:29.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:29.213+0000 7f76bea6c640 1 -- start start 2024-09-16T07:54:29.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:29.214+0000 7f76bea6c640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f76b8130470 0x7f76b8130880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:54:29.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:29.214+0000 7f76bea6c640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f76b80745e0 con 0x7f76b8133250 2024-09-16T07:54:29.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:29.214+0000 7f76bea6c640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 -- 0x7f76b80740f0 con 0x7f76b8130470 2024-09-16T07:54:29.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:29.214+0000 7f76bc808640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f76b8130470 0x7f76b8130880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:54:29.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:29.215+0000 7f76b7fff640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f76b8133250 0x7f76b8133660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:46644/0 (socket says 172.21.15.27:46644) 2024-09-16T07:54:29.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:29.215+0000 7f76bc808640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f76b8130470 0x7f76b8130880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56568/0 (socket says 172.21.15.27:56568) 2024-09-16T07:54:29.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:29.215+0000 7f76b7fff640 1 -- 172.21.15.27:0/2261397371 learned_addr learned my addr 172.21.15.27:0/2261397371 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-09-16T07:54:29.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:29.215+0000 7f76bc808640 1 -- 172.21.15.27:0/2261397371 >> v1:172.21.15.27:6789/0 conn(0x7f76b8133250 legacy=0x7f76b8133660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:54:29.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:29.215+0000 7f76bc808640 1 -- 172.21.15.27:0/2261397371 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76b80749a0 con 0x7f76b8130470 2024-09-16T07:54:29.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:29.215+0000 7f76b77fe640 1 -- 172.21.15.27:0/2261397371 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 948286646 0 0) 0x7f76b80745e0 con 0x7f76b8133250 2024-09-16T07:54:29.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:29.215+0000 7f76bc808640 1 --2- 172.21.15.27:0/2261397371 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f76b8130470 0x7f76b8130880 crc :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c658d310c9c3820 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:54:29.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:29.215+0000 7f76b77fe640 1 -- 172.21.15.27:0/2261397371 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f76a8009d90 con 0x7f76b8130470 2024-09-16T07:54:29.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:29.215+0000 7f76b77fe640 1 -- 172.21.15.27:0/2261397371 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f76a800e230 con 0x7f76b8130470 2024-09-16T07:54:29.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:29.215+0000 7f76b77fe640 1 -- 172.21.15.27:0/2261397371 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f76a800e4d0 con 0x7f76b8130470 2024-09-16T07:54:29.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:29.216+0000 7f76bea6c640 1 -- 172.21.15.27:0/2261397371 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f76b8130470 msgr2=0x7f76b8130880 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:54:29.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:29.216+0000 7f76bea6c640 1 --2- 172.21.15.27:0/2261397371 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f76b8130470 0x7f76b8130880 crc :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:54:29.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:29.216+0000 7f76bea6c640 1 -- 172.21.15.27:0/2261397371 shutdown_connections 2024-09-16T07:54:29.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:29.216+0000 7f76bea6c640 1 --2- 172.21.15.27:0/2261397371 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f76b8130470 0x7f76b8130880 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:54:29.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:29.216+0000 7f76bea6c640 1 -- 172.21.15.27:0/2261397371 >> 172.21.15.27:0/2261397371 conn(0x7f76b80690e0 msgr2=0x7f76b80694f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:54:29.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:29.216+0000 7f76bea6c640 1 -- 172.21.15.27:0/2261397371 shutdown_connections 2024-09-16T07:54:29.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:29.216+0000 7f76bea6c640 1 -- 172.21.15.27:0/2261397371 wait complete. 2024-09-16T07:54:29.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:29.216+0000 7f76bea6c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T07:54:29.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:29.218+0000 7f76bea6c640 1 Processor -- start 2024-09-16T07:54:29.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:29.218+0000 7f76bea6c640 1 -- start start 2024-09-16T07:54:29.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:29.220+0000 7f76bea6c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f76b8130470 0x7f76b811f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:54:29.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:29.220+0000 7f76bea6c640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f76b8123070 con 0x7f76b8130470 2024-09-16T07:54:29.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:29.220+0000 7f76bc808640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f76b8130470 0x7f76b811f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:54:29.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:29.220+0000 7f76bc808640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f76b8130470 0x7f76b811f1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56570/0 (socket says 172.21.15.27:56570) 2024-09-16T07:54:29.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:29.220+0000 7f76bc808640 1 -- 172.21.15.27:0/3846951227 learned_addr learned my addr 172.21.15.27:0/3846951227 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-09-16T07:54:29.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:29.220+0000 7f76bc808640 1 -- 172.21.15.27:0/3846951227 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76b8122930 con 0x7f76b8130470 2024-09-16T07:54:29.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:29.220+0000 7f76bc808640 1 --2- 172.21.15.27:0/3846951227 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f76b8130470 0x7f76b811f1f0 crc :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:54:29.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:29.220+0000 7f76b57fa640 1 -- 172.21.15.27:0/3846951227 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f76a8014620 con 0x7f76b8130470 2024-09-16T07:54:29.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:29.220+0000 7f76bea6c640 1 -- 172.21.15.27:0/3846951227 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76b8120af0 con 0x7f76b8130470 2024-09-16T07:54:29.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:29.220+0000 7f76b57fa640 1 -- 172.21.15.27:0/3846951227 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f76a8014780 con 0x7f76b8130470 2024-09-16T07:54:29.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:29.220+0000 7f76b57fa640 1 -- 172.21.15.27:0/3846951227 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f76a8014a20 con 0x7f76b8130470 2024-09-16T07:54:29.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:29.220+0000 7f76bea6c640 1 -- 172.21.15.27:0/3846951227 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76b8121920 con 0x7f76b8130470 2024-09-16T07:54:29.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:29.221+0000 7f76b57fa640 1 -- 172.21.15.27:0/3846951227 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) ==== 50954+0+0 (crc 0 0 0) 0x7f76a8014c20 con 0x7f76b8130470 2024-09-16T07:54:29.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:29.222+0000 7f76b57fa640 1 --2- 172.21.15.27:0/3846951227 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7f76a403e910 0x7f76a4040e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:54:29.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:29.222+0000 7f76bea6c640 1 -- 172.21.15.27:0/3846951227 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f76b806a300 con 0x7f76b8130470 2024-09-16T07:54:29.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:29.222+0000 7f76b57fa640 1 -- 172.21.15.27:0/3846951227 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(15..15 src has 1..15) ==== 2728+0+0 (crc 0 0 0) 0x7f76a8019040 con 0x7f76b8130470 2024-09-16T07:54:29.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:29.222+0000 7f76b7fff640 1 --2- 172.21.15.27:0/3846951227 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7f76a403e910 0x7f76a4040e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:54:29.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:29.222+0000 7f76b7fff640 1 --2- 172.21.15.27:0/3846951227 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7f76a403e910 0x7f76a4040e20 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:54:29.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:29.227+0000 7f76b57fa640 1 -- 172.21.15.27:0/3846951227 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (crc 0 0 0) 0x7f76a8011570 con 0x7f76b8130470 2024-09-16T07:54:29.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:29.408+0000 7f76bea6c640 1 -- 172.21.15.27:0/3846951227 --> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f76b8072bf0 con 0x7f76a403e910 2024-09-16T07:54:29.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:29.409+0000 7f76b57fa640 1 -- 172.21.15.27:0/3846951227 <== mgr.4102 v2:172.21.15.187:6800/4038049593 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+31201 (crc 0 0 0) 0x7f76ac005930 con 0x7f76a403e910 2024-09-16T07:54:29.408 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-16T07:54:29.408 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-09-16T07:54:29.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:29.411+0000 7f76bea6c640 1 -- 172.21.15.27:0/3846951227 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7f76a403e910 msgr2=0x7f76a4040e20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:54:29.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:29.411+0000 7f76bea6c640 1 --2- 172.21.15.27:0/3846951227 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7f76a403e910 0x7f76a4040e20 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:54:29.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:29.411+0000 7f76bea6c640 1 -- 172.21.15.27:0/3846951227 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f76b8130470 msgr2=0x7f76b811f1f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:54:29.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:29.411+0000 7f76bea6c640 1 --2- 172.21.15.27:0/3846951227 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f76b8130470 0x7f76b811f1f0 crc :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:54:29.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:29.411+0000 7f76bea6c640 1 -- 172.21.15.27:0/3846951227 shutdown_connections 2024-09-16T07:54:29.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:29.411+0000 7f76bea6c640 1 --2- 172.21.15.27:0/3846951227 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7f76a403e910 0x7f76a4040e20 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:54:29.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:29.411+0000 7f76bea6c640 1 --2- 172.21.15.27:0/3846951227 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f76b8130470 0x7f76b811f1f0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:54:29.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:29.411+0000 7f76bea6c640 1 -- 172.21.15.27:0/3846951227 >> 172.21.15.27:0/3846951227 conn(0x7f76b80690e0 msgr2=0x7f76b80a3e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:54:29.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:29.411+0000 7f76bea6c640 1 -- 172.21.15.27:0/3846951227 shutdown_connections 2024-09-16T07:54:29.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:29.411+0000 7f76bea6c640 1 -- 172.21.15.27:0/3846951227 wait complete. 2024-09-16T07:54:29.421 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":12,"stamp":"2024-09-16T07:54:29.083285+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":209190912,"kb_used":3664,"kb_used_data":3664,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":209187248,"statfs":{"total":214211493888,"available":214207741952,"internally_reserved":0,"allocated":3751936,"data_stored":3751936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.785809"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-16T07:54:26.736548+0000","last_change":"2024-09-16T07:54:26.740923+0000","last_active":"2024-09-16T07:54:26.736548+0000","last_peered":"2024-09-16T07:54:26.736548+0000","last_clean":"2024-09-16T07:54:26.736548+0000","last_became_active":"2024-09-16T07:54:24.784360+0000","last_became_peered":"2024-09-16T07:54:24.784360+0000","last_unstale":"2024-09-16T07:54:26.736548+0000","last_undegraded":"2024-09-16T07:54:26.736548+0000","last_fullsized":"2024-09-16T07:54:26.736548+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T07:54:23.687651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T07:54:23.687651+0000","last_clean_scrub_stamp":"2024-09-16T07:54:23.687651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-16T07:54:26.736485+0000","last_change":"2024-09-16T07:54:26.739464+0000","last_active":"2024-09-16T07:54:26.736485+0000","last_peered":"2024-09-16T07:54:26.736485+0000","last_clean":"2024-09-16T07:54:26.736485+0000","last_became_active":"2024-09-16T07:54:24.805913+0000","last_became_peered":"2024-09-16T07:54:24.805913+0000","last_unstale":"2024-09-16T07:54:26.736485+0000","last_undegraded":"2024-09-16T07:54:26.736485+0000","last_fullsized":"2024-09-16T07:54:26.736485+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T07:54:23.687651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T07:54:23.687651+0000","last_clean_scrub_stamp":"2024-09-16T07:54:23.687651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-16T07:54:26.732090+0000","last_change":"2024-09-16T07:54:26.735332+0000","last_active":"2024-09-16T07:54:26.732090+0000","last_peered":"2024-09-16T07:54:26.732090+0000","last_clean":"2024-09-16T07:54:26.732090+0000","last_became_active":"2024-09-16T07:54:24.773365+0000","last_became_peered":"2024-09-16T07:54:24.773365+0000","last_unstale":"2024-09-16T07:54:26.732090+0000","last_undegraded":"2024-09-16T07:54:26.732090+0000","last_fullsized":"2024-09-16T07:54:26.732090+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T07:54:23.687651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T07:54:23.687651+0000","last_clean_scrub_stamp":"2024-09-16T07:54:23.687651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-16T07:54:26.730092+0000","last_change":"2024-09-16T07:54:26.734682+0000","last_active":"2024-09-16T07:54:26.730092+0000","last_peered":"2024-09-16T07:54:26.730092+0000","last_clean":"2024-09-16T07:54:26.730092+0000","last_became_active":"2024-09-16T07:54:24.756974+0000","last_became_peered":"2024-09-16T07:54:24.756974+0000","last_unstale":"2024-09-16T07:54:26.730092+0000","last_undegraded":"2024-09-16T07:54:26.730092+0000","last_fullsized":"2024-09-16T07:54:26.730092+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T07:54:23.687651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T07:54:23.687651+0000","last_clean_scrub_stamp":"2024-09-16T07:54:23.687651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"15'2","reported_seq":17,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-16T07:54:26.741927+0000","last_change":"2024-09-16T07:54:26.741927+0000","last_active":"2024-09-16T07:54:26.741927+0000","last_peered":"2024-09-16T07:54:26.741927+0000","last_clean":"2024-09-16T07:54:26.741927+0000","last_became_active":"2024-09-16T07:54:24.794754+0000","last_became_peered":"2024-09-16T07:54:24.794754+0000","last_unstale":"2024-09-16T07:54:26.741927+0000","last_undegraded":"2024-09-16T07:54:26.741927+0000","last_fullsized":"2024-09-16T07:54:26.741927+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T07:54:23.687651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T07:54:23.687651+0000","last_clean_scrub_stamp":"2024-09-16T07:54:23.687651+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-16T07:54:26.730661+0000","last_change":"2024-09-16T07:54:26.734478+0000","last_active":"2024-09-16T07:54:26.730661+0000","last_peered":"2024-09-16T07:54:26.730661+0000","last_clean":"2024-09-16T07:54:26.730661+0000","last_became_active":"2024-09-16T07:54:24.774923+0000","last_became_peered":"2024-09-16T07:54:24.774923+0000","last_unstale":"2024-09-16T07:54:26.730661+0000","last_undegraded":"2024-09-16T07:54:26.730661+0000","last_fullsized":"2024-09-16T07:54:26.730661+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T07:54:23.687651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T07:54:23.687651+0000","last_clean_scrub_stamp":"2024-09-16T07:54:23.687651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-16T07:54:26.733855+0000","last_change":"2024-09-16T07:54:26.742127+0000","last_active":"2024-09-16T07:54:26.733855+0000","last_peered":"2024-09-16T07:54:26.733855+0000","last_clean":"2024-09-16T07:54:26.733855+0000","last_became_active":"2024-09-16T07:54:24.773472+0000","last_became_peered":"2024-09-16T07:54:24.773472+0000","last_unstale":"2024-09-16T07:54:26.733855+0000","last_undegraded":"2024-09-16T07:54:26.733855+0000","last_fullsized":"2024-09-16T07:54:26.733855+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T07:54:23.687651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T07:54:23.687651+0000","last_clean_scrub_stamp":"2024-09-16T07:54:23.687651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"13'1","reported_seq":16,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-16T07:54:26.735960+0000","last_change":"2024-09-16T07:54:26.739091+0000","last_active":"2024-09-16T07:54:26.735960+0000","last_peered":"2024-09-16T07:54:26.735960+0000","last_clean":"2024-09-16T07:54:26.735960+0000","last_became_active":"2024-09-16T07:54:24.801907+0000","last_became_peered":"2024-09-16T07:54:24.801907+0000","last_unstale":"2024-09-16T07:54:26.735960+0000","last_undegraded":"2024-09-16T07:54:26.735960+0000","last_fullsized":"2024-09-16T07:54:26.735960+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T07:54:23.687651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T07:54:23.687651+0000","last_clean_scrub_stamp":"2024-09-16T07:54:23.687651+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'52","reported_seq":87,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-16T07:54:26.730190+0000","last_change":"2024-09-16T07:54:22.755090+0000","last_active":"2024-09-16T07:54:26.730190+0000","last_peered":"2024-09-16T07:54:26.730190+0000","last_clean":"2024-09-16T07:54:26.730190+0000","last_became_active":"2024-09-16T07:54:22.736266+0000","last_became_peered":"2024-09-16T07:54:22.736266+0000","last_unstale":"2024-09-16T07:54:26.730190+0000","last_undegraded":"2024-09-16T07:54:26.730190+0000","last_fullsized":"2024-09-16T07:54:26.730190+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T07:54:21.678374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T07:54:21.678374+0000","last_clean_scrub_stamp":"2024-09-16T07:54:21.678374+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":214211493888,"available":214207741952,"internally_reserved":0,"allocated":3751936,"data_stored":3751936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53552873472,"available":53551673344,"internally_reserved":0,"allocated":1200128,"data_stored":1200128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705672,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":1172,"kb_used_data":1172,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52296556,"statfs":{"total":53552873472,"available":53551673344,"internally_reserved":0,"allocated":1200128,"data_stored":1200128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705672,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":532,"kb_used_data":532,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52297196,"statfs":{"total":53552873472,"available":53552328704,"internally_reserved":0,"allocated":544768,"data_stored":544768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705672,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":528,"kb_used_data":528,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52297200,"statfs":{"total":53552873472,"available":53552332800,"internally_reserved":0,"allocated":540672,"data_stored":540672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705672,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":1432,"kb_used_data":1432,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52296296,"statfs":{"total":53552873472,"available":53551407104,"internally_reserved":0,"allocated":1466368,"data_stored":1466368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":3,"total":53552873472,"available":53551673344,"internally_reserved":0,"allocated":1200128,"data_stored":1200128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53552873472,"available":53551407104,"internally_reserved":0,"allocated":1466368,"data_stored":1466368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53552873472,"available":53552332800,"internally_reserved":0,"allocated":540672,"data_stored":540672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53552873472,"available":53552328704,"internally_reserved":0,"allocated":544768,"data_stored":544768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":53552873472,"available":53551673344,"internally_reserved":0,"allocated":1200128,"data_stored":1200128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T07:54:29.422 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-16T07:54:29.422 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-16T07:54:29.422 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-09-16T07:54:29.422 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-09-16T07:54:29.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:29.532+0000 7fcf9cb11640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T07:54:29.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:29.536+0000 7fcf9cb11640 1 Processor -- start 2024-09-16T07:54:29.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:29.536+0000 7fcf9cb11640 1 -- start start 2024-09-16T07:54:29.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:29.538+0000 7fcf9cb11640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fcf981333b0 0x7fcf981337c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:54:29.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:29.538+0000 7fcf9cb11640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcf980693f0 con 0x7fcf98133d00 2024-09-16T07:54:29.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:29.538+0000 7fcf9cb11640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 -- 0x7fcf980720d0 con 0x7fcf981333b0 2024-09-16T07:54:29.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:29.538+0000 7fcf9659c640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fcf981333b0 0x7fcf981337c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:54:29.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:29.538+0000 7fcf8ffff640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fcf98133d00 0x7fcf9809a240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:46654/0 (socket says 172.21.15.27:46654) 2024-09-16T07:54:29.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:29.538+0000 7fcf9659c640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fcf981333b0 0x7fcf981337c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56572/0 (socket says 172.21.15.27:56572) 2024-09-16T07:54:29.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:29.538+0000 7fcf8ffff640 1 -- 172.21.15.27:0/4026394941 learned_addr learned my addr 172.21.15.27:0/4026394941 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-09-16T07:54:29.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:29.538+0000 7fcf9659c640 1 -- 172.21.15.27:0/4026394941 >> v1:172.21.15.27:6789/0 conn(0x7fcf98133d00 legacy=0x7fcf9809a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:54:29.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:29.538+0000 7fcf9659c640 1 -- 172.21.15.27:0/4026394941 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf98074b00 con 0x7fcf981333b0 2024-09-16T07:54:29.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:29.538+0000 7fcf9659c640 1 --2- 172.21.15.27:0/4026394941 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcf981333b0 0x7fcf981337c0 crc :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e72f9759ae384ba5 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:54:29.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:29.539+0000 7fcf95d9b640 1 -- 172.21.15.27:0/4026394941 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fcf88009d90 con 0x7fcf981333b0 2024-09-16T07:54:29.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:29.539+0000 7fcf95d9b640 1 -- 172.21.15.27:0/4026394941 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7fcf8800e230 con 0x7fcf981333b0 2024-09-16T07:54:29.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:29.539+0000 7fcf95d9b640 1 -- 172.21.15.27:0/4026394941 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fcf8800e4d0 con 0x7fcf981333b0 2024-09-16T07:54:29.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:29.539+0000 7fcf9cb11640 1 -- 172.21.15.27:0/4026394941 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcf981333b0 msgr2=0x7fcf981337c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:54:29.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:29.539+0000 7fcf9cb11640 1 --2- 172.21.15.27:0/4026394941 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcf981333b0 0x7fcf981337c0 crc :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:54:29.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:29.539+0000 7fcf9cb11640 1 -- 172.21.15.27:0/4026394941 shutdown_connections 2024-09-16T07:54:29.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:29.539+0000 7fcf9cb11640 1 --2- 172.21.15.27:0/4026394941 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcf981333b0 0x7fcf981337c0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:54:29.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:29.539+0000 7fcf9cb11640 1 -- 172.21.15.27:0/4026394941 >> 172.21.15.27:0/4026394941 conn(0x7fcf980a1dd0 msgr2=0x7fcf980a21e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:54:29.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:29.539+0000 7fcf9cb11640 1 -- 172.21.15.27:0/4026394941 shutdown_connections 2024-09-16T07:54:29.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:29.540+0000 7fcf9cb11640 1 -- 172.21.15.27:0/4026394941 wait complete. 2024-09-16T07:54:29.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:29.540+0000 7fcf9cb11640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T07:54:29.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:29.542+0000 7fcf9cb11640 1 Processor -- start 2024-09-16T07:54:29.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:29.542+0000 7fcf9cb11640 1 -- start start 2024-09-16T07:54:29.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:29.543+0000 7fcf9cb11640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcf981333b0 0x7fcf98129d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:54:29.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:29.543+0000 7fcf9cb11640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fcf9812b0c0 con 0x7fcf981333b0 2024-09-16T07:54:29.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:29.543+0000 7fcf9659c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcf981333b0 0x7fcf98129d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:54:29.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:29.543+0000 7fcf9659c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcf981333b0 0x7fcf98129d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56580/0 (socket says 172.21.15.27:56580) 2024-09-16T07:54:29.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:29.543+0000 7fcf9659c640 1 -- 172.21.15.27:0/3032119868 learned_addr learned my addr 172.21.15.27:0/3032119868 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-09-16T07:54:29.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:29.544+0000 7fcf9659c640 1 -- 172.21.15.27:0/3032119868 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf98130e50 con 0x7fcf981333b0 2024-09-16T07:54:29.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:29.544+0000 7fcf9659c640 1 --2- 172.21.15.27:0/3032119868 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcf981333b0 0x7fcf98129d80 crc :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:54:29.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:29.544+0000 7fcf8effd640 1 -- 172.21.15.27:0/3032119868 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fcf88014620 con 0x7fcf981333b0 2024-09-16T07:54:29.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:29.544+0000 7fcf8effd640 1 -- 172.21.15.27:0/3032119868 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7fcf88014780 con 0x7fcf981333b0 2024-09-16T07:54:29.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:29.544+0000 7fcf9cb11640 1 -- 172.21.15.27:0/3032119868 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcf9812dd10 con 0x7fcf981333b0 2024-09-16T07:54:29.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:29.544+0000 7fcf8effd640 1 -- 172.21.15.27:0/3032119868 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fcf88014a20 con 0x7fcf981333b0 2024-09-16T07:54:29.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:29.544+0000 7fcf9cb11640 1 -- 172.21.15.27:0/3032119868 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcf9809fc00 con 0x7fcf981333b0 2024-09-16T07:54:29.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:29.545+0000 7fcf9cb11640 1 -- 172.21.15.27:0/3032119868 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcf98074430 con 0x7fcf981333b0 2024-09-16T07:54:29.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:29.545+0000 7fcf8effd640 1 -- 172.21.15.27:0/3032119868 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) ==== 50954+0+0 (crc 0 0 0) 0x7fcf88014bd0 con 0x7fcf981333b0 2024-09-16T07:54:29.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:29.545+0000 7fcf8effd640 1 --2- 172.21.15.27:0/3032119868 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7fcf7803e930 0x7fcf78040e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:54:29.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:29.545+0000 7fcf8effd640 1 -- 172.21.15.27:0/3032119868 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(15..15 src has 1..15) ==== 2728+0+0 (crc 0 0 0) 0x7fcf88019040 con 0x7fcf981333b0 2024-09-16T07:54:29.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:29.546+0000 7fcf8ffff640 1 --2- 172.21.15.27:0/3032119868 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7fcf7803e930 0x7fcf78040e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:54:29.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:29.546+0000 7fcf8ffff640 1 --2- 172.21.15.27:0/3032119868 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7fcf7803e930 0x7fcf78040e40 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:54:29.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:29.550+0000 7fcf8effd640 1 -- 172.21.15.27:0/3032119868 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (crc 0 0 0) 0x7fcf88012390 con 0x7fcf981333b0 2024-09-16T07:54:29.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:29.778+0000 7fcf9cb11640 1 -- 172.21.15.27:0/3032119868 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7fcf98069cc0 con 0x7fcf981333b0 2024-09-16T07:54:29.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:29.778+0000 7fcf8effd640 1 -- 172.21.15.27:0/3032119868 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (crc 0 0 0) 0x7fcf88008f20 con 0x7fcf981333b0 2024-09-16T07:54:29.777 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-16T07:54:29.777 INFO:teuthology.orchestra.run.smithi027.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-16T07:54:29.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:29.781+0000 7fcf9cb11640 1 -- 172.21.15.27:0/3032119868 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7fcf7803e930 msgr2=0x7fcf78040e40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:54:29.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:29.781+0000 7fcf9cb11640 1 --2- 172.21.15.27:0/3032119868 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7fcf7803e930 0x7fcf78040e40 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:54:29.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:29.781+0000 7fcf9cb11640 1 -- 172.21.15.27:0/3032119868 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcf981333b0 msgr2=0x7fcf98129d80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:54:29.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:29.781+0000 7fcf9cb11640 1 --2- 172.21.15.27:0/3032119868 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcf981333b0 0x7fcf98129d80 crc :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:54:29.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:29.781+0000 7fcf9cb11640 1 -- 172.21.15.27:0/3032119868 shutdown_connections 2024-09-16T07:54:29.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:29.781+0000 7fcf9cb11640 1 --2- 172.21.15.27:0/3032119868 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7fcf7803e930 0x7fcf78040e40 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:54:29.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:29.781+0000 7fcf9cb11640 1 --2- 172.21.15.27:0/3032119868 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcf981333b0 0x7fcf98129d80 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:54:29.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:29.781+0000 7fcf9cb11640 1 -- 172.21.15.27:0/3032119868 >> 172.21.15.27:0/3032119868 conn(0x7fcf980a1dd0 msgr2=0x7fcf981bb600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:54:29.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:29.781+0000 7fcf9cb11640 1 -- 172.21.15.27:0/3032119868 shutdown_connections 2024-09-16T07:54:29.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:54:29.781+0000 7fcf9cb11640 1 -- 172.21.15.27:0/3032119868 wait complete. 2024-09-16T07:54:29.790 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-09-16T07:54:29.790 INFO:teuthology.run_tasks:Running task workunit... 2024-09-16T07:54:29.803 INFO:tasks.workunit:Pulling workunits from ref 0325fd3c11a225c0fd9eb38fab88aad1370ad779 2024-09-16T07:54:29.804 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-09-16T07:54:29.804 DEBUG:teuthology.orchestra.run.smithi027:> stat -- /home/ubuntu/cephtest/mnt.0 2024-09-16T07:54:29.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T07:54:29.820 INFO:teuthology.orchestra.run.smithi027.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-09-16T07:54:29.820 DEBUG:teuthology.orchestra.run.smithi027:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-09-16T07:54:29.877 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-09-16T07:54:29.877 DEBUG:teuthology.orchestra.run.smithi027:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-09-16T07:54:29.932 INFO:tasks.workunit:timeout=3h 2024-09-16T07:54:29.933 INFO:tasks.workunit:cleanup=True 2024-09-16T07:54:29.933 DEBUG:teuthology.orchestra.run.smithi027:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/ceph/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 0325fd3c11a225c0fd9eb38fab88aad1370ad779 2024-09-16T07:54:29.989 INFO:tasks.workunit.client.0.smithi027.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-09-16T07:55:19.727 INFO:tasks.workunit.client.0.smithi027.stderr:Updating files: 98% (11872/12088) Updating files: 99% (11968/12088) Updating files: 100% (12088/12088) Updating files: 100% (12088/12088), done. 2024-09-16T07:55:20.742 INFO:tasks.workunit.client.0.smithi027.stderr:Note: switching to '0325fd3c11a225c0fd9eb38fab88aad1370ad779'. 2024-09-16T07:55:20.742 INFO:tasks.workunit.client.0.smithi027.stderr: 2024-09-16T07:55:20.742 INFO:tasks.workunit.client.0.smithi027.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-09-16T07:55:20.743 INFO:tasks.workunit.client.0.smithi027.stderr:changes and commit them, and you can discard any commits you make in this 2024-09-16T07:55:20.743 INFO:tasks.workunit.client.0.smithi027.stderr:state without impacting any branches by switching back to a branch. 2024-09-16T07:55:20.743 INFO:tasks.workunit.client.0.smithi027.stderr: 2024-09-16T07:55:20.743 INFO:tasks.workunit.client.0.smithi027.stderr:If you want to create a new branch to retain commits you create, you may 2024-09-16T07:55:20.743 INFO:tasks.workunit.client.0.smithi027.stderr:do so (now or later) by using -c with the switch command. Example: 2024-09-16T07:55:20.743 INFO:tasks.workunit.client.0.smithi027.stderr: 2024-09-16T07:55:20.743 INFO:tasks.workunit.client.0.smithi027.stderr: git switch -c 2024-09-16T07:55:20.743 INFO:tasks.workunit.client.0.smithi027.stderr: 2024-09-16T07:55:20.743 INFO:tasks.workunit.client.0.smithi027.stderr:Or undo this operation with: 2024-09-16T07:55:20.743 INFO:tasks.workunit.client.0.smithi027.stderr: 2024-09-16T07:55:20.743 INFO:tasks.workunit.client.0.smithi027.stderr: git switch - 2024-09-16T07:55:20.743 INFO:tasks.workunit.client.0.smithi027.stderr: 2024-09-16T07:55:20.744 INFO:tasks.workunit.client.0.smithi027.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-09-16T07:55:20.744 INFO:tasks.workunit.client.0.smithi027.stderr: 2024-09-16T07:55:20.744 INFO:tasks.workunit.client.0.smithi027.stderr:HEAD is now at 0325fd3c11a Merge pull request #58530 from zdover23/wip-doc-2024-07-11-radosgw-s3-authentication 2024-09-16T07:55:20.748 DEBUG:teuthology.orchestra.run.smithi027:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-09-16T07:55:20.815 INFO:tasks.workunit.client.0.smithi027.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-09-16T07:55:20.817 INFO:tasks.workunit.client.0.smithi027.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-09-16T07:55:20.817 INFO:tasks.workunit.client.0.smithi027.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-09-16T07:55:21.566 INFO:tasks.workunit.client.0.smithi027.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-09-16T07:55:21.639 INFO:tasks.workunit.client.0.smithi027.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-09-16T07:55:21.683 INFO:tasks.workunit.client.0.smithi027.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-09-16T07:55:21.684 INFO:tasks.workunit.client.0.smithi027.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-09-16T07:55:21.684 INFO:tasks.workunit.client.0.smithi027.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-09-16T07:55:21.725 INFO:tasks.workunit.client.0.smithi027.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-09-16T07:55:21.729 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-09-16T07:55:21.729 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-09-16T07:55:21.786 INFO:tasks.workunit:Running workunits matching rados/test.sh on client.0... 2024-09-16T07:55:21.789 INFO:tasks.workunit:Running workunit rados/test.sh... 2024-09-16T07:55:21.789 DEBUG:teuthology.orchestra.run.smithi027:workunit test rados/test.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=0325fd3c11a225c0fd9eb38fab88aad1370ad779 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 CRIMSON_COMPAT=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh --crimson 2024-09-16T07:55:21.847 INFO:tasks.workunit.client.0.smithi027.stderr:+ parallel=1 2024-09-16T07:55:21.847 INFO:tasks.workunit.client.0.smithi027.stderr:+ '[' --crimson = --serial ']' 2024-09-16T07:55:21.847 INFO:tasks.workunit.client.0.smithi027.stderr:+ crimson=0 2024-09-16T07:55:21.848 INFO:tasks.workunit.client.0.smithi027.stderr:+ '[' --crimson = --crimson ']' 2024-09-16T07:55:21.848 INFO:tasks.workunit.client.0.smithi027.stderr:+ parallel=0 2024-09-16T07:55:21.848 INFO:tasks.workunit.client.0.smithi027.stderr:+ crimson=1 2024-09-16T07:55:21.848 INFO:tasks.workunit.client.0.smithi027.stderr:+ color= 2024-09-16T07:55:21.848 INFO:tasks.workunit.client.0.smithi027.stderr:+ '[' -t 1 ']' 2024-09-16T07:55:21.848 INFO:tasks.workunit.client.0.smithi027.stderr:+ trap cleanup EXIT ERR HUP INT QUIT 2024-09-16T07:55:21.848 INFO:tasks.workunit.client.0.smithi027.stderr:+ GTEST_OUTPUT_DIR=/home/ubuntu/cephtest/archive/unit_test_xml_report 2024-09-16T07:55:21.848 INFO:tasks.workunit.client.0.smithi027.stderr:+ mkdir -p /home/ubuntu/cephtest/archive/unit_test_xml_report 2024-09-16T07:55:21.849 INFO:tasks.workunit.client.0.smithi027.stderr:+ declare -A pids 2024-09-16T07:55:21.849 INFO:tasks.workunit.client.0.smithi027.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-16T07:55:21.849 INFO:tasks.workunit.client.0.smithi027.stderr:+ '[' 0 -eq 1 ']' 2024-09-16T07:55:21.849 INFO:tasks.workunit.client.0.smithi027.stderr:+ ceph_test_rados_api_aio 2024-09-16T07:55:21.866 INFO:tasks.workunit.client.0.smithi027.stdout:Running main() from gmock_main.cc 2024-09-16T07:55:21.866 INFO:tasks.workunit.client.0.smithi027.stdout:[==========] Running 42 tests from 2 test suites. 2024-09-16T07:55:21.866 INFO:tasks.workunit.client.0.smithi027.stdout:[----------] Global test environment set-up. 2024-09-16T07:55:21.866 INFO:tasks.workunit.client.0.smithi027.stdout:[----------] 26 tests from LibRadosAio 2024-09-16T07:55:21.866 INFO:tasks.workunit.client.0.smithi027.stdout:[ RUN ] LibRadosAio.TooBig 2024-09-16T07:55:24.756 INFO:tasks.workunit.client.0.smithi027.stdout:[ OK ] LibRadosAio.TooBig (2890 ms) 2024-09-16T07:55:24.756 INFO:tasks.workunit.client.0.smithi027.stdout:[ RUN ] LibRadosAio.SimpleWrite 2024-09-16T07:55:27.781 INFO:tasks.workunit.client.0.smithi027.stdout:[ OK ] LibRadosAio.SimpleWrite (3025 ms) 2024-09-16T07:55:27.781 INFO:tasks.workunit.client.0.smithi027.stdout:[ RUN ] LibRadosAio.WaitForSafe 2024-09-16T07:55:30.793 INFO:tasks.workunit.client.0.smithi027.stdout:[ OK ] LibRadosAio.WaitForSafe (3012 ms) 2024-09-16T07:55:30.793 INFO:tasks.workunit.client.0.smithi027.stdout:[ RUN ] LibRadosAio.RoundTrip 2024-09-16T07:55:33.810 INFO:tasks.workunit.client.0.smithi027.stdout:[ OK ] LibRadosAio.RoundTrip (3017 ms) 2024-09-16T07:55:33.810 INFO:tasks.workunit.client.0.smithi027.stdout:[ RUN ] LibRadosAio.RoundTrip2 2024-09-16T07:55:36.819 INFO:tasks.workunit.client.0.smithi027.stdout:[ OK ] LibRadosAio.RoundTrip2 (3009 ms) 2024-09-16T07:55:36.819 INFO:tasks.workunit.client.0.smithi027.stdout:[ RUN ] LibRadosAio.RoundTrip3 2024-09-16T07:55:39.837 INFO:tasks.workunit.client.0.smithi027.stdout:[ OK ] LibRadosAio.RoundTrip3 (3018 ms) 2024-09-16T07:55:39.837 INFO:tasks.workunit.client.0.smithi027.stdout:[ RUN ] LibRadosAio.RoundTripAppend 2024-09-16T07:55:42.849 INFO:tasks.workunit.client.0.smithi027.stdout:[ OK ] LibRadosAio.RoundTripAppend (3011 ms) 2024-09-16T07:55:42.850 INFO:tasks.workunit.client.0.smithi027.stdout:[ RUN ] LibRadosAio.RemoveTest 2024-09-16T07:55:45.864 INFO:tasks.workunit.client.0.smithi027.stdout:[ OK ] LibRadosAio.RemoveTest (3016 ms) 2024-09-16T07:55:45.864 INFO:tasks.workunit.client.0.smithi027.stdout:[ RUN ] LibRadosAio.XattrsRoundTrip 2024-09-16T07:55:48.873 INFO:tasks.workunit.client.0.smithi027.stdout:[ OK ] LibRadosAio.XattrsRoundTrip (3009 ms) 2024-09-16T07:55:48.873 INFO:tasks.workunit.client.0.smithi027.stdout:[ RUN ] LibRadosAio.RmXattr 2024-09-16T07:55:51.896 INFO:tasks.workunit.client.0.smithi027.stdout:[ OK ] LibRadosAio.RmXattr (3023 ms) 2024-09-16T07:55:51.896 INFO:tasks.workunit.client.0.smithi027.stdout:[ RUN ] LibRadosAio.XattrIter 2024-09-16T07:55:54.912 INFO:tasks.workunit.client.0.smithi027.stdout:[ OK ] LibRadosAio.XattrIter (3016 ms) 2024-09-16T07:55:54.912 INFO:tasks.workunit.client.0.smithi027.stdout:[ RUN ] LibRadosAio.IsComplete 2024-09-16T07:55:57.930 INFO:tasks.workunit.client.0.smithi027.stdout:[ OK ] LibRadosAio.IsComplete (3018 ms) 2024-09-16T07:55:57.930 INFO:tasks.workunit.client.0.smithi027.stdout:[ RUN ] LibRadosAio.IsSafe 2024-09-16T07:56:00.949 INFO:tasks.workunit.client.0.smithi027.stdout:[ OK ] LibRadosAio.IsSafe (3020 ms) 2024-09-16T07:56:00.950 INFO:tasks.workunit.client.0.smithi027.stdout:[ RUN ] LibRadosAio.ReturnValue 2024-09-16T07:56:03.962 INFO:tasks.workunit.client.0.smithi027.stdout:[ OK ] LibRadosAio.ReturnValue (3013 ms) 2024-09-16T07:56:03.962 INFO:tasks.workunit.client.0.smithi027.stdout:[ RUN ] LibRadosAio.Flush 2024-09-16T07:56:06.974 INFO:tasks.workunit.client.0.smithi027.stdout:[ OK ] LibRadosAio.Flush (3011 ms) 2024-09-16T07:56:06.974 INFO:tasks.workunit.client.0.smithi027.stdout:[ RUN ] LibRadosAio.FlushAsync 2024-09-16T07:56:09.991 INFO:tasks.workunit.client.0.smithi027.stdout:[ OK ] LibRadosAio.FlushAsync (3017 ms) 2024-09-16T07:56:09.991 INFO:tasks.workunit.client.0.smithi027.stdout:[ RUN ] LibRadosAio.RoundTripWriteFull 2024-09-16T07:56:13.002 INFO:tasks.workunit.client.0.smithi027.stdout:[ OK ] LibRadosAio.RoundTripWriteFull (3011 ms) 2024-09-16T07:56:13.002 INFO:tasks.workunit.client.0.smithi027.stdout:[ RUN ] LibRadosAio.RoundTripWriteSame 2024-09-16T07:56:16.022 INFO:tasks.workunit.client.0.smithi027.stdout:[ OK ] LibRadosAio.RoundTripWriteSame (3021 ms) 2024-09-16T07:56:16.023 INFO:tasks.workunit.client.0.smithi027.stdout:[ RUN ] LibRadosAio.SimpleStat 2024-09-16T07:56:18.596 INFO:tasks.workunit.client.0.smithi027.stdout:[ OK ] LibRadosAio.SimpleStat (2573 ms) 2024-09-16T07:56:18.596 INFO:tasks.workunit.client.0.smithi027.stdout:[ RUN ] LibRadosAio.OperateMtime 2024-09-16T07:56:21.605 INFO:tasks.workunit.client.0.smithi027.stdout:[ OK ] LibRadosAio.OperateMtime (3009 ms) 2024-09-16T07:56:21.605 INFO:tasks.workunit.client.0.smithi027.stdout:[ RUN ] LibRadosAio.Operate2Mtime 2024-09-16T07:56:24.606 INFO:tasks.workunit.client.0.smithi027.stdout:[ OK ] LibRadosAio.Operate2Mtime (3001 ms) 2024-09-16T07:56:24.606 INFO:tasks.workunit.client.0.smithi027.stdout:[ RUN ] LibRadosAio.SimpleStatNS 2024-09-16T07:56:27.625 INFO:tasks.workunit.client.0.smithi027.stdout:[ OK ] LibRadosAio.SimpleStatNS (3019 ms) 2024-09-16T07:56:27.625 INFO:tasks.workunit.client.0.smithi027.stdout:[ RUN ] LibRadosAio.StatRemove 2024-09-16T07:56:30.632 INFO:tasks.workunit.client.0.smithi027.stdout:[ OK ] LibRadosAio.StatRemove (3007 ms) 2024-09-16T07:56:30.632 INFO:tasks.workunit.client.0.smithi027.stdout:[ RUN ] LibRadosAio.ExecuteClass 2024-09-16T07:56:33.643 INFO:tasks.workunit.client.0.smithi027.stdout:[ OK ] LibRadosAio.ExecuteClass (3012 ms) 2024-09-16T07:56:33.644 INFO:tasks.workunit.client.0.smithi027.stdout:[ RUN ] LibRadosAio.MultiWrite 2024-09-16T07:56:36.664 INFO:tasks.workunit.client.0.smithi027.stdout:[ OK ] LibRadosAio.MultiWrite (3020 ms) 2024-09-16T07:56:36.665 INFO:tasks.workunit.client.0.smithi027.stdout:[ RUN ] LibRadosAio.AioUnlock 2024-09-16T07:56:39.692 INFO:tasks.workunit.client.0.smithi027.stdout:[ OK ] LibRadosAio.AioUnlock (3025 ms) 2024-09-16T07:56:39.692 INFO:tasks.workunit.client.0.smithi027.stdout:[----------] 26 tests from LibRadosAio (77823 ms total) 2024-09-16T07:56:39.692 INFO:tasks.workunit.client.0.smithi027.stdout: 2024-09-16T07:56:39.692 INFO:tasks.workunit.client.0.smithi027.stdout:[----------] 16 tests from LibRadosAioEC 2024-09-16T07:56:39.692 INFO:tasks.workunit.client.0.smithi027.stdout:[ RUN ] LibRadosAioEC.SimpleWrite 2024-09-16T07:56:39.693 INFO:tasks.workunit.client.0.smithi027.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.3.0-4879-g81fedcb3/rpm/el9/BUILD/ceph-19.3.0-4879-g81fedcb3/src/test/librados/aio.cc:1100: Skipped 2024-09-16T07:56:39.693 INFO:tasks.workunit.client.0.smithi027.stdout:Not supported by crimson yet. Skipped 2024-09-16T07:56:39.693 INFO:tasks.workunit.client.0.smithi027.stdout:[ SKIPPED ] LibRadosAioEC.SimpleWrite (1 ms) 2024-09-16T07:56:39.693 INFO:tasks.workunit.client.0.smithi027.stdout:[ RUN ] LibRadosAioEC.WaitForComplete 2024-09-16T07:56:39.693 INFO:tasks.workunit.client.0.smithi027.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.3.0-4879-g81fedcb3/rpm/el9/BUILD/ceph-19.3.0-4879-g81fedcb3/src/test/librados/aio.cc:1132: Skipped 2024-09-16T07:56:39.693 INFO:tasks.workunit.client.0.smithi027.stdout:Not supported by crimson yet. Skipped 2024-09-16T07:56:39.693 INFO:tasks.workunit.client.0.smithi027.stdout:[ SKIPPED ] LibRadosAioEC.WaitForComplete (0 ms) 2024-09-16T07:56:39.693 INFO:tasks.workunit.client.0.smithi027.stdout:[ RUN ] LibRadosAioEC.RoundTrip 2024-09-16T07:56:39.693 INFO:tasks.workunit.client.0.smithi027.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.3.0-4879-g81fedcb3/rpm/el9/BUILD/ceph-19.3.0-4879-g81fedcb3/src/test/librados/aio.cc:1149: Skipped 2024-09-16T07:56:39.693 INFO:tasks.workunit.client.0.smithi027.stdout:Not supported by crimson yet. Skipped 2024-09-16T07:56:39.693 INFO:tasks.workunit.client.0.smithi027.stdout:[ SKIPPED ] LibRadosAioEC.RoundTrip (0 ms) 2024-09-16T07:56:39.693 INFO:tasks.workunit.client.0.smithi027.stdout:[ RUN ] LibRadosAioEC.RoundTrip2 2024-09-16T07:56:39.693 INFO:tasks.workunit.client.0.smithi027.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.3.0-4879-g81fedcb3/rpm/el9/BUILD/ceph-19.3.0-4879-g81fedcb3/src/test/librados/aio.cc:1182: Skipped 2024-09-16T07:56:39.694 INFO:tasks.workunit.client.0.smithi027.stdout:Not supported by crimson yet. Skipped 2024-09-16T07:56:39.694 INFO:tasks.workunit.client.0.smithi027.stdout:[ SKIPPED ] LibRadosAioEC.RoundTrip2 (0 ms) 2024-09-16T07:56:39.694 INFO:tasks.workunit.client.0.smithi027.stdout:[ RUN ] LibRadosAioEC.RoundTripAppend 2024-09-16T07:56:39.694 INFO:tasks.workunit.client.0.smithi027.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.3.0-4879-g81fedcb3/rpm/el9/BUILD/ceph-19.3.0-4879-g81fedcb3/src/test/librados/aio.cc:1215: Skipped 2024-09-16T07:56:39.694 INFO:tasks.workunit.client.0.smithi027.stdout:Not supported by crimson yet. Skipped 2024-09-16T07:56:39.694 INFO:tasks.workunit.client.0.smithi027.stdout:[ SKIPPED ] LibRadosAioEC.RoundTripAppend (0 ms) 2024-09-16T07:56:39.694 INFO:tasks.workunit.client.0.smithi027.stdout:[ RUN ] LibRadosAioEC.IsComplete 2024-09-16T07:56:39.694 INFO:tasks.workunit.client.0.smithi027.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.3.0-4879-g81fedcb3/rpm/el9/BUILD/ceph-19.3.0-4879-g81fedcb3/src/test/librados/aio.cc:1286: Skipped 2024-09-16T07:56:39.694 INFO:tasks.workunit.client.0.smithi027.stdout:Not supported by crimson yet. Skipped 2024-09-16T07:56:39.694 INFO:tasks.workunit.client.0.smithi027.stdout:[ SKIPPED ] LibRadosAioEC.IsComplete (0 ms) 2024-09-16T07:56:39.694 INFO:tasks.workunit.client.0.smithi027.stdout:[ RUN ] LibRadosAioEC.IsSafe 2024-09-16T07:56:39.694 INFO:tasks.workunit.client.0.smithi027.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.3.0-4879-g81fedcb3/rpm/el9/BUILD/ceph-19.3.0-4879-g81fedcb3/src/test/librados/aio.cc:1326: Skipped 2024-09-16T07:56:39.694 INFO:tasks.workunit.client.0.smithi027.stdout:Not supported by crimson yet. Skipped 2024-09-16T07:56:39.695 INFO:tasks.workunit.client.0.smithi027.stdout:[ SKIPPED ] LibRadosAioEC.IsSafe (0 ms) 2024-09-16T07:56:39.695 INFO:tasks.workunit.client.0.smithi027.stdout:[ RUN ] LibRadosAioEC.ReturnValue 2024-09-16T07:56:39.695 INFO:tasks.workunit.client.0.smithi027.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.3.0-4879-g81fedcb3/rpm/el9/BUILD/ceph-19.3.0-4879-g81fedcb3/src/test/librados/aio.cc:1366: Skipped 2024-09-16T07:56:39.695 INFO:tasks.workunit.client.0.smithi027.stdout:Not supported by crimson yet. Skipped 2024-09-16T07:56:39.695 INFO:tasks.workunit.client.0.smithi027.stdout:[ SKIPPED ] LibRadosAioEC.ReturnValue (0 ms) 2024-09-16T07:56:39.695 INFO:tasks.workunit.client.0.smithi027.stdout:[ RUN ] LibRadosAioEC.Flush 2024-09-16T07:56:39.695 INFO:tasks.workunit.client.0.smithi027.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.3.0-4879-g81fedcb3/rpm/el9/BUILD/ceph-19.3.0-4879-g81fedcb3/src/test/librados/aio.cc:1385: Skipped 2024-09-16T07:56:39.695 INFO:tasks.workunit.client.0.smithi027.stdout:Not supported by crimson yet. Skipped 2024-09-16T07:56:39.695 INFO:tasks.workunit.client.0.smithi027.stdout:[ SKIPPED ] LibRadosAioEC.Flush (0 ms) 2024-09-16T07:56:39.695 INFO:tasks.workunit.client.0.smithi027.stdout:[ RUN ] LibRadosAioEC.FlushAsync 2024-09-16T07:56:39.695 INFO:tasks.workunit.client.0.smithi027.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.3.0-4879-g81fedcb3/rpm/el9/BUILD/ceph-19.3.0-4879-g81fedcb3/src/test/librados/aio.cc:1415: Skipped 2024-09-16T07:56:39.695 INFO:tasks.workunit.client.0.smithi027.stdout:Not supported by crimson yet. Skipped 2024-09-16T07:56:39.696 INFO:tasks.workunit.client.0.smithi027.stdout:[ SKIPPED ] LibRadosAioEC.FlushAsync (0 ms) 2024-09-16T07:56:39.696 INFO:tasks.workunit.client.0.smithi027.stdout:[ RUN ] LibRadosAioEC.RoundTripWriteFull 2024-09-16T07:56:39.696 INFO:tasks.workunit.client.0.smithi027.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.3.0-4879-g81fedcb3/rpm/el9/BUILD/ceph-19.3.0-4879-g81fedcb3/src/test/librados/aio.cc:1454: Skipped 2024-09-16T07:56:39.696 INFO:tasks.workunit.client.0.smithi027.stdout:Not supported by crimson yet. Skipped 2024-09-16T07:56:39.696 INFO:tasks.workunit.client.0.smithi027.stdout:[ SKIPPED ] LibRadosAioEC.RoundTripWriteFull (0 ms) 2024-09-16T07:56:39.696 INFO:tasks.workunit.client.0.smithi027.stdout:[ RUN ] LibRadosAioEC.SimpleStat 2024-09-16T07:56:39.696 INFO:tasks.workunit.client.0.smithi027.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.3.0-4879-g81fedcb3/rpm/el9/BUILD/ceph-19.3.0-4879-g81fedcb3/src/test/librados/aio.cc:1498: Skipped 2024-09-16T07:56:39.696 INFO:tasks.workunit.client.0.smithi027.stdout:Not supported by crimson yet. Skipped 2024-09-16T07:56:39.696 INFO:tasks.workunit.client.0.smithi027.stdout:[ SKIPPED ] LibRadosAioEC.SimpleStat (0 ms) 2024-09-16T07:56:39.696 INFO:tasks.workunit.client.0.smithi027.stdout:[ RUN ] LibRadosAioEC.SimpleStatNS 2024-09-16T07:56:39.696 INFO:tasks.workunit.client.0.smithi027.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.3.0-4879-g81fedcb3/rpm/el9/BUILD/ceph-19.3.0-4879-g81fedcb3/src/test/librados/aio.cc:1532: Skipped 2024-09-16T07:56:39.696 INFO:tasks.workunit.client.0.smithi027.stdout:Not supported by crimson yet. Skipped 2024-09-16T07:56:39.697 INFO:tasks.workunit.client.0.smithi027.stdout:[ SKIPPED ] LibRadosAioEC.SimpleStatNS (0 ms) 2024-09-16T07:56:39.697 INFO:tasks.workunit.client.0.smithi027.stdout:[ RUN ] LibRadosAioEC.StatRemove 2024-09-16T07:56:39.697 INFO:tasks.workunit.client.0.smithi027.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.3.0-4879-g81fedcb3/rpm/el9/BUILD/ceph-19.3.0-4879-g81fedcb3/src/test/librados/aio.cc:1593: Skipped 2024-09-16T07:56:39.697 INFO:tasks.workunit.client.0.smithi027.stdout:Not supported by crimson yet. Skipped 2024-09-16T07:56:39.697 INFO:tasks.workunit.client.0.smithi027.stdout:[ SKIPPED ] LibRadosAioEC.StatRemove (0 ms) 2024-09-16T07:56:39.697 INFO:tasks.workunit.client.0.smithi027.stdout:[ RUN ] LibRadosAioEC.ExecuteClass 2024-09-16T07:56:39.697 INFO:tasks.workunit.client.0.smithi027.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.3.0-4879-g81fedcb3/rpm/el9/BUILD/ceph-19.3.0-4879-g81fedcb3/src/test/librados/aio.cc:1649: Skipped 2024-09-16T07:56:39.697 INFO:tasks.workunit.client.0.smithi027.stdout:Not supported by crimson yet. Skipped 2024-09-16T07:56:39.697 INFO:tasks.workunit.client.0.smithi027.stdout:[ SKIPPED ] LibRadosAioEC.ExecuteClass (0 ms) 2024-09-16T07:56:39.697 INFO:tasks.workunit.client.0.smithi027.stdout:[ RUN ] LibRadosAioEC.MultiWrite 2024-09-16T07:56:39.697 INFO:tasks.workunit.client.0.smithi027.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.3.0-4879-g81fedcb3/rpm/el9/BUILD/ceph-19.3.0-4879-g81fedcb3/src/test/librados/aio.cc:1681: Skipped 2024-09-16T07:56:39.697 INFO:tasks.workunit.client.0.smithi027.stdout:Not supported by crimson yet. Skipped 2024-09-16T07:56:39.697 INFO:tasks.workunit.client.0.smithi027.stdout:[ SKIPPED ] LibRadosAioEC.MultiWrite (0 ms) 2024-09-16T07:56:39.697 INFO:tasks.workunit.client.0.smithi027.stdout:[----------] 16 tests from LibRadosAioEC (1 ms total) 2024-09-16T07:56:39.698 INFO:tasks.workunit.client.0.smithi027.stdout: 2024-09-16T07:56:39.698 INFO:tasks.workunit.client.0.smithi027.stdout:[----------] Global test environment tear-down 2024-09-16T07:56:39.698 INFO:tasks.workunit.client.0.smithi027.stdout:[==========] 42 tests from 2 test suites ran. (77824 ms total) 2024-09-16T07:56:39.698 INFO:tasks.workunit.client.0.smithi027.stdout:[ PASSED ] 26 tests. 2024-09-16T07:56:39.698 INFO:tasks.workunit.client.0.smithi027.stdout:[ SKIPPED ] 16 tests, listed below: 2024-09-16T07:56:39.698 INFO:tasks.workunit.client.0.smithi027.stdout:[ SKIPPED ] LibRadosAioEC.SimpleWrite 2024-09-16T07:56:39.698 INFO:tasks.workunit.client.0.smithi027.stdout:[ SKIPPED ] LibRadosAioEC.WaitForComplete 2024-09-16T07:56:39.698 INFO:tasks.workunit.client.0.smithi027.stdout:[ SKIPPED ] LibRadosAioEC.RoundTrip 2024-09-16T07:56:39.698 INFO:tasks.workunit.client.0.smithi027.stdout:[ SKIPPED ] LibRadosAioEC.RoundTrip2 2024-09-16T07:56:39.698 INFO:tasks.workunit.client.0.smithi027.stdout:[ SKIPPED ] LibRadosAioEC.RoundTripAppend 2024-09-16T07:56:39.698 INFO:tasks.workunit.client.0.smithi027.stdout:[ SKIPPED ] LibRadosAioEC.IsComplete 2024-09-16T07:56:39.698 INFO:tasks.workunit.client.0.smithi027.stdout:[ SKIPPED ] LibRadosAioEC.IsSafe 2024-09-16T07:56:39.698 INFO:tasks.workunit.client.0.smithi027.stdout:[ SKIPPED ] LibRadosAioEC.ReturnValue 2024-09-16T07:56:39.698 INFO:tasks.workunit.client.0.smithi027.stdout:[ SKIPPED ] LibRadosAioEC.Flush 2024-09-16T07:56:39.699 INFO:tasks.workunit.client.0.smithi027.stdout:[ SKIPPED ] LibRadosAioEC.FlushAsync 2024-09-16T07:56:39.699 INFO:tasks.workunit.client.0.smithi027.stdout:[ SKIPPED ] LibRadosAioEC.RoundTripWriteFull 2024-09-16T07:56:39.699 INFO:tasks.workunit.client.0.smithi027.stdout:[ SKIPPED ] LibRadosAioEC.SimpleStat 2024-09-16T07:56:39.699 INFO:tasks.workunit.client.0.smithi027.stdout:[ SKIPPED ] LibRadosAioEC.SimpleStatNS 2024-09-16T07:56:39.699 INFO:tasks.workunit.client.0.smithi027.stdout:[ SKIPPED ] LibRadosAioEC.StatRemove 2024-09-16T07:56:39.699 INFO:tasks.workunit.client.0.smithi027.stdout:[ SKIPPED ] LibRadosAioEC.ExecuteClass 2024-09-16T07:56:39.699 INFO:tasks.workunit.client.0.smithi027.stdout:[ SKIPPED ] LibRadosAioEC.MultiWrite 2024-09-16T07:56:39.699 INFO:tasks.workunit.client.0.smithi027.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-16T07:56:39.700 INFO:tasks.workunit.client.0.smithi027.stderr:+ '[' 0 -eq 1 ']' 2024-09-16T07:56:39.700 INFO:tasks.workunit.client.0.smithi027.stderr:+ ceph_test_rados_api_aio_pp 2024-09-16T07:56:39.717 INFO:tasks.workunit.client.0.smithi027.stdout:Running main() from gmock_main.cc 2024-09-16T07:56:39.717 INFO:tasks.workunit.client.0.smithi027.stdout:[==========] Running 57 tests from 4 test suites. 2024-09-16T07:56:39.717 INFO:tasks.workunit.client.0.smithi027.stdout:[----------] Global test environment set-up. 2024-09-16T07:56:39.717 INFO:tasks.workunit.client.0.smithi027.stdout:[----------] 32 tests from LibRadosAio 2024-09-16T07:56:39.717 INFO:tasks.workunit.client.0.smithi027.stdout:[ RUN ] LibRadosAio.TooBigPP 2024-09-16T07:56:42.592 INFO:tasks.workunit.client.0.smithi027.stdout:[ OK ] LibRadosAio.TooBigPP (2875 ms) 2024-09-16T07:56:42.593 INFO:tasks.workunit.client.0.smithi027.stdout:[ RUN ] LibRadosAio.PoolQuotaPP 2024-09-16T07:56:59.630 INFO:tasks.workunit.client.0.smithi027.stdout:[ OK ] LibRadosAio.PoolQuotaPP (17038 ms) 2024-09-16T07:56:59.631 INFO:tasks.workunit.client.0.smithi027.stdout:[ RUN ] LibRadosAio.SimpleWritePP 2024-09-16T07:57:05.680 INFO:tasks.workunit.client.0.smithi027.stdout:[ OK ] LibRadosAio.SimpleWritePP (6050 ms) 2024-09-16T07:57:05.680 INFO:tasks.workunit.client.0.smithi027.stdout:[ RUN ] LibRadosAio.WaitForSafePP 2024-09-16T07:57:08.698 INFO:tasks.workunit.client.0.smithi027.stdout:[ OK ] LibRadosAio.WaitForSafePP (3018 ms) 2024-09-16T07:57:08.698 INFO:tasks.workunit.client.0.smithi027.stdout:[ RUN ] LibRadosAio.RoundTripPP 2024-09-16T07:57:11.720 INFO:tasks.workunit.client.0.smithi027.stdout:[ OK ] LibRadosAio.RoundTripPP (3021 ms) 2024-09-16T07:57:11.720 INFO:tasks.workunit.client.0.smithi027.stdout:[ RUN ] LibRadosAio.RoundTripPP2 2024-09-16T07:57:14.727 INFO:tasks.workunit.client.0.smithi027.stdout:[ OK ] LibRadosAio.RoundTripPP2 (3008 ms) 2024-09-16T07:57:14.728 INFO:tasks.workunit.client.0.smithi027.stdout:[ RUN ] LibRadosAio.RoundTripPP3 2024-09-16T07:57:17.599 INFO:tasks.workunit.client.0.smithi027.stdout:[ OK ] LibRadosAio.RoundTripPP3 (2872 ms) 2024-09-16T07:57:17.600 INFO:tasks.workunit.client.0.smithi027.stdout:[ RUN ] LibRadosAio.RoundTripSparseReadPP 2024-09-16T07:57:20.619 INFO:tasks.workunit.client.0.smithi027.stdout:[ OK ] LibRadosAio.RoundTripSparseReadPP (3019 ms) 2024-09-16T07:57:20.619 INFO:tasks.workunit.client.0.smithi027.stdout:[ RUN ] LibRadosAio.IsCompletePP 2024-09-16T07:57:23.639 INFO:tasks.workunit.client.0.smithi027.stdout:[ OK ] LibRadosAio.IsCompletePP (3020 ms) 2024-09-16T07:57:23.639 INFO:tasks.workunit.client.0.smithi027.stdout:[ RUN ] LibRadosAio.IsSafePP 2024-09-16T07:57:26.653 INFO:tasks.workunit.client.0.smithi027.stdout:[ OK ] LibRadosAio.IsSafePP (3014 ms) 2024-09-16T07:57:26.653 INFO:tasks.workunit.client.0.smithi027.stdout:[ RUN ] LibRadosAio.ReturnValuePP 2024-09-16T07:57:29.664 INFO:tasks.workunit.client.0.smithi027.stdout:[ OK ] LibRadosAio.ReturnValuePP (3011 ms) 2024-09-16T07:57:29.665 INFO:tasks.workunit.client.0.smithi027.stdout:[ RUN ] LibRadosAio.FlushPP 2024-09-16T07:57:32.680 INFO:tasks.workunit.client.0.smithi027.stdout:[ OK ] LibRadosAio.FlushPP (3016 ms) 2024-09-16T07:57:32.680 INFO:tasks.workunit.client.0.smithi027.stdout:[ RUN ] LibRadosAio.FlushAsyncPP 2024-09-16T07:57:35.695 INFO:tasks.workunit.client.0.smithi027.stdout:[ OK ] LibRadosAio.FlushAsyncPP (3015 ms) 2024-09-16T07:57:35.695 INFO:tasks.workunit.client.0.smithi027.stdout:[ RUN ] LibRadosAio.RoundTripWriteFullPP 2024-09-16T07:57:38.709 INFO:tasks.workunit.client.0.smithi027.stdout:[ OK ] LibRadosAio.RoundTripWriteFullPP (3015 ms) 2024-09-16T07:57:38.710 INFO:tasks.workunit.client.0.smithi027.stdout:[ RUN ] LibRadosAio.RoundTripWriteFullPP2 2024-09-16T07:57:41.726 INFO:tasks.workunit.client.0.smithi027.stdout:[ OK ] LibRadosAio.RoundTripWriteFullPP2 (3016 ms) 2024-09-16T07:57:41.726 INFO:tasks.workunit.client.0.smithi027.stdout:[ RUN ] LibRadosAio.RoundTripWriteSamePP 2024-09-16T07:57:44.745 INFO:tasks.workunit.client.0.smithi027.stdout:[ OK ] LibRadosAio.RoundTripWriteSamePP (3019 ms) 2024-09-16T07:57:44.745 INFO:tasks.workunit.client.0.smithi027.stdout:[ RUN ] LibRadosAio.RoundTripWriteSamePP2 2024-09-16T07:57:47.606 INFO:tasks.workunit.client.0.smithi027.stdout:[ OK ] LibRadosAio.RoundTripWriteSamePP2 (2861 ms) 2024-09-16T07:57:47.606 INFO:tasks.workunit.client.0.smithi027.stdout:[ RUN ] LibRadosAio.SimpleStatPPNS 2024-09-16T07:57:50.612 INFO:tasks.workunit.client.0.smithi027.stdout:[ OK ] LibRadosAio.SimpleStatPPNS (3007 ms) 2024-09-16T07:57:50.613 INFO:tasks.workunit.client.0.smithi027.stdout:[ RUN ] LibRadosAio.SimpleStatPP 2024-09-16T07:57:53.627 INFO:tasks.workunit.client.0.smithi027.stdout:[ OK ] LibRadosAio.SimpleStatPP (3014 ms) 2024-09-16T07:57:53.627 INFO:tasks.workunit.client.0.smithi027.stdout:[ RUN ] LibRadosAio.OperateMtime 2024-09-16T07:57:56.640 INFO:tasks.workunit.client.0.smithi027.stdout:[ OK ] LibRadosAio.OperateMtime (3014 ms) 2024-09-16T07:57:56.641 INFO:tasks.workunit.client.0.smithi027.stdout:[ RUN ] LibRadosAio.OperateMtime2 2024-09-16T07:57:59.656 INFO:tasks.workunit.client.0.smithi027.stdout:[ OK ] LibRadosAio.OperateMtime2 (3015 ms) 2024-09-16T07:57:59.656 INFO:tasks.workunit.client.0.smithi027.stdout:[ RUN ] LibRadosAio.StatRemovePP 2024-09-16T07:58:02.662 INFO:tasks.workunit.client.0.smithi027.stdout:[ OK ] LibRadosAio.StatRemovePP (3006 ms) 2024-09-16T07:58:02.662 INFO:tasks.workunit.client.0.smithi027.stdout:[ RUN ] LibRadosAio.ExecuteClassPP 2024-09-16T07:58:05.687 INFO:tasks.workunit.client.0.smithi027.stdout:[ OK ] LibRadosAio.ExecuteClassPP (3025 ms) 2024-09-16T07:58:05.687 INFO:tasks.workunit.client.0.smithi027.stdout:[ RUN ] LibRadosAio.OmapPP 2024-09-16T07:58:08.697 INFO:tasks.workunit.client.0.smithi027.stdout:[ OK ] LibRadosAio.OmapPP (3010 ms) 2024-09-16T07:58:08.697 INFO:tasks.workunit.client.0.smithi027.stdout:[ RUN ] LibRadosAio.MultiWritePP 2024-09-16T07:58:11.711 INFO:tasks.workunit.client.0.smithi027.stdout:[ OK ] LibRadosAio.MultiWritePP (3013 ms) 2024-09-16T07:58:11.711 INFO:tasks.workunit.client.0.smithi027.stdout:[ RUN ] LibRadosAio.AioUnlockPP 2024-09-16T07:58:14.727 INFO:tasks.workunit.client.0.smithi027.stdout:[ OK ] LibRadosAio.AioUnlockPP (3017 ms) 2024-09-16T07:58:14.727 INFO:tasks.workunit.client.0.smithi027.stdout:[ RUN ] LibRadosAio.RoundTripAppendPP 2024-09-16T07:58:17.613 INFO:tasks.workunit.client.0.smithi027.stdout:[ OK ] LibRadosAio.RoundTripAppendPP (2886 ms) 2024-09-16T07:58:17.613 INFO:tasks.workunit.client.0.smithi027.stdout:[ RUN ] LibRadosAio.RacingRemovePP 2024-09-16T07:58:20.622 INFO:tasks.workunit.client.0.smithi027.stdout:[ OK ] LibRadosAio.RacingRemovePP (3009 ms) 2024-09-16T07:58:20.622 INFO:tasks.workunit.client.0.smithi027.stdout:[ RUN ] LibRadosAio.RoundTripCmpExtPP 2024-09-16T07:58:23.635 INFO:tasks.workunit.client.0.smithi027.stdout:[ OK ] LibRadosAio.RoundTripCmpExtPP (3013 ms) 2024-09-16T07:58:23.635 INFO:tasks.workunit.client.0.smithi027.stdout:[ RUN ] LibRadosAio.RoundTripCmpExtPP2 2024-09-16T07:58:26.651 INFO:tasks.workunit.client.0.smithi027.stdout:[ OK ] LibRadosAio.RoundTripCmpExtPP2 (3016 ms) 2024-09-16T07:58:26.651 INFO:tasks.workunit.client.0.smithi027.stdout:[ RUN ] LibRadosAio.PoolEIOFlag 2024-09-16T07:58:29.115 INFO:tasks.workunit.client.0.smithi027.stdout:setting pool EIO 2024-09-16T07:58:38.776 INFO:tasks.workunit.client.0.smithi027.stdout:waiting for 723 2024-09-16T07:58:39.776 INFO:tasks.workunit.client.0.smithi027.stdout:waiting for 663 2024-09-16T07:58:40.776 INFO:tasks.workunit.client.0.smithi027.stdout:waiting for 600 2024-09-16T07:58:41.776 INFO:tasks.workunit.client.0.smithi027.stdout:waiting for 537 2024-09-16T07:58:42.776 INFO:tasks.workunit.client.0.smithi027.stdout:waiting for 478 2024-09-16T07:58:43.777 INFO:tasks.workunit.client.0.smithi027.stdout:waiting for 392 2024-09-16T07:58:44.777 INFO:tasks.workunit.client.0.smithi027.stdout:waiting for 273 2024-09-16T07:58:45.777 INFO:tasks.workunit.client.0.smithi027.stdout:waiting for 155 2024-09-16T07:58:46.777 INFO:tasks.workunit.client.0.smithi027.stdout:waiting for 33 2024-09-16T07:58:47.777 INFO:tasks.workunit.client.0.smithi027.stdout:max_success 1000, min_failed 0 2024-09-16T07:58:47.777 INFO:tasks.workunit.client.0.smithi027.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.3.0-4879-g81fedcb3/rpm/el9/BUILD/ceph-19.3.0-4879-g81fedcb3/src/test/librados/aio_cxx.cc:2421: Failure 2024-09-16T07:58:47.777 INFO:tasks.workunit.client.0.smithi027.stdout:Value of: max_success + 1 == min_failed 2024-09-16T07:58:47.777 INFO:tasks.workunit.client.0.smithi027.stdout: Actual: false 2024-09-16T07:58:47.777 INFO:tasks.workunit.client.0.smithi027.stdout:Expected: true 2024-09-16T07:58:48.685 INFO:tasks.workunit.client.0.smithi027.stdout:[ FAILED ] LibRadosAio.PoolEIOFlag (22034 ms) 2024-09-16T07:58:48.685 INFO:tasks.workunit.client.0.smithi027.stdout:[ RUN ] LibRadosAio.MultiReads 2024-09-16T07:58:48.685 INFO:tasks.workunit.client.0.smithi027.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.3.0-4879-g81fedcb3/rpm/el9/BUILD/ceph-19.3.0-4879-g81fedcb3/src/test/librados/aio_cxx.cc:2427: Skipped 2024-09-16T07:58:48.685 INFO:tasks.workunit.client.0.smithi027.stdout:Not supported by crimson yet. Skipped 2024-09-16T07:58:48.685 INFO:tasks.workunit.client.0.smithi027.stdout:[ SKIPPED ] LibRadosAio.MultiReads (0 ms) 2024-09-16T07:58:48.685 INFO:tasks.workunit.client.0.smithi027.stdout:[----------] 32 tests from LibRadosAio (128968 ms total) 2024-09-16T07:58:48.686 INFO:tasks.workunit.client.0.smithi027.stdout: 2024-09-16T07:58:48.686 INFO:tasks.workunit.client.0.smithi027.stdout:[----------] 4 tests from LibRadosAioPP 2024-09-16T07:58:48.686 INFO:tasks.workunit.client.0.smithi027.stdout:[ RUN ] LibRadosAioPP.ReadIntoBufferlist 2024-09-16T07:58:51.696 INFO:tasks.workunit.client.0.smithi027.stdout:[ OK ] LibRadosAioPP.ReadIntoBufferlist (3011 ms) 2024-09-16T07:58:51.696 INFO:tasks.workunit.client.0.smithi027.stdout:[ RUN ] LibRadosAioPP.XattrsRoundTripPP 2024-09-16T07:59:00.626 INFO:tasks.workunit.client.0.smithi027.stdout:[ OK ] LibRadosAioPP.XattrsRoundTripPP (8930 ms) 2024-09-16T07:59:00.626 INFO:tasks.workunit.client.0.smithi027.stdout:[ RUN ] LibRadosAioPP.RmXattrPP 2024-09-16T07:59:15.711 INFO:tasks.workunit.client.0.smithi027.stdout:[ OK ] LibRadosAioPP.RmXattrPP (15084 ms) 2024-09-16T07:59:15.711 INFO:tasks.workunit.client.0.smithi027.stdout:[ RUN ] LibRadosAioPP.RemoveTestPP 2024-09-16T07:59:18.721 INFO:tasks.workunit.client.0.smithi027.stdout:[ OK ] LibRadosAioPP.RemoveTestPP (3011 ms) 2024-09-16T07:59:18.721 INFO:tasks.workunit.client.0.smithi027.stdout:[----------] 4 tests from LibRadosAioPP (30036 ms total) 2024-09-16T07:59:18.721 INFO:tasks.workunit.client.0.smithi027.stdout: 2024-09-16T07:59:18.721 INFO:tasks.workunit.client.0.smithi027.stdout:[----------] 1 test from LibRadosIoPP 2024-09-16T07:59:18.721 INFO:tasks.workunit.client.0.smithi027.stdout:[ RUN ] LibRadosIoPP.XattrListPP 2024-09-16T07:59:21.749 INFO:tasks.workunit.client.0.smithi027.stdout:[ OK ] LibRadosIoPP.XattrListPP (3024 ms) 2024-09-16T07:59:21.749 INFO:tasks.workunit.client.0.smithi027.stdout:[----------] 1 test from LibRadosIoPP (3024 ms total) 2024-09-16T07:59:21.749 INFO:tasks.workunit.client.0.smithi027.stdout: 2024-09-16T07:59:21.749 INFO:tasks.workunit.client.0.smithi027.stdout:[----------] 20 tests from LibRadosAioEC 2024-09-16T07:59:21.749 INFO:tasks.workunit.client.0.smithi027.stdout:[ RUN ] LibRadosAioEC.SimpleWritePP 2024-09-16T07:59:21.749 INFO:tasks.workunit.client.0.smithi027.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.3.0-4879-g81fedcb3/rpm/el9/BUILD/ceph-19.3.0-4879-g81fedcb3/src/test/librados/aio_cxx.cc:1384: Skipped 2024-09-16T07:59:21.749 INFO:tasks.workunit.client.0.smithi027.stdout:Not supported by crimson yet. Skipped 2024-09-16T07:59:21.749 INFO:tasks.workunit.client.0.smithi027.stdout:[ SKIPPED ] LibRadosAioEC.SimpleWritePP (0 ms) 2024-09-16T07:59:21.749 INFO:tasks.workunit.client.0.smithi027.stdout:[ RUN ] LibRadosAioEC.WaitForSafePP 2024-09-16T07:59:21.749 INFO:tasks.workunit.client.0.smithi027.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.3.0-4879-g81fedcb3/rpm/el9/BUILD/ceph-19.3.0-4879-g81fedcb3/src/test/librados/aio_cxx.cc:1419: Skipped 2024-09-16T07:59:21.750 INFO:tasks.workunit.client.0.smithi027.stdout:Not supported by crimson yet. Skipped 2024-09-16T07:59:21.750 INFO:tasks.workunit.client.0.smithi027.stdout:[ SKIPPED ] LibRadosAioEC.WaitForSafePP (0 ms) 2024-09-16T07:59:21.750 INFO:tasks.workunit.client.0.smithi027.stdout:[ RUN ] LibRadosAioEC.RoundTripPP 2024-09-16T07:59:21.750 INFO:tasks.workunit.client.0.smithi027.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.3.0-4879-g81fedcb3/rpm/el9/BUILD/ceph-19.3.0-4879-g81fedcb3/src/test/librados/aio_cxx.cc:1436: Skipped 2024-09-16T07:59:21.750 INFO:tasks.workunit.client.0.smithi027.stdout:Not supported by crimson yet. Skipped 2024-09-16T07:59:21.750 INFO:tasks.workunit.client.0.smithi027.stdout:[ SKIPPED ] LibRadosAioEC.RoundTripPP (0 ms) 2024-09-16T07:59:21.750 INFO:tasks.workunit.client.0.smithi027.stdout:[ RUN ] LibRadosAioEC.RoundTripPP2 2024-09-16T07:59:21.750 INFO:tasks.workunit.client.0.smithi027.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.3.0-4879-g81fedcb3/rpm/el9/BUILD/ceph-19.3.0-4879-g81fedcb3/src/test/librados/aio_cxx.cc:1467: Skipped 2024-09-16T07:59:21.750 INFO:tasks.workunit.client.0.smithi027.stdout:Not supported by crimson yet. Skipped 2024-09-16T07:59:21.750 INFO:tasks.workunit.client.0.smithi027.stdout:[ SKIPPED ] LibRadosAioEC.RoundTripPP2 (0 ms) 2024-09-16T07:59:21.750 INFO:tasks.workunit.client.0.smithi027.stdout:[ RUN ] LibRadosAioEC.RoundTripPP3 2024-09-16T07:59:21.750 INFO:tasks.workunit.client.0.smithi027.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.3.0-4879-g81fedcb3/rpm/el9/BUILD/ceph-19.3.0-4879-g81fedcb3/src/test/librados/aio_cxx.cc:1500: Skipped 2024-09-16T07:59:21.750 INFO:tasks.workunit.client.0.smithi027.stdout:Not supported by crimson yet. Skipped 2024-09-16T07:59:21.751 INFO:tasks.workunit.client.0.smithi027.stdout:[ SKIPPED ] LibRadosAioEC.RoundTripPP3 (0 ms) 2024-09-16T07:59:21.751 INFO:tasks.workunit.client.0.smithi027.stdout:[ RUN ] LibRadosAioEC.RoundTripSparseReadPP 2024-09-16T07:59:21.751 INFO:tasks.workunit.client.0.smithi027.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.3.0-4879-g81fedcb3/rpm/el9/BUILD/ceph-19.3.0-4879-g81fedcb3/src/test/librados/aio_cxx.cc:1605: Skipped 2024-09-16T07:59:21.751 INFO:tasks.workunit.client.0.smithi027.stdout:Not supported by crimson yet. Skipped 2024-09-16T07:59:21.751 INFO:tasks.workunit.client.0.smithi027.stdout:[ SKIPPED ] LibRadosAioEC.RoundTripSparseReadPP (0 ms) 2024-09-16T07:59:21.751 INFO:tasks.workunit.client.0.smithi027.stdout:[ RUN ] LibRadosAioEC.RoundTripAppendPP 2024-09-16T07:59:21.751 INFO:tasks.workunit.client.0.smithi027.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.3.0-4879-g81fedcb3/rpm/el9/BUILD/ceph-19.3.0-4879-g81fedcb3/src/test/librados/aio_cxx.cc:1637: Skipped 2024-09-16T07:59:21.751 INFO:tasks.workunit.client.0.smithi027.stdout:Not supported by crimson yet. Skipped 2024-09-16T07:59:21.751 INFO:tasks.workunit.client.0.smithi027.stdout:[ SKIPPED ] LibRadosAioEC.RoundTripAppendPP (0 ms) 2024-09-16T07:59:21.751 INFO:tasks.workunit.client.0.smithi027.stdout:[ RUN ] LibRadosAioEC.IsCompletePP 2024-09-16T07:59:21.751 INFO:tasks.workunit.client.0.smithi027.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.3.0-4879-g81fedcb3/rpm/el9/BUILD/ceph-19.3.0-4879-g81fedcb3/src/test/librados/aio_cxx.cc:1705: Skipped 2024-09-16T07:59:21.751 INFO:tasks.workunit.client.0.smithi027.stdout:Not supported by crimson yet. Skipped 2024-09-16T07:59:21.751 INFO:tasks.workunit.client.0.smithi027.stdout:[ SKIPPED ] LibRadosAioEC.IsCompletePP (0 ms) 2024-09-16T07:59:21.752 INFO:tasks.workunit.client.0.smithi027.stdout:[ RUN ] LibRadosAioEC.IsSafePP 2024-09-16T07:59:21.752 INFO:tasks.workunit.client.0.smithi027.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.3.0-4879-g81fedcb3/rpm/el9/BUILD/ceph-19.3.0-4879-g81fedcb3/src/test/librados/aio_cxx.cc:1742: Skipped 2024-09-16T07:59:21.752 INFO:tasks.workunit.client.0.smithi027.stdout:Not supported by crimson yet. Skipped 2024-09-16T07:59:21.752 INFO:tasks.workunit.client.0.smithi027.stdout:[ SKIPPED ] LibRadosAioEC.IsSafePP (0 ms) 2024-09-16T07:59:21.752 INFO:tasks.workunit.client.0.smithi027.stdout:[ RUN ] LibRadosAioEC.ReturnValuePP 2024-09-16T07:59:21.752 INFO:tasks.workunit.client.0.smithi027.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.3.0-4879-g81fedcb3/rpm/el9/BUILD/ceph-19.3.0-4879-g81fedcb3/src/test/librados/aio_cxx.cc:1780: Skipped 2024-09-16T07:59:21.752 INFO:tasks.workunit.client.0.smithi027.stdout:Not supported by crimson yet. Skipped 2024-09-16T07:59:21.752 INFO:tasks.workunit.client.0.smithi027.stdout:[ SKIPPED ] LibRadosAioEC.ReturnValuePP (0 ms) 2024-09-16T07:59:21.752 INFO:tasks.workunit.client.0.smithi027.stdout:[ RUN ] LibRadosAioEC.FlushPP 2024-09-16T07:59:21.752 INFO:tasks.workunit.client.0.smithi027.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.3.0-4879-g81fedcb3/rpm/el9/BUILD/ceph-19.3.0-4879-g81fedcb3/src/test/librados/aio_cxx.cc:1796: Skipped 2024-09-16T07:59:21.752 INFO:tasks.workunit.client.0.smithi027.stdout:Not supported by crimson yet. Skipped 2024-09-16T07:59:21.752 INFO:tasks.workunit.client.0.smithi027.stdout:[ SKIPPED ] LibRadosAioEC.FlushPP (0 ms) 2024-09-16T07:59:21.752 INFO:tasks.workunit.client.0.smithi027.stdout:[ RUN ] LibRadosAioEC.FlushAsyncPP 2024-09-16T07:59:21.753 INFO:tasks.workunit.client.0.smithi027.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.3.0-4879-g81fedcb3/rpm/el9/BUILD/ceph-19.3.0-4879-g81fedcb3/src/test/librados/aio_cxx.cc:1824: Skipped 2024-09-16T07:59:21.753 INFO:tasks.workunit.client.0.smithi027.stdout:Not supported by crimson yet. Skipped 2024-09-16T07:59:21.753 INFO:tasks.workunit.client.0.smithi027.stdout:[ SKIPPED ] LibRadosAioEC.FlushAsyncPP (0 ms) 2024-09-16T07:59:21.753 INFO:tasks.workunit.client.0.smithi027.stdout:[ RUN ] LibRadosAioEC.RoundTripWriteFullPP 2024-09-16T07:59:21.753 INFO:tasks.workunit.client.0.smithi027.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.3.0-4879-g81fedcb3/rpm/el9/BUILD/ceph-19.3.0-4879-g81fedcb3/src/test/librados/aio_cxx.cc:1859: Skipped 2024-09-16T07:59:21.753 INFO:tasks.workunit.client.0.smithi027.stdout:Not supported by crimson yet. Skipped 2024-09-16T07:59:21.753 INFO:tasks.workunit.client.0.smithi027.stdout:[ SKIPPED ] LibRadosAioEC.RoundTripWriteFullPP (0 ms) 2024-09-16T07:59:21.753 INFO:tasks.workunit.client.0.smithi027.stdout:[ RUN ] LibRadosAioEC.RoundTripWriteFullPP2 2024-09-16T07:59:21.753 INFO:tasks.workunit.client.0.smithi027.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.3.0-4879-g81fedcb3/rpm/el9/BUILD/ceph-19.3.0-4879-g81fedcb3/src/test/librados/aio_cxx.cc:1904: Skipped 2024-09-16T07:59:21.753 INFO:tasks.workunit.client.0.smithi027.stdout:Not supported by crimson yet. Skipped 2024-09-16T07:59:21.753 INFO:tasks.workunit.client.0.smithi027.stdout:[ SKIPPED ] LibRadosAioEC.RoundTripWriteFullPP2 (0 ms) 2024-09-16T07:59:21.753 INFO:tasks.workunit.client.0.smithi027.stdout:[ RUN ] LibRadosAioEC.SimpleStatPP 2024-09-16T07:59:21.753 INFO:tasks.workunit.client.0.smithi027.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.3.0-4879-g81fedcb3/rpm/el9/BUILD/ceph-19.3.0-4879-g81fedcb3/src/test/librados/aio_cxx.cc:1946: Skipped 2024-09-16T07:59:21.754 INFO:tasks.workunit.client.0.smithi027.stdout:Not supported by crimson yet. Skipped 2024-09-16T07:59:21.754 INFO:tasks.workunit.client.0.smithi027.stdout:[ SKIPPED ] LibRadosAioEC.SimpleStatPP (0 ms) 2024-09-16T07:59:21.754 INFO:tasks.workunit.client.0.smithi027.stdout:[ RUN ] LibRadosAioEC.SimpleStatPPNS 2024-09-16T07:59:21.754 INFO:tasks.workunit.client.0.smithi027.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.3.0-4879-g81fedcb3/rpm/el9/BUILD/ceph-19.3.0-4879-g81fedcb3/src/test/librados/aio_cxx.cc:1977: Skipped 2024-09-16T07:59:21.754 INFO:tasks.workunit.client.0.smithi027.stdout:Not supported by crimson yet. Skipped 2024-09-16T07:59:21.754 INFO:tasks.workunit.client.0.smithi027.stdout:[ SKIPPED ] LibRadosAioEC.SimpleStatPPNS (0 ms) 2024-09-16T07:59:21.754 INFO:tasks.workunit.client.0.smithi027.stdout:[ RUN ] LibRadosAioEC.StatRemovePP 2024-09-16T07:59:21.754 INFO:tasks.workunit.client.0.smithi027.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.3.0-4879-g81fedcb3/rpm/el9/BUILD/ceph-19.3.0-4879-g81fedcb3/src/test/librados/aio_cxx.cc:2008: Skipped 2024-09-16T07:59:21.754 INFO:tasks.workunit.client.0.smithi027.stdout:Not supported by crimson yet. Skipped 2024-09-16T07:59:21.754 INFO:tasks.workunit.client.0.smithi027.stdout:[ SKIPPED ] LibRadosAioEC.StatRemovePP (0 ms) 2024-09-16T07:59:21.754 INFO:tasks.workunit.client.0.smithi027.stdout:[ RUN ] LibRadosAioEC.ExecuteClassPP 2024-09-16T07:59:21.754 INFO:tasks.workunit.client.0.smithi027.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.3.0-4879-g81fedcb3/rpm/el9/BUILD/ceph-19.3.0-4879-g81fedcb3/src/test/librados/aio_cxx.cc:2059: Skipped 2024-09-16T07:59:21.754 INFO:tasks.workunit.client.0.smithi027.stdout:Not supported by crimson yet. Skipped 2024-09-16T07:59:21.754 INFO:tasks.workunit.client.0.smithi027.stdout:[ SKIPPED ] LibRadosAioEC.ExecuteClassPP (0 ms) 2024-09-16T07:59:21.754 INFO:tasks.workunit.client.0.smithi027.stdout:[ RUN ] LibRadosAioEC.OmapPP 2024-09-16T07:59:21.755 INFO:tasks.workunit.client.0.smithi027.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.3.0-4879-g81fedcb3/rpm/el9/BUILD/ceph-19.3.0-4879-g81fedcb3/src/test/librados/aio_cxx.cc:2089: Skipped 2024-09-16T07:59:21.755 INFO:tasks.workunit.client.0.smithi027.stdout:Not supported by crimson yet. Skipped 2024-09-16T07:59:21.755 INFO:tasks.workunit.client.0.smithi027.stdout:[ SKIPPED ] LibRadosAioEC.OmapPP (0 ms) 2024-09-16T07:59:21.755 INFO:tasks.workunit.client.0.smithi027.stdout:[ RUN ] LibRadosAioEC.MultiWritePP 2024-09-16T07:59:21.755 INFO:tasks.workunit.client.0.smithi027.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.3.0-4879-g81fedcb3/rpm/el9/BUILD/ceph-19.3.0-4879-g81fedcb3/src/test/librados/aio_cxx.cc:2124: Skipped 2024-09-16T07:59:21.755 INFO:tasks.workunit.client.0.smithi027.stdout:Not supported by crimson yet. Skipped 2024-09-16T07:59:21.755 INFO:tasks.workunit.client.0.smithi027.stdout:[ SKIPPED ] LibRadosAioEC.MultiWritePP (0 ms) 2024-09-16T07:59:21.755 INFO:tasks.workunit.client.0.smithi027.stdout:[----------] 20 tests from LibRadosAioEC (0 ms total) 2024-09-16T07:59:21.755 INFO:tasks.workunit.client.0.smithi027.stdout: 2024-09-16T07:59:21.755 INFO:tasks.workunit.client.0.smithi027.stdout:[----------] Global test environment tear-down 2024-09-16T07:59:21.755 INFO:tasks.workunit.client.0.smithi027.stdout:[==========] 57 tests from 4 test suites ran. (162029 ms total) 2024-09-16T07:59:21.756 INFO:tasks.workunit.client.0.smithi027.stdout:[ PASSED ] 35 tests. 2024-09-16T07:59:21.756 INFO:tasks.workunit.client.0.smithi027.stdout:[ SKIPPED ] 21 tests, listed below: 2024-09-16T07:59:21.756 INFO:tasks.workunit.client.0.smithi027.stdout:[ SKIPPED ] LibRadosAio.MultiReads 2024-09-16T07:59:21.756 INFO:tasks.workunit.client.0.smithi027.stdout:[ SKIPPED ] LibRadosAioEC.SimpleWritePP 2024-09-16T07:59:21.756 INFO:tasks.workunit.client.0.smithi027.stdout:[ SKIPPED ] LibRadosAioEC.WaitForSafePP 2024-09-16T07:59:21.756 INFO:tasks.workunit.client.0.smithi027.stdout:[ SKIPPED ] LibRadosAioEC.RoundTripPP 2024-09-16T07:59:21.756 INFO:tasks.workunit.client.0.smithi027.stdout:[ SKIPPED ] LibRadosAioEC.RoundTripPP2 2024-09-16T07:59:21.756 INFO:tasks.workunit.client.0.smithi027.stdout:[ SKIPPED ] LibRadosAioEC.RoundTripPP3 2024-09-16T07:59:21.756 INFO:tasks.workunit.client.0.smithi027.stdout:[ SKIPPED ] LibRadosAioEC.RoundTripSparseReadPP 2024-09-16T07:59:21.756 INFO:tasks.workunit.client.0.smithi027.stdout:[ SKIPPED ] LibRadosAioEC.RoundTripAppendPP 2024-09-16T07:59:21.756 INFO:tasks.workunit.client.0.smithi027.stdout:[ SKIPPED ] LibRadosAioEC.IsCompletePP 2024-09-16T07:59:21.756 INFO:tasks.workunit.client.0.smithi027.stdout:[ SKIPPED ] LibRadosAioEC.IsSafePP 2024-09-16T07:59:21.756 INFO:tasks.workunit.client.0.smithi027.stdout:[ SKIPPED ] LibRadosAioEC.ReturnValuePP 2024-09-16T07:59:21.756 INFO:tasks.workunit.client.0.smithi027.stdout:[ SKIPPED ] LibRadosAioEC.FlushPP 2024-09-16T07:59:21.756 INFO:tasks.workunit.client.0.smithi027.stdout:[ SKIPPED ] LibRadosAioEC.FlushAsyncPP 2024-09-16T07:59:21.757 INFO:tasks.workunit.client.0.smithi027.stdout:[ SKIPPED ] LibRadosAioEC.RoundTripWriteFullPP 2024-09-16T07:59:21.757 INFO:tasks.workunit.client.0.smithi027.stdout:[ SKIPPED ] LibRadosAioEC.RoundTripWriteFullPP2 2024-09-16T07:59:21.757 INFO:tasks.workunit.client.0.smithi027.stdout:[ SKIPPED ] LibRadosAioEC.SimpleStatPP 2024-09-16T07:59:21.757 INFO:tasks.workunit.client.0.smithi027.stdout:[ SKIPPED ] LibRadosAioEC.SimpleStatPPNS 2024-09-16T07:59:21.757 INFO:tasks.workunit.client.0.smithi027.stdout:[ SKIPPED ] LibRadosAioEC.StatRemovePP 2024-09-16T07:59:21.757 INFO:tasks.workunit.client.0.smithi027.stdout:[ SKIPPED ] LibRadosAioEC.ExecuteClassPP 2024-09-16T07:59:21.757 INFO:tasks.workunit.client.0.smithi027.stdout:[ SKIPPED ] LibRadosAioEC.OmapPP 2024-09-16T07:59:21.757 INFO:tasks.workunit.client.0.smithi027.stdout:[ SKIPPED ] LibRadosAioEC.MultiWritePP 2024-09-16T07:59:21.757 INFO:tasks.workunit.client.0.smithi027.stdout:[ FAILED ] 1 test, listed below: 2024-09-16T07:59:21.757 INFO:tasks.workunit.client.0.smithi027.stdout:[ FAILED ] LibRadosAio.PoolEIOFlag 2024-09-16T07:59:21.757 INFO:tasks.workunit.client.0.smithi027.stdout: 2024-09-16T07:59:21.757 INFO:tasks.workunit.client.0.smithi027.stdout: 1 FAILED TEST 2024-09-16T07:59:21.758 INFO:tasks.workunit.client.0.smithi027.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.3.0-4879-g81fedcb3/rpm/el9/BUILD/ceph-19.3.0-4879-g81fedcb3/src/common/mutex_debug.h: In function 'ceph::mutex_debug_detail::mutex_debug_impl< >::~mutex_debug_impl() [with bool Recursive = false]' thread 7fe318c5d800 time 2024-09-16T07:59:21.748830+0000 2024-09-16T07:59:21.758 INFO:tasks.workunit.client.0.smithi027.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.3.0-4879-g81fedcb3/rpm/el9/BUILD/ceph-19.3.0-4879-g81fedcb3/src/common/mutex_debug.h: 110: FAILED ceph_assert(r == 0) 2024-09-16T07:59:21.758 INFO:tasks.workunit.client.0.smithi027.stderr: ceph version 19.3.0-4879-g81fedcb3 (81fedcb3b484c843d22ffbc8d179cb567b7083b5) squid (dev) 2024-09-16T07:59:21.758 INFO:tasks.workunit.client.0.smithi027.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x129) [0x7fe317e7e5ad] 2024-09-16T07:59:21.758 INFO:tasks.workunit.client.0.smithi027.stderr: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fe317e7e7f3] 2024-09-16T07:59:21.758 INFO:tasks.workunit.client.0.smithi027.stderr: 3: ceph_test_rados_api_aio_pp() [0x4a0f69] 2024-09-16T07:59:21.758 INFO:tasks.workunit.client.0.smithi027.stderr: 4: /lib64/libc.so.6(+0x40ded) [0x7fe316c40ded] 2024-09-16T07:59:21.758 INFO:tasks.workunit.client.0.smithi027.stderr: 5: on_exit() 2024-09-16T07:59:21.758 INFO:tasks.workunit.client.0.smithi027.stderr: 6: /lib64/libc.so.6(+0x29597) [0x7fe316c29597] 2024-09-16T07:59:21.758 INFO:tasks.workunit.client.0.smithi027.stderr: 7: __libc_start_main() 2024-09-16T07:59:21.758 INFO:tasks.workunit.client.0.smithi027.stderr: 8: _start() 2024-09-16T07:59:22.256 INFO:tasks.workunit.client.0.smithi027.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh: line 24: 29926 Aborted (core dumped) ceph_test_rados_$f 2024-09-16T07:59:22.256 INFO:tasks.workunit.client.0.smithi027.stderr:++ cleanup 2024-09-16T07:59:22.256 INFO:tasks.workunit.client.0.smithi027.stderr:++ pkill -P 29455 2024-09-16T07:59:22.291 INFO:tasks.workunit.client.0.smithi027.stderr:++ true 2024-09-16T07:59:22.291 INFO:tasks.workunit.client.0.smithi027.stderr:+ cleanup 2024-09-16T07:59:22.292 INFO:tasks.workunit.client.0.smithi027.stderr:+ pkill -P 29455 2024-09-16T07:59:22.295 DEBUG:teuthology.orchestra.run:got remote process result: 134 2024-09-16T07:59:22.296 INFO:tasks.workunit.client.0.smithi027.stderr:+ true 2024-09-16T07:59:22.296 INFO:tasks.workunit:Stopping ['rados/test.sh --crimson', 'rados/test_pool_quota.sh'] on client.0... 2024-09-16T07:59:22.296 DEBUG:teuthology.orchestra.run.smithi027:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-09-16T07:59:22.633 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_0325fd3c11a225c0fd9eb38fab88aad1370ad779/qa/tasks/workunit.py", line 126, in task with parallel() as p: File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_0325fd3c11a225c0fd9eb38fab88aad1370ad779/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test rados/test.sh) on smithi027 with status 134: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=0325fd3c11a225c0fd9eb38fab88aad1370ad779 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 CRIMSON_COMPAT=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh --crimson' 2024-09-16T07:59:22.874 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=cd4f55a4386a422d8fb332ae19e0f572 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_0325fd3c11a225c0fd9eb38fab88aad1370ad779/qa/tasks/workunit.py", line 126, in task with parallel() as p: File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_0325fd3c11a225c0fd9eb38fab88aad1370ad779/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test rados/test.sh) on smithi027 with status 134: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=0325fd3c11a225c0fd9eb38fab88aad1370ad779 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 CRIMSON_COMPAT=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh --crimson' 2024-09-16T07:59:22.877 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-09-16T07:59:22.907 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-16T07:59:22.907 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T07:59:23.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:23.024+0000 7f1a01d48640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T07:59:23.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:23.029+0000 7f1a01d48640 1 Processor -- start 2024-09-16T07:59:23.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:23.029+0000 7f1a01d48640 1 -- start start 2024-09-16T07:59:23.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:23.031+0000 7f1a01d48640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f19fc132520 0x7f19fc132930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:59:23.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:23.031+0000 7f1a01d48640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f19fc06f6d0 con 0x7f19fc0a3450 2024-09-16T07:59:23.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:23.031+0000 7f1a01d48640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 -- 0x7f19fc06f3d0 con 0x7f19fc132520 2024-09-16T07:59:23.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:23.031+0000 7f19fb7fe640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f19fc132520 0x7f19fc132930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:59:23.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:23.031+0000 7f19fb7fe640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f19fc132520 0x7f19fc132930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40346/0 (socket says 172.21.15.27:40346) 2024-09-16T07:59:23.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:23.031+0000 7f19faffd640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f19fc0a3450 0x7f19fc0a3860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:36072/0 (socket says 172.21.15.27:36072) 2024-09-16T07:59:23.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:23.031+0000 7f19fb7fe640 1 -- 172.21.15.27:0/3393606711 learned_addr learned my addr 172.21.15.27:0/3393606711 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-09-16T07:59:23.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:23.031+0000 7f19fb7fe640 1 -- 172.21.15.27:0/3393606711 >> v1:172.21.15.27:6789/0 conn(0x7f19fc0a3450 legacy=0x7f19fc0a3860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:59:23.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:23.031+0000 7f19fa7fc640 1 -- 172.21.15.27:0/3393606711 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1078725351 0 0) 0x7f19fc06f6d0 con 0x7f19fc0a3450 2024-09-16T07:59:23.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:23.031+0000 7f19fb7fe640 1 -- 172.21.15.27:0/3393606711 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19fc06f0c0 con 0x7f19fc132520 2024-09-16T07:59:23.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:23.031+0000 7f19fb7fe640 1 --2- 172.21.15.27:0/3393606711 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19fc132520 0x7f19fc132930 crc :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83214b85f1477723 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:59:23.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:23.031+0000 7f19fa7fc640 1 -- 172.21.15.27:0/3393606711 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f19ec009e50 con 0x7f19fc132520 2024-09-16T07:59:23.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:23.032+0000 7f19fa7fc640 1 -- 172.21.15.27:0/3393606711 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f19ec00e230 con 0x7f19fc132520 2024-09-16T07:59:23.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:23.032+0000 7f19fa7fc640 1 -- 172.21.15.27:0/3393606711 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f19ec00e4d0 con 0x7f19fc132520 2024-09-16T07:59:23.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:23.032+0000 7f1a01d48640 1 -- 172.21.15.27:0/3393606711 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19fc132520 msgr2=0x7f19fc132930 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:59:23.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:23.032+0000 7f1a01d48640 1 --2- 172.21.15.27:0/3393606711 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19fc132520 0x7f19fc132930 crc :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:59:23.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:23.032+0000 7f1a01d48640 1 -- 172.21.15.27:0/3393606711 shutdown_connections 2024-09-16T07:59:23.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:23.032+0000 7f1a01d48640 1 --2- 172.21.15.27:0/3393606711 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19fc132520 0x7f19fc132930 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:59:23.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:23.032+0000 7f1a01d48640 1 -- 172.21.15.27:0/3393606711 >> 172.21.15.27:0/3393606711 conn(0x7f19fc06e490 msgr2=0x7f19fc06e8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:59:23.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:23.032+0000 7f1a01d48640 1 -- 172.21.15.27:0/3393606711 shutdown_connections 2024-09-16T07:59:23.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:23.032+0000 7f1a01d48640 1 -- 172.21.15.27:0/3393606711 wait complete. 2024-09-16T07:59:23.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:23.033+0000 7f1a01d48640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T07:59:23.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:23.034+0000 7f1a01d48640 1 Processor -- start 2024-09-16T07:59:23.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:23.035+0000 7f1a01d48640 1 -- start start 2024-09-16T07:59:23.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:23.036+0000 7f1a01d48640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19fc0a3450 0x7f19fc11e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:59:23.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:23.036+0000 7f1a01d48640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f19fc121ca0 con 0x7f19fc0a3450 2024-09-16T07:59:23.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:23.036+0000 7f19fb7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19fc0a3450 0x7f19fc11e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:59:23.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:23.036+0000 7f19fb7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19fc0a3450 0x7f19fc11e090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40350/0 (socket says 172.21.15.27:40350) 2024-09-16T07:59:23.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:23.036+0000 7f19fb7fe640 1 -- 172.21.15.27:0/2150846703 learned_addr learned my addr 172.21.15.27:0/2150846703 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-09-16T07:59:23.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:23.037+0000 7f19fb7fe640 1 -- 172.21.15.27:0/2150846703 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19fc121f40 con 0x7f19fc0a3450 2024-09-16T07:59:23.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:23.037+0000 7f19fb7fe640 1 --2- 172.21.15.27:0/2150846703 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19fc0a3450 0x7f19fc11e090 crc :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:59:23.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:23.037+0000 7f19e3fff640 1 -- 172.21.15.27:0/2150846703 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f19ec00edc0 con 0x7f19fc0a3450 2024-09-16T07:59:23.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:23.037+0000 7f19e3fff640 1 -- 172.21.15.27:0/2150846703 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f19ec015420 con 0x7f19fc0a3450 2024-09-16T07:59:23.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:23.037+0000 7f1a01d48640 1 -- 172.21.15.27:0/2150846703 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f19fc1214a0 con 0x7f19fc0a3450 2024-09-16T07:59:23.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:23.037+0000 7f19e3fff640 1 -- 172.21.15.27:0/2150846703 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f19ec0156c0 con 0x7f19fc0a3450 2024-09-16T07:59:23.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:23.037+0000 7f1a01d48640 1 -- 172.21.15.27:0/2150846703 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19fc09ce40 con 0x7f19fc0a3450 2024-09-16T07:59:23.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:23.038+0000 7f19e3fff640 1 -- 172.21.15.27:0/2150846703 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) ==== 50954+0+0 (crc 0 0 0) 0x7f19ec015870 con 0x7f19fc0a3450 2024-09-16T07:59:23.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:23.038+0000 7f1a01d48640 1 -- 172.21.15.27:0/2150846703 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f19fc06a300 con 0x7f19fc0a3450 2024-09-16T07:59:23.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:23.038+0000 7f19e3fff640 1 --2- 172.21.15.27:0/2150846703 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7f19e803e930 0x7f19e8040e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:59:23.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:23.038+0000 7f19e3fff640 1 -- 172.21.15.27:0/2150846703 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(228..228 src has 1..228) ==== 2700+0+0 (crc 0 0 0) 0x7f19ec050d40 con 0x7f19fc0a3450 2024-09-16T07:59:23.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:23.039+0000 7f19faffd640 1 --2- 172.21.15.27:0/2150846703 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7f19e803e930 0x7f19e8040e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:59:23.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:23.039+0000 7f19faffd640 1 --2- 172.21.15.27:0/2150846703 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7f19e803e930 0x7f19e8040e40 crc :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:59:23.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:23.044+0000 7f19e3fff640 1 -- 172.21.15.27:0/2150846703 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (crc 0 0 0) 0x7f19ec014c60 con 0x7f19fc0a3450 2024-09-16T07:59:23.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:23.224+0000 7f1a01d48640 1 -- 172.21.15.27:0/2150846703 --> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f19fc09c830 con 0x7f19e803e930 2024-09-16T07:59:23.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:23.225+0000 7f19e3fff640 1 -- 172.21.15.27:0/2150846703 <== mgr.4102 v2:172.21.15.187:6800/4038049593 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+31238 (crc 0 0 0) 0x7f19f00070b0 con 0x7f19e803e930 2024-09-16T07:59:23.224 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-16T07:59:23.224 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-09-16T07:59:23.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:23.228+0000 7f1a01d48640 1 -- 172.21.15.27:0/2150846703 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7f19e803e930 msgr2=0x7f19e8040e40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:59:23.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:23.228+0000 7f1a01d48640 1 --2- 172.21.15.27:0/2150846703 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7f19e803e930 0x7f19e8040e40 crc :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:59:23.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:23.228+0000 7f1a01d48640 1 -- 172.21.15.27:0/2150846703 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19fc0a3450 msgr2=0x7f19fc11e090 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:59:23.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:23.228+0000 7f1a01d48640 1 --2- 172.21.15.27:0/2150846703 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19fc0a3450 0x7f19fc11e090 crc :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:59:23.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:23.228+0000 7f1a01d48640 1 -- 172.21.15.27:0/2150846703 shutdown_connections 2024-09-16T07:59:23.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:23.228+0000 7f1a01d48640 1 --2- 172.21.15.27:0/2150846703 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7f19e803e930 0x7f19e8040e40 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:59:23.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:23.228+0000 7f1a01d48640 1 --2- 172.21.15.27:0/2150846703 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19fc0a3450 0x7f19fc11e090 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:59:23.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:23.228+0000 7f1a01d48640 1 -- 172.21.15.27:0/2150846703 >> 172.21.15.27:0/2150846703 conn(0x7f19fc06e490 msgr2=0x7f19fc1342d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:59:23.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:23.228+0000 7f1a01d48640 1 -- 172.21.15.27:0/2150846703 shutdown_connections 2024-09-16T07:59:23.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:23.228+0000 7f1a01d48640 1 -- 172.21.15.27:0/2150846703 wait complete. 2024-09-16T07:59:23.238 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":372,"stamp":"2024-09-16T07:59:23.147926+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":209190912,"kb_used":15568,"kb_used_data":15568,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":209175344,"statfs":{"total":214211493888,"available":214195552256,"internally_reserved":0,"allocated":15941632,"data_stored":15941632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.428583"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":645,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-09-16T07:59:18.795976+0000","last_change":"2024-09-16T07:54:26.740923+0000","last_active":"2024-09-16T07:59:18.795976+0000","last_peered":"2024-09-16T07:59:18.795976+0000","last_clean":"2024-09-16T07:59:18.795976+0000","last_became_active":"2024-09-16T07:54:24.784360+0000","last_became_peered":"2024-09-16T07:54:24.784360+0000","last_unstale":"2024-09-16T07:59:18.795976+0000","last_undegraded":"2024-09-16T07:59:18.795976+0000","last_fullsized":"2024-09-16T07:59:18.795976+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T07:54:23.687651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T07:54:23.687651+0000","last_clean_scrub_stamp":"2024-09-16T07:54:23.687651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":645,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-09-16T07:59:18.761001+0000","last_change":"2024-09-16T07:54:26.739464+0000","last_active":"2024-09-16T07:59:18.761001+0000","last_peered":"2024-09-16T07:59:18.761001+0000","last_clean":"2024-09-16T07:59:18.761001+0000","last_became_active":"2024-09-16T07:54:24.805913+0000","last_became_peered":"2024-09-16T07:54:24.805913+0000","last_unstale":"2024-09-16T07:59:18.761001+0000","last_undegraded":"2024-09-16T07:59:18.761001+0000","last_fullsized":"2024-09-16T07:59:18.761001+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T07:54:23.687651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T07:54:23.687651+0000","last_clean_scrub_stamp":"2024-09-16T07:54:23.687651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":645,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-09-16T07:59:18.782573+0000","last_change":"2024-09-16T07:54:26.735332+0000","last_active":"2024-09-16T07:59:18.782573+0000","last_peered":"2024-09-16T07:59:18.782573+0000","last_clean":"2024-09-16T07:59:18.782573+0000","last_became_active":"2024-09-16T07:54:24.773365+0000","last_became_peered":"2024-09-16T07:54:24.773365+0000","last_unstale":"2024-09-16T07:59:18.782573+0000","last_undegraded":"2024-09-16T07:59:18.782573+0000","last_fullsized":"2024-09-16T07:59:18.782573+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T07:54:23.687651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T07:54:23.687651+0000","last_clean_scrub_stamp":"2024-09-16T07:54:23.687651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":645,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-09-16T07:59:18.751250+0000","last_change":"2024-09-16T07:54:26.734682+0000","last_active":"2024-09-16T07:59:18.751250+0000","last_peered":"2024-09-16T07:59:18.751250+0000","last_clean":"2024-09-16T07:59:18.751250+0000","last_became_active":"2024-09-16T07:54:24.756974+0000","last_became_peered":"2024-09-16T07:54:24.756974+0000","last_unstale":"2024-09-16T07:59:18.751250+0000","last_undegraded":"2024-09-16T07:59:18.751250+0000","last_fullsized":"2024-09-16T07:59:18.751250+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T07:54:23.687651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T07:54:23.687651+0000","last_clean_scrub_stamp":"2024-09-16T07:54:23.687651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"15'2","reported_seq":647,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-09-16T07:59:18.780803+0000","last_change":"2024-09-16T07:54:26.741927+0000","last_active":"2024-09-16T07:59:18.780803+0000","last_peered":"2024-09-16T07:59:18.780803+0000","last_clean":"2024-09-16T07:59:18.780803+0000","last_became_active":"2024-09-16T07:54:24.794754+0000","last_became_peered":"2024-09-16T07:54:24.794754+0000","last_unstale":"2024-09-16T07:59:18.780803+0000","last_undegraded":"2024-09-16T07:59:18.780803+0000","last_fullsized":"2024-09-16T07:59:18.780803+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T07:54:23.687651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T07:54:23.687651+0000","last_clean_scrub_stamp":"2024-09-16T07:54:23.687651+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":645,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-09-16T07:59:18.758597+0000","last_change":"2024-09-16T07:54:26.734478+0000","last_active":"2024-09-16T07:59:18.758597+0000","last_peered":"2024-09-16T07:59:18.758597+0000","last_clean":"2024-09-16T07:59:18.758597+0000","last_became_active":"2024-09-16T07:54:24.774923+0000","last_became_peered":"2024-09-16T07:54:24.774923+0000","last_unstale":"2024-09-16T07:59:18.758597+0000","last_undegraded":"2024-09-16T07:59:18.758597+0000","last_fullsized":"2024-09-16T07:59:18.758597+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T07:54:23.687651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T07:54:23.687651+0000","last_clean_scrub_stamp":"2024-09-16T07:54:23.687651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":645,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-09-16T07:59:18.810530+0000","last_change":"2024-09-16T07:54:26.742127+0000","last_active":"2024-09-16T07:59:18.810530+0000","last_peered":"2024-09-16T07:59:18.810530+0000","last_clean":"2024-09-16T07:59:18.810530+0000","last_became_active":"2024-09-16T07:54:24.773472+0000","last_became_peered":"2024-09-16T07:54:24.773472+0000","last_unstale":"2024-09-16T07:59:18.810530+0000","last_undegraded":"2024-09-16T07:59:18.810530+0000","last_fullsized":"2024-09-16T07:59:18.810530+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T07:54:23.687651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T07:54:23.687651+0000","last_clean_scrub_stamp":"2024-09-16T07:54:23.687651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"13'1","reported_seq":646,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-09-16T07:59:18.805158+0000","last_change":"2024-09-16T07:54:26.739091+0000","last_active":"2024-09-16T07:59:18.805158+0000","last_peered":"2024-09-16T07:59:18.805158+0000","last_clean":"2024-09-16T07:59:18.805158+0000","last_became_active":"2024-09-16T07:54:24.801907+0000","last_became_peered":"2024-09-16T07:54:24.801907+0000","last_unstale":"2024-09-16T07:59:18.805158+0000","last_undegraded":"2024-09-16T07:59:18.805158+0000","last_fullsized":"2024-09-16T07:59:18.805158+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T07:54:23.687651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T07:54:23.687651+0000","last_clean_scrub_stamp":"2024-09-16T07:54:23.687651+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'52","reported_seq":717,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-09-16T07:59:18.755371+0000","last_change":"2024-09-16T07:54:22.755090+0000","last_active":"2024-09-16T07:59:18.755371+0000","last_peered":"2024-09-16T07:59:18.755371+0000","last_clean":"2024-09-16T07:59:18.755371+0000","last_became_active":"2024-09-16T07:54:22.736266+0000","last_became_peered":"2024-09-16T07:54:22.736266+0000","last_unstale":"2024-09-16T07:59:18.755371+0000","last_undegraded":"2024-09-16T07:59:18.755371+0000","last_fullsized":"2024-09-16T07:59:18.755371+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T07:54:21.678374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T07:54:21.678374+0000","last_clean_scrub_stamp":"2024-09-16T07:54:21.678374+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":214211493888,"available":214198435840,"internally_reserved":0,"allocated":13058048,"data_stored":13058048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53552873472,"available":53549346816,"internally_reserved":0,"allocated":3526656,"data_stored":3526656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705963,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":4340,"kb_used_data":4340,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52293388,"statfs":{"total":53552873472,"available":53548429312,"internally_reserved":0,"allocated":4444160,"data_stored":4444160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705963,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":3300,"kb_used_data":3300,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52294428,"statfs":{"total":53552873472,"available":53549494272,"internally_reserved":0,"allocated":3379200,"data_stored":3379200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705963,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":3376,"kb_used_data":3376,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52294352,"statfs":{"total":53552873472,"available":53549416448,"internally_reserved":0,"allocated":3457024,"data_stored":3457024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705963,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":4552,"kb_used_data":4552,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52293176,"statfs":{"total":53552873472,"available":53548212224,"internally_reserved":0,"allocated":4661248,"data_stored":4661248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":3,"total":53552873472,"available":53549346816,"internally_reserved":0,"allocated":3526656,"data_stored":3526656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53552873472,"available":53549064192,"internally_reserved":0,"allocated":3809280,"data_stored":3809280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53552873472,"available":53550006272,"internally_reserved":0,"allocated":2867200,"data_stored":2867200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53552873472,"available":53550018560,"internally_reserved":0,"allocated":2854912,"data_stored":2854912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":53552873472,"available":53549346816,"internally_reserved":0,"allocated":3526656,"data_stored":3526656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T07:59:23.238 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T07:59:23.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:23.350+0000 7ff5ad191640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T07:59:23.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:23.355+0000 7ff5ad191640 1 Processor -- start 2024-09-16T07:59:23.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:23.355+0000 7ff5ad191640 1 -- start start 2024-09-16T07:59:23.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:23.356+0000 7ff5ad191640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ff5a8132520 0x7ff5a8132930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:59:23.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:23.356+0000 7ff5ad191640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff5a806f6d0 con 0x7ff5a80a3450 2024-09-16T07:59:23.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:23.356+0000 7ff5ad191640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 -- 0x7ff5a806f3d0 con 0x7ff5a8132520 2024-09-16T07:59:23.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:23.357+0000 7ff5a6d9d640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ff5a8132520 0x7ff5a8132930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:59:23.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:23.357+0000 7ff5a659c640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7ff5a80a3450 0x7ff5a80a3860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:36080/0 (socket says 172.21.15.27:36080) 2024-09-16T07:59:23.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:23.357+0000 7ff5a6d9d640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ff5a8132520 0x7ff5a8132930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40360/0 (socket says 172.21.15.27:40360) 2024-09-16T07:59:23.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:23.357+0000 7ff5a659c640 1 -- 172.21.15.27:0/3131080517 learned_addr learned my addr 172.21.15.27:0/3131080517 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-09-16T07:59:23.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:23.357+0000 7ff5a6d9d640 1 -- 172.21.15.27:0/3131080517 >> v1:172.21.15.27:6789/0 conn(0x7ff5a80a3450 legacy=0x7ff5a80a3860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:59:23.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:23.357+0000 7ff5a6d9d640 1 -- 172.21.15.27:0/3131080517 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5a806f0c0 con 0x7ff5a8132520 2024-09-16T07:59:23.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:23.357+0000 7ff5a6d9d640 1 --2- 172.21.15.27:0/3131080517 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff5a8132520 0x7ff5a8132930 crc :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18c1b211461cc89b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:59:23.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:23.357+0000 7ff5a5d9b640 1 -- 172.21.15.27:0/3131080517 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7ff598009d90 con 0x7ff5a8132520 2024-09-16T07:59:23.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:23.357+0000 7ff5a5d9b640 1 -- 172.21.15.27:0/3131080517 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7ff59800e230 con 0x7ff5a8132520 2024-09-16T07:59:23.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:23.357+0000 7ff5a5d9b640 1 -- 172.21.15.27:0/3131080517 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7ff59800e4d0 con 0x7ff5a8132520 2024-09-16T07:59:23.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:23.358+0000 7ff5ad191640 1 -- 172.21.15.27:0/3131080517 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff5a8132520 msgr2=0x7ff5a8132930 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:59:23.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:23.358+0000 7ff5ad191640 1 --2- 172.21.15.27:0/3131080517 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff5a8132520 0x7ff5a8132930 crc :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:59:23.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:23.358+0000 7ff5ad191640 1 -- 172.21.15.27:0/3131080517 shutdown_connections 2024-09-16T07:59:23.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:23.358+0000 7ff5ad191640 1 --2- 172.21.15.27:0/3131080517 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff5a8132520 0x7ff5a8132930 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:59:23.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:23.358+0000 7ff5ad191640 1 -- 172.21.15.27:0/3131080517 >> 172.21.15.27:0/3131080517 conn(0x7ff5a806e490 msgr2=0x7ff5a806e8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:59:23.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:23.358+0000 7ff5ad191640 1 -- 172.21.15.27:0/3131080517 shutdown_connections 2024-09-16T07:59:23.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:23.358+0000 7ff5ad191640 1 -- 172.21.15.27:0/3131080517 wait complete. 2024-09-16T07:59:23.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:23.358+0000 7ff5ad191640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T07:59:23.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:23.361+0000 7ff5ad191640 1 Processor -- start 2024-09-16T07:59:23.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:23.361+0000 7ff5ad191640 1 -- start start 2024-09-16T07:59:23.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:23.362+0000 7ff5ad191640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff5a80a3450 0x7ff5a811e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:59:23.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:23.362+0000 7ff5ad191640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7ff5a8121f40 con 0x7ff5a80a3450 2024-09-16T07:59:23.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:23.363+0000 7ff5a6d9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff5a80a3450 0x7ff5a811e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:59:23.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:23.363+0000 7ff5a6d9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff5a80a3450 0x7ff5a811e090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40368/0 (socket says 172.21.15.27:40368) 2024-09-16T07:59:23.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:23.363+0000 7ff5a6d9d640 1 -- 172.21.15.27:0/2862046546 learned_addr learned my addr 172.21.15.27:0/2862046546 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-09-16T07:59:23.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:23.363+0000 7ff5a6d9d640 1 -- 172.21.15.27:0/2862046546 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5a81320a0 con 0x7ff5a80a3450 2024-09-16T07:59:23.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:23.363+0000 7ff5a6d9d640 1 --2- 172.21.15.27:0/2862046546 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff5a80a3450 0x7ff5a811e090 crc :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:59:23.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:23.363+0000 7ff58f7fe640 1 -- 172.21.15.27:0/2862046546 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7ff598014620 con 0x7ff5a80a3450 2024-09-16T07:59:23.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:23.363+0000 7ff5ad191640 1 -- 172.21.15.27:0/2862046546 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5a8135960 con 0x7ff5a80a3450 2024-09-16T07:59:23.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:23.363+0000 7ff58f7fe640 1 -- 172.21.15.27:0/2862046546 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7ff598014780 con 0x7ff5a80a3450 2024-09-16T07:59:23.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:23.363+0000 7ff58f7fe640 1 -- 172.21.15.27:0/2862046546 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7ff598014a20 con 0x7ff5a80a3450 2024-09-16T07:59:23.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:23.364+0000 7ff5ad191640 1 -- 172.21.15.27:0/2862046546 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5a81219f0 con 0x7ff5a80a3450 2024-09-16T07:59:23.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:23.365+0000 7ff5ad191640 1 -- 172.21.15.27:0/2862046546 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff5a8120a40 con 0x7ff5a80a3450 2024-09-16T07:59:23.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:23.365+0000 7ff58f7fe640 1 -- 172.21.15.27:0/2862046546 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) ==== 50954+0+0 (crc 0 0 0) 0x7ff598014c20 con 0x7ff5a80a3450 2024-09-16T07:59:23.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:23.365+0000 7ff58f7fe640 1 --2- 172.21.15.27:0/2862046546 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7ff59403e8e0 0x7ff594040df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:59:23.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:23.365+0000 7ff58f7fe640 1 -- 172.21.15.27:0/2862046546 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(228..228 src has 1..228) ==== 2700+0+0 (crc 0 0 0) 0x7ff598019040 con 0x7ff5a80a3450 2024-09-16T07:59:23.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:23.365+0000 7ff5a659c640 1 --2- 172.21.15.27:0/2862046546 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7ff59403e8e0 0x7ff594040df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:59:23.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:23.366+0000 7ff5a659c640 1 --2- 172.21.15.27:0/2862046546 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7ff59403e8e0 0x7ff594040df0 crc :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:59:23.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:23.370+0000 7ff58f7fe640 1 -- 172.21.15.27:0/2862046546 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (crc 0 0 0) 0x7ff598018190 con 0x7ff5a80a3450 2024-09-16T07:59:23.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:23.552+0000 7ff5ad191640 1 -- 172.21.15.27:0/2862046546 --> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7ff5a809c830 con 0x7ff59403e8e0 2024-09-16T07:59:23.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:23.553+0000 7ff58f7fe640 1 -- 172.21.15.27:0/2862046546 <== mgr.4102 v2:172.21.15.187:6800/4038049593 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+31238 (crc 0 0 0) 0x7ff59c005910 con 0x7ff59403e8e0 2024-09-16T07:59:23.552 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-16T07:59:23.553 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-09-16T07:59:23.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:23.556+0000 7ff5ad191640 1 -- 172.21.15.27:0/2862046546 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7ff59403e8e0 msgr2=0x7ff594040df0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:59:23.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:23.556+0000 7ff5ad191640 1 --2- 172.21.15.27:0/2862046546 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7ff59403e8e0 0x7ff594040df0 crc :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:59:23.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:23.556+0000 7ff5ad191640 1 -- 172.21.15.27:0/2862046546 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff5a80a3450 msgr2=0x7ff5a811e090 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:59:23.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:23.556+0000 7ff5ad191640 1 --2- 172.21.15.27:0/2862046546 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff5a80a3450 0x7ff5a811e090 crc :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:59:23.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:23.556+0000 7ff5ad191640 1 -- 172.21.15.27:0/2862046546 shutdown_connections 2024-09-16T07:59:23.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:23.556+0000 7ff5ad191640 1 --2- 172.21.15.27:0/2862046546 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7ff59403e8e0 0x7ff594040df0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:59:23.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:23.556+0000 7ff5ad191640 1 --2- 172.21.15.27:0/2862046546 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff5a80a3450 0x7ff5a811e090 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:59:23.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:23.556+0000 7ff5ad191640 1 -- 172.21.15.27:0/2862046546 >> 172.21.15.27:0/2862046546 conn(0x7ff5a806e490 msgr2=0x7ff5a81bd520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:59:23.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:23.556+0000 7ff5ad191640 1 -- 172.21.15.27:0/2862046546 shutdown_connections 2024-09-16T07:59:23.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:23.556+0000 7ff5ad191640 1 -- 172.21.15.27:0/2862046546 wait complete. 2024-09-16T07:59:23.566 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":372,"stamp":"2024-09-16T07:59:23.147926+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":209190912,"kb_used":15568,"kb_used_data":15568,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":209175344,"statfs":{"total":214211493888,"available":214195552256,"internally_reserved":0,"allocated":15941632,"data_stored":15941632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.428583"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":645,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-09-16T07:59:18.795976+0000","last_change":"2024-09-16T07:54:26.740923+0000","last_active":"2024-09-16T07:59:18.795976+0000","last_peered":"2024-09-16T07:59:18.795976+0000","last_clean":"2024-09-16T07:59:18.795976+0000","last_became_active":"2024-09-16T07:54:24.784360+0000","last_became_peered":"2024-09-16T07:54:24.784360+0000","last_unstale":"2024-09-16T07:59:18.795976+0000","last_undegraded":"2024-09-16T07:59:18.795976+0000","last_fullsized":"2024-09-16T07:59:18.795976+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T07:54:23.687651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T07:54:23.687651+0000","last_clean_scrub_stamp":"2024-09-16T07:54:23.687651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":645,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-09-16T07:59:18.761001+0000","last_change":"2024-09-16T07:54:26.739464+0000","last_active":"2024-09-16T07:59:18.761001+0000","last_peered":"2024-09-16T07:59:18.761001+0000","last_clean":"2024-09-16T07:59:18.761001+0000","last_became_active":"2024-09-16T07:54:24.805913+0000","last_became_peered":"2024-09-16T07:54:24.805913+0000","last_unstale":"2024-09-16T07:59:18.761001+0000","last_undegraded":"2024-09-16T07:59:18.761001+0000","last_fullsized":"2024-09-16T07:59:18.761001+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T07:54:23.687651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T07:54:23.687651+0000","last_clean_scrub_stamp":"2024-09-16T07:54:23.687651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":645,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-09-16T07:59:18.782573+0000","last_change":"2024-09-16T07:54:26.735332+0000","last_active":"2024-09-16T07:59:18.782573+0000","last_peered":"2024-09-16T07:59:18.782573+0000","last_clean":"2024-09-16T07:59:18.782573+0000","last_became_active":"2024-09-16T07:54:24.773365+0000","last_became_peered":"2024-09-16T07:54:24.773365+0000","last_unstale":"2024-09-16T07:59:18.782573+0000","last_undegraded":"2024-09-16T07:59:18.782573+0000","last_fullsized":"2024-09-16T07:59:18.782573+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T07:54:23.687651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T07:54:23.687651+0000","last_clean_scrub_stamp":"2024-09-16T07:54:23.687651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":645,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-09-16T07:59:18.751250+0000","last_change":"2024-09-16T07:54:26.734682+0000","last_active":"2024-09-16T07:59:18.751250+0000","last_peered":"2024-09-16T07:59:18.751250+0000","last_clean":"2024-09-16T07:59:18.751250+0000","last_became_active":"2024-09-16T07:54:24.756974+0000","last_became_peered":"2024-09-16T07:54:24.756974+0000","last_unstale":"2024-09-16T07:59:18.751250+0000","last_undegraded":"2024-09-16T07:59:18.751250+0000","last_fullsized":"2024-09-16T07:59:18.751250+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T07:54:23.687651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T07:54:23.687651+0000","last_clean_scrub_stamp":"2024-09-16T07:54:23.687651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"15'2","reported_seq":647,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-09-16T07:59:18.780803+0000","last_change":"2024-09-16T07:54:26.741927+0000","last_active":"2024-09-16T07:59:18.780803+0000","last_peered":"2024-09-16T07:59:18.780803+0000","last_clean":"2024-09-16T07:59:18.780803+0000","last_became_active":"2024-09-16T07:54:24.794754+0000","last_became_peered":"2024-09-16T07:54:24.794754+0000","last_unstale":"2024-09-16T07:59:18.780803+0000","last_undegraded":"2024-09-16T07:59:18.780803+0000","last_fullsized":"2024-09-16T07:59:18.780803+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T07:54:23.687651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T07:54:23.687651+0000","last_clean_scrub_stamp":"2024-09-16T07:54:23.687651+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":645,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-09-16T07:59:18.758597+0000","last_change":"2024-09-16T07:54:26.734478+0000","last_active":"2024-09-16T07:59:18.758597+0000","last_peered":"2024-09-16T07:59:18.758597+0000","last_clean":"2024-09-16T07:59:18.758597+0000","last_became_active":"2024-09-16T07:54:24.774923+0000","last_became_peered":"2024-09-16T07:54:24.774923+0000","last_unstale":"2024-09-16T07:59:18.758597+0000","last_undegraded":"2024-09-16T07:59:18.758597+0000","last_fullsized":"2024-09-16T07:59:18.758597+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T07:54:23.687651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T07:54:23.687651+0000","last_clean_scrub_stamp":"2024-09-16T07:54:23.687651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":645,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-09-16T07:59:18.810530+0000","last_change":"2024-09-16T07:54:26.742127+0000","last_active":"2024-09-16T07:59:18.810530+0000","last_peered":"2024-09-16T07:59:18.810530+0000","last_clean":"2024-09-16T07:59:18.810530+0000","last_became_active":"2024-09-16T07:54:24.773472+0000","last_became_peered":"2024-09-16T07:54:24.773472+0000","last_unstale":"2024-09-16T07:59:18.810530+0000","last_undegraded":"2024-09-16T07:59:18.810530+0000","last_fullsized":"2024-09-16T07:59:18.810530+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T07:54:23.687651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T07:54:23.687651+0000","last_clean_scrub_stamp":"2024-09-16T07:54:23.687651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"13'1","reported_seq":646,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-09-16T07:59:18.805158+0000","last_change":"2024-09-16T07:54:26.739091+0000","last_active":"2024-09-16T07:59:18.805158+0000","last_peered":"2024-09-16T07:59:18.805158+0000","last_clean":"2024-09-16T07:59:18.805158+0000","last_became_active":"2024-09-16T07:54:24.801907+0000","last_became_peered":"2024-09-16T07:54:24.801907+0000","last_unstale":"2024-09-16T07:59:18.805158+0000","last_undegraded":"2024-09-16T07:59:18.805158+0000","last_fullsized":"2024-09-16T07:59:18.805158+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T07:54:23.687651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T07:54:23.687651+0000","last_clean_scrub_stamp":"2024-09-16T07:54:23.687651+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'52","reported_seq":717,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-09-16T07:59:18.755371+0000","last_change":"2024-09-16T07:54:22.755090+0000","last_active":"2024-09-16T07:59:18.755371+0000","last_peered":"2024-09-16T07:59:18.755371+0000","last_clean":"2024-09-16T07:59:18.755371+0000","last_became_active":"2024-09-16T07:54:22.736266+0000","last_became_peered":"2024-09-16T07:54:22.736266+0000","last_unstale":"2024-09-16T07:59:18.755371+0000","last_undegraded":"2024-09-16T07:59:18.755371+0000","last_fullsized":"2024-09-16T07:59:18.755371+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T07:54:21.678374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T07:54:21.678374+0000","last_clean_scrub_stamp":"2024-09-16T07:54:21.678374+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":214211493888,"available":214198435840,"internally_reserved":0,"allocated":13058048,"data_stored":13058048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53552873472,"available":53549346816,"internally_reserved":0,"allocated":3526656,"data_stored":3526656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705963,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":4340,"kb_used_data":4340,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52293388,"statfs":{"total":53552873472,"available":53548429312,"internally_reserved":0,"allocated":4444160,"data_stored":4444160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705963,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":3300,"kb_used_data":3300,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52294428,"statfs":{"total":53552873472,"available":53549494272,"internally_reserved":0,"allocated":3379200,"data_stored":3379200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705963,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":3376,"kb_used_data":3376,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52294352,"statfs":{"total":53552873472,"available":53549416448,"internally_reserved":0,"allocated":3457024,"data_stored":3457024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705963,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":4552,"kb_used_data":4552,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52293176,"statfs":{"total":53552873472,"available":53548212224,"internally_reserved":0,"allocated":4661248,"data_stored":4661248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":3,"total":53552873472,"available":53549346816,"internally_reserved":0,"allocated":3526656,"data_stored":3526656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53552873472,"available":53549064192,"internally_reserved":0,"allocated":3809280,"data_stored":3809280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53552873472,"available":53550006272,"internally_reserved":0,"allocated":2867200,"data_stored":2867200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53552873472,"available":53550018560,"internally_reserved":0,"allocated":2854912,"data_stored":2854912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":53552873472,"available":53549346816,"internally_reserved":0,"allocated":3526656,"data_stored":3526656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T07:59:23.567 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-16T07:59:23.567 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T07:59:23.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:23.679+0000 7fa5bc8c8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T07:59:23.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:23.684+0000 7fa5bc8c8640 1 Processor -- start 2024-09-16T07:59:23.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:23.684+0000 7fa5bc8c8640 1 -- start start 2024-09-16T07:59:23.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:23.686+0000 7fa5bc8c8640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa5b8138720 0x7fa5b813ab40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:59:23.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:23.686+0000 7fa5bc8c8640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa5b8133040 con 0x7fa5b8132520 2024-09-16T07:59:23.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:23.686+0000 7fa5bc8c8640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 -- 0x7fa5b806f6d0 con 0x7fa5b8138720 2024-09-16T07:59:23.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:23.686+0000 7fa5b5d9b640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa5b8138720 0x7fa5b813ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:59:23.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:23.686+0000 7fa5b5d9b640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa5b8138720 0x7fa5b813ab40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40374/0 (socket says 172.21.15.27:40374) 2024-09-16T07:59:23.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:23.686+0000 7fa5b5d9b640 1 -- 172.21.15.27:0/3146829980 learned_addr learned my addr 172.21.15.27:0/3146829980 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-09-16T07:59:23.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:23.686+0000 7fa5b659c640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fa5b8132520 0x7fa5b8132930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:36094/0 (socket says 172.21.15.27:36094) 2024-09-16T07:59:23.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:23.686+0000 7fa5b5d9b640 1 -- 172.21.15.27:0/3146829980 >> v1:172.21.15.27:6789/0 conn(0x7fa5b8132520 legacy=0x7fa5b8132930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:59:23.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:23.686+0000 7fa5b5d9b640 1 -- 172.21.15.27:0/3146829980 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5b806f3d0 con 0x7fa5b8138720 2024-09-16T07:59:23.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:23.687+0000 7fa5b5d9b640 1 --2- 172.21.15.27:0/3146829980 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5b8138720 0x7fa5b813ab40 crc :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=157639343372be58 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:59:23.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:23.687+0000 7fa5b559a640 1 -- 172.21.15.27:0/3146829980 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fa5a0009e50 con 0x7fa5b8138720 2024-09-16T07:59:23.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:23.687+0000 7fa5b559a640 1 -- 172.21.15.27:0/3146829980 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7fa5a000e230 con 0x7fa5b8138720 2024-09-16T07:59:23.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:23.687+0000 7fa5b559a640 1 -- 172.21.15.27:0/3146829980 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fa5a000e4d0 con 0x7fa5b8138720 2024-09-16T07:59:23.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:23.688+0000 7fa5bc8c8640 1 -- 172.21.15.27:0/3146829980 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5b8138720 msgr2=0x7fa5b813ab40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:59:23.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:23.688+0000 7fa5bc8c8640 1 --2- 172.21.15.27:0/3146829980 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5b8138720 0x7fa5b813ab40 crc :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:59:23.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:23.688+0000 7fa5bc8c8640 1 -- 172.21.15.27:0/3146829980 shutdown_connections 2024-09-16T07:59:23.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:23.688+0000 7fa5bc8c8640 1 --2- 172.21.15.27:0/3146829980 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5b8138720 0x7fa5b813ab40 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:59:23.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:23.688+0000 7fa5bc8c8640 1 -- 172.21.15.27:0/3146829980 >> 172.21.15.27:0/3146829980 conn(0x7fa5b806e490 msgr2=0x7fa5b806e8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:59:23.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:23.688+0000 7fa5bc8c8640 1 -- 172.21.15.27:0/3146829980 shutdown_connections 2024-09-16T07:59:23.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:23.688+0000 7fa5bc8c8640 1 -- 172.21.15.27:0/3146829980 wait complete. 2024-09-16T07:59:23.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:23.688+0000 7fa5bc8c8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T07:59:23.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:23.690+0000 7fa5bc8c8640 1 Processor -- start 2024-09-16T07:59:23.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:23.690+0000 7fa5bc8c8640 1 -- start start 2024-09-16T07:59:23.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:23.692+0000 7fa5bc8c8640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5b8132520 0x7fa5b8130c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:59:23.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:23.692+0000 7fa5bc8c8640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fa5b81313d0 con 0x7fa5b8132520 2024-09-16T07:59:23.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:23.692+0000 7fa5b659c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5b8132520 0x7fa5b8130c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:59:23.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:23.692+0000 7fa5b659c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5b8132520 0x7fa5b8130c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40390/0 (socket says 172.21.15.27:40390) 2024-09-16T07:59:23.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:23.692+0000 7fa5b659c640 1 -- 172.21.15.27:0/1426088058 learned_addr learned my addr 172.21.15.27:0/1426088058 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-09-16T07:59:23.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:23.692+0000 7fa5b659c640 1 -- 172.21.15.27:0/1426088058 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5b80a07b0 con 0x7fa5b8132520 2024-09-16T07:59:23.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:23.692+0000 7fa5b659c640 1 --2- 172.21.15.27:0/1426088058 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5b8132520 0x7fa5b8130c70 crc :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:59:23.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:23.692+0000 7fa5aeffd640 1 -- 172.21.15.27:0/1426088058 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fa5a80059a0 con 0x7fa5b8132520 2024-09-16T07:59:23.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:23.692+0000 7fa5aeffd640 1 -- 172.21.15.27:0/1426088058 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7fa5a8005b00 con 0x7fa5b8132520 2024-09-16T07:59:23.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:23.692+0000 7fa5bc8c8640 1 -- 172.21.15.27:0/1426088058 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5b8120bc0 con 0x7fa5b8132520 2024-09-16T07:59:23.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:23.692+0000 7fa5aeffd640 1 -- 172.21.15.27:0/1426088058 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fa5a8005da0 con 0x7fa5b8132520 2024-09-16T07:59:23.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:23.693+0000 7fa5bc8c8640 1 -- 172.21.15.27:0/1426088058 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5b8120030 con 0x7fa5b8132520 2024-09-16T07:59:23.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:23.694+0000 7fa5bc8c8640 1 -- 172.21.15.27:0/1426088058 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa5b806a300 con 0x7fa5b8132520 2024-09-16T07:59:23.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:23.694+0000 7fa5aeffd640 1 -- 172.21.15.27:0/1426088058 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) ==== 50954+0+0 (crc 0 0 0) 0x7fa5a80070b0 con 0x7fa5b8132520 2024-09-16T07:59:23.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:23.695+0000 7fa5aeffd640 1 --2- 172.21.15.27:0/1426088058 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7fa59803e930 0x7fa598040e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:59:23.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:23.695+0000 7fa5aeffd640 1 -- 172.21.15.27:0/1426088058 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(228..228 src has 1..228) ==== 2700+0+0 (crc 0 0 0) 0x7fa5a804f500 con 0x7fa5b8132520 2024-09-16T07:59:23.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:23.695+0000 7fa5b5d9b640 1 --2- 172.21.15.27:0/1426088058 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7fa59803e930 0x7fa598040e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:59:23.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:23.696+0000 7fa5b5d9b640 1 --2- 172.21.15.27:0/1426088058 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7fa59803e930 0x7fa598040e40 crc :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:59:23.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:23.700+0000 7fa5aeffd640 1 -- 172.21.15.27:0/1426088058 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (crc 0 0 0) 0x7fa5a800f230 con 0x7fa5b8132520 2024-09-16T07:59:23.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:23.885+0000 7fa5bc8c8640 1 -- 172.21.15.27:0/1426088058 --> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fa5b80a0640 con 0x7fa59803e930 2024-09-16T07:59:23.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:23.886+0000 7fa5aeffd640 1 -- 172.21.15.27:0/1426088058 <== mgr.4102 v2:172.21.15.187:6800/4038049593 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+31238 (crc 0 0 0) 0x7fa5b80a0640 con 0x7fa59803e930 2024-09-16T07:59:23.885 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-16T07:59:23.885 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-09-16T07:59:23.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:23.888+0000 7fa5bc8c8640 1 -- 172.21.15.27:0/1426088058 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7fa59803e930 msgr2=0x7fa598040e40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:59:23.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:23.888+0000 7fa5bc8c8640 1 --2- 172.21.15.27:0/1426088058 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7fa59803e930 0x7fa598040e40 crc :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:59:23.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:23.888+0000 7fa5bc8c8640 1 -- 172.21.15.27:0/1426088058 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5b8132520 msgr2=0x7fa5b8130c70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:59:23.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:23.888+0000 7fa5bc8c8640 1 --2- 172.21.15.27:0/1426088058 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5b8132520 0x7fa5b8130c70 crc :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:59:23.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:23.888+0000 7fa5bc8c8640 1 -- 172.21.15.27:0/1426088058 shutdown_connections 2024-09-16T07:59:23.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:23.888+0000 7fa5bc8c8640 1 --2- 172.21.15.27:0/1426088058 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7fa59803e930 0x7fa598040e40 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:59:23.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:23.888+0000 7fa5bc8c8640 1 --2- 172.21.15.27:0/1426088058 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5b8132520 0x7fa5b8130c70 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:59:23.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:23.888+0000 7fa5bc8c8640 1 -- 172.21.15.27:0/1426088058 >> 172.21.15.27:0/1426088058 conn(0x7fa5b806e490 msgr2=0x7fa5b80a3190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:59:23.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:23.888+0000 7fa5bc8c8640 1 -- 172.21.15.27:0/1426088058 shutdown_connections 2024-09-16T07:59:23.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:23.888+0000 7fa5bc8c8640 1 -- 172.21.15.27:0/1426088058 wait complete. 2024-09-16T07:59:23.898 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":372,"stamp":"2024-09-16T07:59:23.147926+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":209190912,"kb_used":15568,"kb_used_data":15568,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":209175344,"statfs":{"total":214211493888,"available":214195552256,"internally_reserved":0,"allocated":15941632,"data_stored":15941632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.428583"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":645,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-09-16T07:59:18.795976+0000","last_change":"2024-09-16T07:54:26.740923+0000","last_active":"2024-09-16T07:59:18.795976+0000","last_peered":"2024-09-16T07:59:18.795976+0000","last_clean":"2024-09-16T07:59:18.795976+0000","last_became_active":"2024-09-16T07:54:24.784360+0000","last_became_peered":"2024-09-16T07:54:24.784360+0000","last_unstale":"2024-09-16T07:59:18.795976+0000","last_undegraded":"2024-09-16T07:59:18.795976+0000","last_fullsized":"2024-09-16T07:59:18.795976+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T07:54:23.687651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T07:54:23.687651+0000","last_clean_scrub_stamp":"2024-09-16T07:54:23.687651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":645,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-09-16T07:59:18.761001+0000","last_change":"2024-09-16T07:54:26.739464+0000","last_active":"2024-09-16T07:59:18.761001+0000","last_peered":"2024-09-16T07:59:18.761001+0000","last_clean":"2024-09-16T07:59:18.761001+0000","last_became_active":"2024-09-16T07:54:24.805913+0000","last_became_peered":"2024-09-16T07:54:24.805913+0000","last_unstale":"2024-09-16T07:59:18.761001+0000","last_undegraded":"2024-09-16T07:59:18.761001+0000","last_fullsized":"2024-09-16T07:59:18.761001+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T07:54:23.687651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T07:54:23.687651+0000","last_clean_scrub_stamp":"2024-09-16T07:54:23.687651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":645,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-09-16T07:59:18.782573+0000","last_change":"2024-09-16T07:54:26.735332+0000","last_active":"2024-09-16T07:59:18.782573+0000","last_peered":"2024-09-16T07:59:18.782573+0000","last_clean":"2024-09-16T07:59:18.782573+0000","last_became_active":"2024-09-16T07:54:24.773365+0000","last_became_peered":"2024-09-16T07:54:24.773365+0000","last_unstale":"2024-09-16T07:59:18.782573+0000","last_undegraded":"2024-09-16T07:59:18.782573+0000","last_fullsized":"2024-09-16T07:59:18.782573+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T07:54:23.687651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T07:54:23.687651+0000","last_clean_scrub_stamp":"2024-09-16T07:54:23.687651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":645,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-09-16T07:59:18.751250+0000","last_change":"2024-09-16T07:54:26.734682+0000","last_active":"2024-09-16T07:59:18.751250+0000","last_peered":"2024-09-16T07:59:18.751250+0000","last_clean":"2024-09-16T07:59:18.751250+0000","last_became_active":"2024-09-16T07:54:24.756974+0000","last_became_peered":"2024-09-16T07:54:24.756974+0000","last_unstale":"2024-09-16T07:59:18.751250+0000","last_undegraded":"2024-09-16T07:59:18.751250+0000","last_fullsized":"2024-09-16T07:59:18.751250+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T07:54:23.687651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T07:54:23.687651+0000","last_clean_scrub_stamp":"2024-09-16T07:54:23.687651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"15'2","reported_seq":647,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-09-16T07:59:18.780803+0000","last_change":"2024-09-16T07:54:26.741927+0000","last_active":"2024-09-16T07:59:18.780803+0000","last_peered":"2024-09-16T07:59:18.780803+0000","last_clean":"2024-09-16T07:59:18.780803+0000","last_became_active":"2024-09-16T07:54:24.794754+0000","last_became_peered":"2024-09-16T07:54:24.794754+0000","last_unstale":"2024-09-16T07:59:18.780803+0000","last_undegraded":"2024-09-16T07:59:18.780803+0000","last_fullsized":"2024-09-16T07:59:18.780803+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T07:54:23.687651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T07:54:23.687651+0000","last_clean_scrub_stamp":"2024-09-16T07:54:23.687651+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":645,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-09-16T07:59:18.758597+0000","last_change":"2024-09-16T07:54:26.734478+0000","last_active":"2024-09-16T07:59:18.758597+0000","last_peered":"2024-09-16T07:59:18.758597+0000","last_clean":"2024-09-16T07:59:18.758597+0000","last_became_active":"2024-09-16T07:54:24.774923+0000","last_became_peered":"2024-09-16T07:54:24.774923+0000","last_unstale":"2024-09-16T07:59:18.758597+0000","last_undegraded":"2024-09-16T07:59:18.758597+0000","last_fullsized":"2024-09-16T07:59:18.758597+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T07:54:23.687651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T07:54:23.687651+0000","last_clean_scrub_stamp":"2024-09-16T07:54:23.687651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":645,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-09-16T07:59:18.810530+0000","last_change":"2024-09-16T07:54:26.742127+0000","last_active":"2024-09-16T07:59:18.810530+0000","last_peered":"2024-09-16T07:59:18.810530+0000","last_clean":"2024-09-16T07:59:18.810530+0000","last_became_active":"2024-09-16T07:54:24.773472+0000","last_became_peered":"2024-09-16T07:54:24.773472+0000","last_unstale":"2024-09-16T07:59:18.810530+0000","last_undegraded":"2024-09-16T07:59:18.810530+0000","last_fullsized":"2024-09-16T07:59:18.810530+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T07:54:23.687651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T07:54:23.687651+0000","last_clean_scrub_stamp":"2024-09-16T07:54:23.687651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"13'1","reported_seq":646,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-09-16T07:59:18.805158+0000","last_change":"2024-09-16T07:54:26.739091+0000","last_active":"2024-09-16T07:59:18.805158+0000","last_peered":"2024-09-16T07:59:18.805158+0000","last_clean":"2024-09-16T07:59:18.805158+0000","last_became_active":"2024-09-16T07:54:24.801907+0000","last_became_peered":"2024-09-16T07:54:24.801907+0000","last_unstale":"2024-09-16T07:59:18.805158+0000","last_undegraded":"2024-09-16T07:59:18.805158+0000","last_fullsized":"2024-09-16T07:59:18.805158+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T07:54:23.687651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T07:54:23.687651+0000","last_clean_scrub_stamp":"2024-09-16T07:54:23.687651+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'52","reported_seq":717,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-09-16T07:59:18.755371+0000","last_change":"2024-09-16T07:54:22.755090+0000","last_active":"2024-09-16T07:59:18.755371+0000","last_peered":"2024-09-16T07:59:18.755371+0000","last_clean":"2024-09-16T07:59:18.755371+0000","last_became_active":"2024-09-16T07:54:22.736266+0000","last_became_peered":"2024-09-16T07:54:22.736266+0000","last_unstale":"2024-09-16T07:59:18.755371+0000","last_undegraded":"2024-09-16T07:59:18.755371+0000","last_fullsized":"2024-09-16T07:59:18.755371+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T07:54:21.678374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T07:54:21.678374+0000","last_clean_scrub_stamp":"2024-09-16T07:54:21.678374+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":214211493888,"available":214198435840,"internally_reserved":0,"allocated":13058048,"data_stored":13058048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53552873472,"available":53549346816,"internally_reserved":0,"allocated":3526656,"data_stored":3526656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705963,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":4340,"kb_used_data":4340,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52293388,"statfs":{"total":53552873472,"available":53548429312,"internally_reserved":0,"allocated":4444160,"data_stored":4444160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705963,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":3300,"kb_used_data":3300,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52294428,"statfs":{"total":53552873472,"available":53549494272,"internally_reserved":0,"allocated":3379200,"data_stored":3379200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705963,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":3376,"kb_used_data":3376,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52294352,"statfs":{"total":53552873472,"available":53549416448,"internally_reserved":0,"allocated":3457024,"data_stored":3457024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705963,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":4552,"kb_used_data":4552,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52293176,"statfs":{"total":53552873472,"available":53548212224,"internally_reserved":0,"allocated":4661248,"data_stored":4661248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":3,"total":53552873472,"available":53549346816,"internally_reserved":0,"allocated":3526656,"data_stored":3526656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53552873472,"available":53549064192,"internally_reserved":0,"allocated":3809280,"data_stored":3809280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53552873472,"available":53550006272,"internally_reserved":0,"allocated":2867200,"data_stored":2867200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53552873472,"available":53550018560,"internally_reserved":0,"allocated":2854912,"data_stored":2854912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":53552873472,"available":53549346816,"internally_reserved":0,"allocated":3526656,"data_stored":3526656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T07:59:23.899 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T07:59:24.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:24.011+0000 7f6c1969c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T07:59:24.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:24.016+0000 7f6c1969c640 1 Processor -- start 2024-09-16T07:59:24.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:24.016+0000 7f6c1969c640 1 -- start start 2024-09-16T07:59:24.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:24.018+0000 7f6c1969c640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f6c14134ac0 0x7f6c14136ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:59:24.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:24.018+0000 7f6c1969c640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6c1412f190 con 0x7f6c1409daa0 2024-09-16T07:59:24.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:24.018+0000 7f6c1969c640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 -- 0x7f6c140706a0 con 0x7f6c14134ac0 2024-09-16T07:59:24.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:24.018+0000 7f6c12ffd640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f6c1409daa0 0x7f6c1409ca60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:36110/0 (socket says 172.21.15.27:36110) 2024-09-16T07:59:24.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:24.018+0000 7f6c12ffd640 1 -- 172.21.15.27:0/1284479708 learned_addr learned my addr 172.21.15.27:0/1284479708 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-09-16T07:59:24.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:24.018+0000 7f6c127fc640 1 --2- 172.21.15.27:0/1284479708 >> v2:172.21.15.27:3300/0 conn(0x7f6c14134ac0 0x7f6c14136ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:59:24.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:24.018+0000 7f6c127fc640 1 -- 172.21.15.27:0/1284479708 >> v1:172.21.15.27:6789/0 conn(0x7f6c1409daa0 legacy=0x7f6c1409ca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:59:24.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:24.018+0000 7f6c127fc640 1 -- 172.21.15.27:0/1284479708 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c1412e970 con 0x7f6c14134ac0 2024-09-16T07:59:24.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:24.018+0000 7f6c127fc640 1 --2- 172.21.15.27:0/1284479708 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6c14134ac0 0x7f6c14136ee0 crc :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dcea187fa6380818 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:59:24.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:24.018+0000 7f6c11ffb640 1 -- 172.21.15.27:0/1284479708 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f6c08009d90 con 0x7f6c14134ac0 2024-09-16T07:59:24.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:24.018+0000 7f6c11ffb640 1 -- 172.21.15.27:0/1284479708 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f6c0800f340 con 0x7f6c14134ac0 2024-09-16T07:59:24.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:24.018+0000 7f6c11ffb640 1 -- 172.21.15.27:0/1284479708 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f6c0800f5e0 con 0x7f6c14134ac0 2024-09-16T07:59:24.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:24.019+0000 7f6c1969c640 1 -- 172.21.15.27:0/1284479708 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6c14134ac0 msgr2=0x7f6c14136ee0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:59:24.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:24.019+0000 7f6c1969c640 1 --2- 172.21.15.27:0/1284479708 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6c14134ac0 0x7f6c14136ee0 crc :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:59:24.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:24.019+0000 7f6c1969c640 1 -- 172.21.15.27:0/1284479708 shutdown_connections 2024-09-16T07:59:24.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:24.019+0000 7f6c1969c640 1 --2- 172.21.15.27:0/1284479708 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6c14134ac0 0x7f6c14136ee0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:59:24.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:24.019+0000 7f6c1969c640 1 -- 172.21.15.27:0/1284479708 >> 172.21.15.27:0/1284479708 conn(0x7f6c14069d60 msgr2=0x7f6c1409e1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:59:24.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:24.019+0000 7f6c1969c640 1 -- 172.21.15.27:0/1284479708 shutdown_connections 2024-09-16T07:59:24.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:24.019+0000 7f6c1969c640 1 -- 172.21.15.27:0/1284479708 wait complete. 2024-09-16T07:59:24.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:24.020+0000 7f6c1969c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T07:59:24.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:24.021+0000 7f6c1969c640 1 Processor -- start 2024-09-16T07:59:24.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:24.022+0000 7f6c1969c640 1 -- start start 2024-09-16T07:59:24.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:24.023+0000 7f6c1969c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6c1409daa0 0x7f6c14119e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:59:24.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:24.023+0000 7f6c1969c640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f6c1411b940 con 0x7f6c1409daa0 2024-09-16T07:59:24.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:24.023+0000 7f6c12ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6c1409daa0 0x7f6c14119e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:59:24.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:24.023+0000 7f6c12ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6c1409daa0 0x7f6c14119e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40420/0 (socket says 172.21.15.27:40420) 2024-09-16T07:59:24.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:24.023+0000 7f6c12ffd640 1 -- 172.21.15.27:0/4176529464 learned_addr learned my addr 172.21.15.27:0/4176529464 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-09-16T07:59:24.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:24.023+0000 7f6c12ffd640 1 -- 172.21.15.27:0/4176529464 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c1411d690 con 0x7f6c1409daa0 2024-09-16T07:59:24.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:24.024+0000 7f6c12ffd640 1 --2- 172.21.15.27:0/4176529464 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6c1409daa0 0x7f6c14119e40 crc :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:59:24.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:24.024+0000 7f6bfb7fe640 1 -- 172.21.15.27:0/4176529464 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f6c0400ec30 con 0x7f6c1409daa0 2024-09-16T07:59:24.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:24.024+0000 7f6bfb7fe640 1 -- 172.21.15.27:0/4176529464 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f6c0400ed90 con 0x7f6c1409daa0 2024-09-16T07:59:24.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:24.024+0000 7f6c1969c640 1 -- 172.21.15.27:0/4176529464 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c1406fcd0 con 0x7f6c1409daa0 2024-09-16T07:59:24.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:24.024+0000 7f6bfb7fe640 1 -- 172.21.15.27:0/4176529464 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f6c04005a60 con 0x7f6c1409daa0 2024-09-16T07:59:24.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:24.024+0000 7f6c1969c640 1 -- 172.21.15.27:0/4176529464 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c1411d010 con 0x7f6c1409daa0 2024-09-16T07:59:24.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:24.025+0000 7f6bfb7fe640 1 -- 172.21.15.27:0/4176529464 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) ==== 50954+0+0 (crc 0 0 0) 0x7f6c04005c10 con 0x7f6c1409daa0 2024-09-16T07:59:24.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:24.025+0000 7f6c1969c640 1 -- 172.21.15.27:0/4176529464 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6c1412e270 con 0x7f6c1409daa0 2024-09-16T07:59:24.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:24.025+0000 7f6bfb7fe640 1 --2- 172.21.15.27:0/4176529464 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7f6c0003e930 0x7f6c00040e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:59:24.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:24.025+0000 7f6bfb7fe640 1 -- 172.21.15.27:0/4176529464 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(228..228 src has 1..228) ==== 2700+0+0 (crc 0 0 0) 0x7f6c040521b0 con 0x7f6c1409daa0 2024-09-16T07:59:24.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:24.026+0000 7f6c127fc640 1 --2- 172.21.15.27:0/4176529464 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7f6c0003e930 0x7f6c00040e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:59:24.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:24.027+0000 7f6c127fc640 1 --2- 172.21.15.27:0/4176529464 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7f6c0003e930 0x7f6c00040e40 crc :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:59:24.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:24.031+0000 7f6bfb7fe640 1 -- 172.21.15.27:0/4176529464 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (crc 0 0 0) 0x7f6c04001690 con 0x7f6c1409daa0 2024-09-16T07:59:24.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:24.217+0000 7f6c1969c640 1 -- 172.21.15.27:0/4176529464 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f6c1406fcd0 con 0x7f6c1409daa0 2024-09-16T07:59:24.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:24.218+0000 7f6bfb7fe640 1 -- 172.21.15.27:0/4176529464 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v228) ==== 74+0+8830 (crc 0 0 0) 0x7f6c1406fcd0 con 0x7f6c1409daa0 2024-09-16T07:59:24.216 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-16T07:59:24.216 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":228,"fsid":"16b22dd1-2c19-4258-b5fa-641ce257086f","created":"2024-09-16T07:54:12.578772+0000","modified":"2024-09-16T07:59:21.731069+0000","last_up_change":"2024-09-16T07:54:19.672678+0000","last_in_change":"2024-09-16T07:54:14.737256+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":72,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T07:54:21.103580+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-16T07:54:22.986244+0000","flags":270369,"flags_names":"hashpspool,nopgchange,selfmanaged_snaps,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":15,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9a91e4ff-5797-4c44-a7e4-c088c97e1063","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":226,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":250146213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6801","nonce":250146213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":250146213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6805","nonce":250146213}]},"public_addr":"172.21.15.27:6802/250146213","cluster_addr":"172.21.15.27:6801/250146213","heartbeat_back_addr":"172.21.15.27:6804/250146213","heartbeat_front_addr":"172.21.15.27:6805/250146213","state":["exists","up"]},{"osd":1,"uuid":"a1b27d22-cc01-4cc9-88c0-c28e8c60a99e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":226,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":584480168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6803","nonce":584480168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":584480168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6807","nonce":584480168}]},"public_addr":"172.21.15.27:6800/584480168","cluster_addr":"172.21.15.27:6803/584480168","heartbeat_back_addr":"172.21.15.27:6808/584480168","heartbeat_front_addr":"172.21.15.27:6807/584480168","state":["exists","up"]},{"osd":2,"uuid":"04f4b772-d2c6-4d73-894b-d5635c4d2dda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":226,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6809","nonce":4110688419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":4110688419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6807","nonce":4110688419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":4110688419}]},"public_addr":"172.21.15.187:6809/4110688419","cluster_addr":"172.21.15.187:6808/4110688419","heartbeat_back_addr":"172.21.15.187:6807/4110688419","heartbeat_front_addr":"172.21.15.187:6812/4110688419","state":["exists","up"]},{"osd":3,"uuid":"611aad1c-1d66-400c-a54d-b66c9579b6d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":226,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6811","nonce":1136904278}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":1136904278}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":1136904278}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6803","nonce":1136904278}]},"public_addr":"172.21.15.187:6811/1136904278","cluster_addr":"172.21.15.187:6810/1136904278","heartbeat_back_addr":"172.21.15.187:6804/1136904278","heartbeat_front_addr":"172.21.15.187:6803/1136904278","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T07:59:24.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:24.220+0000 7f6c1969c640 1 -- 172.21.15.27:0/4176529464 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7f6c0003e930 msgr2=0x7f6c00040e40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:59:24.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:24.220+0000 7f6c1969c640 1 --2- 172.21.15.27:0/4176529464 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7f6c0003e930 0x7f6c00040e40 crc :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:59:24.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:24.220+0000 7f6c1969c640 1 -- 172.21.15.27:0/4176529464 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6c1409daa0 msgr2=0x7f6c14119e40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:59:24.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:24.220+0000 7f6c1969c640 1 --2- 172.21.15.27:0/4176529464 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6c1409daa0 0x7f6c14119e40 crc :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:59:24.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:24.220+0000 7f6c1969c640 1 -- 172.21.15.27:0/4176529464 shutdown_connections 2024-09-16T07:59:24.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:24.220+0000 7f6c1969c640 1 --2- 172.21.15.27:0/4176529464 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7f6c0003e930 0x7f6c00040e40 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:59:24.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:24.220+0000 7f6c1969c640 1 --2- 172.21.15.27:0/4176529464 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6c1409daa0 0x7f6c14119e40 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:59:24.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:24.220+0000 7f6c1969c640 1 -- 172.21.15.27:0/4176529464 >> 172.21.15.27:0/4176529464 conn(0x7f6c14069d60 msgr2=0x7f6c14133e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:59:24.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:24.220+0000 7f6c1969c640 1 -- 172.21.15.27:0/4176529464 shutdown_connections 2024-09-16T07:59:24.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:24.220+0000 7f6c1969c640 1 -- 172.21.15.27:0/4176529464 wait complete. 2024-09-16T07:59:25.231 INFO:tasks.ceph:Scrubbing osd.0 2024-09-16T07:59:25.232 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-09-16T07:59:25.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.351+0000 7f375e391640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T07:59:25.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.361+0000 7f375e391640 1 Processor -- start 2024-09-16T07:59:25.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.361+0000 7f375e391640 1 -- start start 2024-09-16T07:59:25.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.363+0000 7f375e391640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3758133a30 0x7f3758138850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:59:25.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.363+0000 7f375e391640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3758074420 con 0x7f37580733a0 2024-09-16T07:59:25.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.363+0000 7f375e391640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 -- 0x7f37580749a0 con 0x7f3758133a30 2024-09-16T07:59:25.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.363+0000 7f37577fe640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3758133a30 0x7f3758138850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:59:25.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.363+0000 7f37577fe640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3758133a30 0x7f3758138850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40428/0 (socket says 172.21.15.27:40428) 2024-09-16T07:59:25.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.363+0000 7f3757fff640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f37580733a0 0x7f3758133320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:36114/0 (socket says 172.21.15.27:36114) 2024-09-16T07:59:25.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.363+0000 7f37577fe640 1 -- 172.21.15.27:0/1569275454 learned_addr learned my addr 172.21.15.27:0/1569275454 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-09-16T07:59:25.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.363+0000 7f37577fe640 1 -- 172.21.15.27:0/1569275454 >> v1:172.21.15.27:6789/0 conn(0x7f37580733a0 legacy=0x7f3758133320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:59:25.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.363+0000 7f37577fe640 1 -- 172.21.15.27:0/1569275454 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3758074110 con 0x7f3758133a30 2024-09-16T07:59:25.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.363+0000 7f3756ffd640 1 -- 172.21.15.27:0/1569275454 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3848406514 0 0) 0x7f3758074420 con 0x7f37580733a0 2024-09-16T07:59:25.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.364+0000 7f37577fe640 1 --2- 172.21.15.27:0/1569275454 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3758133a30 0x7f3758138850 crc :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3aab741cb1fb26dd server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:59:25.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.364+0000 7f3756ffd640 1 -- 172.21.15.27:0/1569275454 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f374c009e50 con 0x7f3758133a30 2024-09-16T07:59:25.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.364+0000 7f3756ffd640 1 -- 172.21.15.27:0/1569275454 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f374c00e230 con 0x7f3758133a30 2024-09-16T07:59:25.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.364+0000 7f3756ffd640 1 -- 172.21.15.27:0/1569275454 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f374c00e4d0 con 0x7f3758133a30 2024-09-16T07:59:25.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.364+0000 7f375e391640 1 -- 172.21.15.27:0/1569275454 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3758133a30 msgr2=0x7f3758138850 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:59:25.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.364+0000 7f375e391640 1 --2- 172.21.15.27:0/1569275454 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3758133a30 0x7f3758138850 crc :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:59:25.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.365+0000 7f375e391640 1 -- 172.21.15.27:0/1569275454 shutdown_connections 2024-09-16T07:59:25.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.365+0000 7f375e391640 1 --2- 172.21.15.27:0/1569275454 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3758133a30 0x7f3758138850 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:59:25.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.365+0000 7f375e391640 1 -- 172.21.15.27:0/1569275454 >> 172.21.15.27:0/1569275454 conn(0x7f37580a0de0 msgr2=0x7f37580a11f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:59:25.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.365+0000 7f375e391640 1 -- 172.21.15.27:0/1569275454 shutdown_connections 2024-09-16T07:59:25.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.365+0000 7f375e391640 1 -- 172.21.15.27:0/1569275454 wait complete. 2024-09-16T07:59:25.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.365+0000 7f375e391640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T07:59:25.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.367+0000 7f375e391640 1 Processor -- start 2024-09-16T07:59:25.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.367+0000 7f375e391640 1 -- start start 2024-09-16T07:59:25.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.368+0000 7f375e391640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3758133a30 0x7f37581c7a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:59:25.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.368+0000 7f375e391640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f3758130360 con 0x7f3758133a30 2024-09-16T07:59:25.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.368+0000 7f3757fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3758133a30 0x7f37581c7a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:59:25.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.369+0000 7f3757fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3758133a30 0x7f37581c7a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40436/0 (socket says 172.21.15.27:40436) 2024-09-16T07:59:25.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.369+0000 7f3757fff640 1 -- 172.21.15.27:0/3218637997 learned_addr learned my addr 172.21.15.27:0/3218637997 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-09-16T07:59:25.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.369+0000 7f3757fff640 1 -- 172.21.15.27:0/3218637997 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3758130cf0 con 0x7f3758133a30 2024-09-16T07:59:25.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.369+0000 7f3757fff640 1 --2- 172.21.15.27:0/3218637997 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3758133a30 0x7f37581c7a10 crc :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:59:25.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.369+0000 7f3754ff9640 1 -- 172.21.15.27:0/3218637997 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f3748004b80 con 0x7f3758133a30 2024-09-16T07:59:25.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.369+0000 7f3754ff9640 1 -- 172.21.15.27:0/3218637997 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f3748004ce0 con 0x7f3758133a30 2024-09-16T07:59:25.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.369+0000 7f375e391640 1 -- 172.21.15.27:0/3218637997 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f375812f620 con 0x7f3758133a30 2024-09-16T07:59:25.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.369+0000 7f3754ff9640 1 -- 172.21.15.27:0/3218637997 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f3748007120 con 0x7f3758133a30 2024-09-16T07:59:25.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.369+0000 7f375e391640 1 -- 172.21.15.27:0/3218637997 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f375808d6c0 con 0x7f3758133a30 2024-09-16T07:59:25.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.370+0000 7f375e391640 1 -- 172.21.15.27:0/3218637997 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f3758068fa0 con 0x7f3758133a30 2024-09-16T07:59:25.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.370+0000 7f3754ff9640 1 -- 172.21.15.27:0/3218637997 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) ==== 50954+0+0 (crc 0 0 0) 0x7f37480072d0 con 0x7f3758133a30 2024-09-16T07:59:25.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.371+0000 7f3754ff9640 1 --2- 172.21.15.27:0/3218637997 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7f374403e930 0x7f3744040e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:59:25.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.371+0000 7f3754ff9640 1 -- 172.21.15.27:0/3218637997 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(228..228 src has 1..228) ==== 2700+0+0 (crc 0 0 0) 0x7f374804f7c0 con 0x7f3758133a30 2024-09-16T07:59:25.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.371+0000 7f3754ff9640 1 --2- 172.21.15.27:0/3218637997 >> v2:172.21.15.27:6802/250146213 conn(0x7f3744043930 0x7f3744045de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:59:25.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.371+0000 7f3754ff9640 1 -- 172.21.15.27:0/3218637997 --> v2:172.21.15.27:6802/250146213 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3744046470 con 0x7f3744043930 2024-09-16T07:59:25.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.371+0000 7f3754ff9640 1 -- 172.21.15.27:0/3218637997 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_get_version_reply(handle=1 version=228) ==== 24+0+0 (crc 0 0 0) 0x7f3748051c30 con 0x7f3758133a30 2024-09-16T07:59:25.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.371+0000 7f37577fe640 1 --2- 172.21.15.27:0/3218637997 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7f374403e930 0x7f3744040e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:59:25.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.372+0000 7f37577fe640 1 --2- 172.21.15.27:0/3218637997 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7f374403e930 0x7f3744040e40 crc :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:59:25.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.372+0000 7f375c92e640 1 --2- 172.21.15.27:0/3218637997 >> v2:172.21.15.27:6802/250146213 conn(0x7f3744043930 0x7f3744045de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-09-16T07:59:25.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.375+0000 7f375c92e640 1 --2- 172.21.15.27:0/3218637997 >> v2:172.21.15.27:6802/250146213 conn(0x7f3744043930 0x7f3744045de0 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:59:25.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.377+0000 7f3754ff9640 1 -- 172.21.15.27:0/3218637997 <== osd.0 v2:172.21.15.27:6802/250146213 1 ==== command_reply(tid 1: 0 ) ==== 8+0+5790 (crc 0 0 0) 0x7f3744046470 con 0x7f3744043930 2024-09-16T07:59:25.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.380+0000 7f375e391640 1 -- 172.21.15.27:0/3218637997 --> v2:172.21.15.27:6802/250146213 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f37580733a0 con 0x7f3744043930 2024-09-16T07:59:25.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.382+0000 7f3754ff9640 1 -- 172.21.15.27:0/3218637997 <== osd.0 v2:172.21.15.27:6802/250146213 2 ==== command_reply(tid 2: 0 ) ==== 8+0+22 (crc 0 0 0) 0x7f37580733a0 con 0x7f3744043930 2024-09-16T07:59:25.380 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-09-16T07:59:25.381 INFO:teuthology.orchestra.run.smithi027.stdout: "success": "" 2024-09-16T07:59:25.381 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-09-16T07:59:25.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.382+0000 7f375e391640 1 -- 172.21.15.27:0/3218637997 >> v2:172.21.15.27:6802/250146213 conn(0x7f3744043930 msgr2=0x7f3744045de0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:59:25.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.382+0000 7f375e391640 1 --2- 172.21.15.27:0/3218637997 >> v2:172.21.15.27:6802/250146213 conn(0x7f3744043930 0x7f3744045de0 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:59:25.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.382+0000 7f375e391640 1 -- 172.21.15.27:0/3218637997 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7f374403e930 msgr2=0x7f3744040e40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:59:25.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.382+0000 7f375e391640 1 --2- 172.21.15.27:0/3218637997 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7f374403e930 0x7f3744040e40 crc :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:59:25.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.382+0000 7f375e391640 1 -- 172.21.15.27:0/3218637997 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3758133a30 msgr2=0x7f37581c7a10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:59:25.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.382+0000 7f375e391640 1 --2- 172.21.15.27:0/3218637997 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3758133a30 0x7f37581c7a10 crc :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:59:25.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.382+0000 7f375e391640 1 -- 172.21.15.27:0/3218637997 shutdown_connections 2024-09-16T07:59:25.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.382+0000 7f375e391640 1 --2- 172.21.15.27:0/3218637997 >> v2:172.21.15.27:6802/250146213 conn(0x7f3744043930 0x7f3744045de0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:59:25.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.382+0000 7f375e391640 1 --2- 172.21.15.27:0/3218637997 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7f374403e930 0x7f3744040e40 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:59:25.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.382+0000 7f375e391640 1 --2- 172.21.15.27:0/3218637997 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3758133a30 0x7f37581c7a10 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:59:25.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.382+0000 7f375e391640 1 -- 172.21.15.27:0/3218637997 >> 172.21.15.27:0/3218637997 conn(0x7f37580a0de0 msgr2=0x7f37580a1d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:59:25.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.383+0000 7f375e391640 1 -- 172.21.15.27:0/3218637997 shutdown_connections 2024-09-16T07:59:25.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.383+0000 7f375e391640 1 -- 172.21.15.27:0/3218637997 wait complete. 2024-09-16T07:59:25.392 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-09-16T07:59:25.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.501+0000 7fe7a18e3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T07:59:25.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.508+0000 7fe7a18e3640 1 Processor -- start 2024-09-16T07:59:25.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.508+0000 7fe7a18e3640 1 -- start start 2024-09-16T07:59:25.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.509+0000 7fe7a18e3640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe79c132520 0x7fe79c132930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:59:25.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.509+0000 7fe7a18e3640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe79c06f6d0 con 0x7fe79c0a3450 2024-09-16T07:59:25.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.509+0000 7fe7a18e3640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 -- 0x7fe79c06f3d0 con 0x7fe79c132520 2024-09-16T07:59:25.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.509+0000 7fe79affd640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe79c132520 0x7fe79c132930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:59:25.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.509+0000 7fe79affd640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe79c132520 0x7fe79c132930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40438/0 (socket says 172.21.15.27:40438) 2024-09-16T07:59:25.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.509+0000 7fe79a7fc640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fe79c0a3450 0x7fe79c0a3860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:36124/0 (socket says 172.21.15.27:36124) 2024-09-16T07:59:25.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.509+0000 7fe79affd640 1 -- 172.21.15.27:0/250394753 learned_addr learned my addr 172.21.15.27:0/250394753 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-09-16T07:59:25.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.510+0000 7fe79affd640 1 -- 172.21.15.27:0/250394753 >> v1:172.21.15.27:6789/0 conn(0x7fe79c0a3450 legacy=0x7fe79c0a3860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:59:25.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.510+0000 7fe79affd640 1 -- 172.21.15.27:0/250394753 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe79c06f0c0 con 0x7fe79c132520 2024-09-16T07:59:25.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.510+0000 7fe799ffb640 1 -- 172.21.15.27:0/250394753 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 425000971 0 0) 0x7fe79c06f6d0 con 0x7fe79c0a3450 2024-09-16T07:59:25.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.510+0000 7fe79affd640 1 --2- 172.21.15.27:0/250394753 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe79c132520 0x7fe79c132930 crc :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77ba625afc36c25c server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:59:25.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.510+0000 7fe799ffb640 1 -- 172.21.15.27:0/250394753 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fe78c00e040 con 0x7fe79c132520 2024-09-16T07:59:25.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.510+0000 7fe799ffb640 1 -- 172.21.15.27:0/250394753 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7fe78c0091e0 con 0x7fe79c132520 2024-09-16T07:59:25.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.510+0000 7fe799ffb640 1 -- 172.21.15.27:0/250394753 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fe78c007730 con 0x7fe79c132520 2024-09-16T07:59:25.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.511+0000 7fe7a18e3640 1 -- 172.21.15.27:0/250394753 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe79c132520 msgr2=0x7fe79c132930 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:59:25.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.511+0000 7fe7a18e3640 1 --2- 172.21.15.27:0/250394753 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe79c132520 0x7fe79c132930 crc :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:59:25.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.511+0000 7fe7a18e3640 1 -- 172.21.15.27:0/250394753 shutdown_connections 2024-09-16T07:59:25.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.511+0000 7fe7a18e3640 1 --2- 172.21.15.27:0/250394753 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe79c132520 0x7fe79c132930 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:59:25.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.511+0000 7fe7a18e3640 1 -- 172.21.15.27:0/250394753 >> 172.21.15.27:0/250394753 conn(0x7fe79c06e490 msgr2=0x7fe79c06e8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:59:25.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.511+0000 7fe7a18e3640 1 -- 172.21.15.27:0/250394753 shutdown_connections 2024-09-16T07:59:25.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.511+0000 7fe7a18e3640 1 -- 172.21.15.27:0/250394753 wait complete. 2024-09-16T07:59:25.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.511+0000 7fe7a18e3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T07:59:25.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.513+0000 7fe7a18e3640 1 Processor -- start 2024-09-16T07:59:25.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.513+0000 7fe7a18e3640 1 -- start start 2024-09-16T07:59:25.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.515+0000 7fe7a18e3640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe79c0a3450 0x7fe79c120f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:59:25.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.515+0000 7fe7a18e3640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fe79c1320e0 con 0x7fe79c0a3450 2024-09-16T07:59:25.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.515+0000 7fe79affd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe79c0a3450 0x7fe79c120f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:59:25.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.515+0000 7fe79affd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe79c0a3450 0x7fe79c120f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40454/0 (socket says 172.21.15.27:40454) 2024-09-16T07:59:25.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.515+0000 7fe79affd640 1 -- 172.21.15.27:0/560290173 learned_addr learned my addr 172.21.15.27:0/560290173 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-09-16T07:59:25.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.515+0000 7fe79affd640 1 -- 172.21.15.27:0/560290173 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe79c1219c0 con 0x7fe79c0a3450 2024-09-16T07:59:25.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.515+0000 7fe79affd640 1 --2- 172.21.15.27:0/560290173 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe79c0a3450 0x7fe79c120f90 crc :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:59:25.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.515+0000 7fe793fff640 1 -- 172.21.15.27:0/560290173 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fe78c00e040 con 0x7fe79c0a3450 2024-09-16T07:59:25.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.515+0000 7fe793fff640 1 -- 172.21.15.27:0/560290173 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7fe78c0128b0 con 0x7fe79c0a3450 2024-09-16T07:59:25.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.515+0000 7fe7a18e3640 1 -- 172.21.15.27:0/560290173 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe79c11f8e0 con 0x7fe79c0a3450 2024-09-16T07:59:25.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.515+0000 7fe793fff640 1 -- 172.21.15.27:0/560290173 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fe78c012b50 con 0x7fe79c0a3450 2024-09-16T07:59:25.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.516+0000 7fe7a18e3640 1 -- 172.21.15.27:0/560290173 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe79c135790 con 0x7fe79c0a3450 2024-09-16T07:59:25.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.516+0000 7fe7a18e3640 1 -- 172.21.15.27:0/560290173 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe79c06a300 con 0x7fe79c0a3450 2024-09-16T07:59:25.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.516+0000 7fe793fff640 1 -- 172.21.15.27:0/560290173 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) ==== 50954+0+0 (crc 0 0 0) 0x7fe78c012d00 con 0x7fe79c0a3450 2024-09-16T07:59:25.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.517+0000 7fe793fff640 1 --2- 172.21.15.27:0/560290173 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7fe77c03e930 0x7fe77c040e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:59:25.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.517+0000 7fe793fff640 1 -- 172.21.15.27:0/560290173 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(228..228 src has 1..228) ==== 2700+0+0 (crc 0 0 0) 0x7fe78c050c70 con 0x7fe79c0a3450 2024-09-16T07:59:25.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.517+0000 7fe79a7fc640 1 --2- 172.21.15.27:0/560290173 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7fe77c03e930 0x7fe77c040e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:59:25.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.518+0000 7fe79a7fc640 1 --2- 172.21.15.27:0/560290173 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7fe77c03e930 0x7fe77c040e40 crc :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:59:25.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.522+0000 7fe793fff640 1 -- 172.21.15.27:0/560290173 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (crc 0 0 0) 0x7fe78c014570 con 0x7fe79c0a3450 2024-09-16T07:59:25.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.706+0000 7fe7a18e3640 1 -- 172.21.15.27:0/560290173 --> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) -- 0x7fe79c121850 con 0x7fe77c03e930 2024-09-16T07:59:25.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.707+0000 7fe793fff640 1 -- 172.21.15.27:0/560290173 <== mgr.4102 v2:172.21.15.187:6800/4038049593 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) ==== 41+0+0 (crc 0 0 0) 0x7fe79c121850 con 0x7fe77c03e930 2024-09-16T07:59:25.705 INFO:teuthology.orchestra.run.smithi027.stderr:instructed osd(s) 0 to deep-scrub 2024-09-16T07:59:25.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.709+0000 7fe7a18e3640 1 -- 172.21.15.27:0/560290173 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7fe77c03e930 msgr2=0x7fe77c040e40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:59:25.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.709+0000 7fe7a18e3640 1 --2- 172.21.15.27:0/560290173 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7fe77c03e930 0x7fe77c040e40 crc :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:59:25.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.709+0000 7fe7a18e3640 1 -- 172.21.15.27:0/560290173 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe79c0a3450 msgr2=0x7fe79c120f90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:59:25.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.709+0000 7fe7a18e3640 1 --2- 172.21.15.27:0/560290173 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe79c0a3450 0x7fe79c120f90 crc :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:59:25.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.709+0000 7fe7a18e3640 1 -- 172.21.15.27:0/560290173 shutdown_connections 2024-09-16T07:59:25.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.709+0000 7fe7a18e3640 1 --2- 172.21.15.27:0/560290173 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7fe77c03e930 0x7fe77c040e40 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:59:25.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.709+0000 7fe7a18e3640 1 --2- 172.21.15.27:0/560290173 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe79c0a3450 0x7fe79c120f90 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:59:25.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.709+0000 7fe7a18e3640 1 -- 172.21.15.27:0/560290173 >> 172.21.15.27:0/560290173 conn(0x7fe79c06e490 msgr2=0x7fe79c0a5840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:59:25.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.709+0000 7fe7a18e3640 1 -- 172.21.15.27:0/560290173 shutdown_connections 2024-09-16T07:59:25.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.709+0000 7fe7a18e3640 1 -- 172.21.15.27:0/560290173 wait complete. 2024-09-16T07:59:25.721 INFO:tasks.ceph:Scrubbing osd.1 2024-09-16T07:59:25.721 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-09-16T07:59:25.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.836+0000 7ff1074ac640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T07:59:25.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.846+0000 7ff1074ac640 1 Processor -- start 2024-09-16T07:59:25.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.846+0000 7ff1074ac640 1 -- start start 2024-09-16T07:59:25.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.847+0000 7ff1074ac640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ff100132520 0x7ff100132930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:59:25.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.847+0000 7ff1074ac640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff10006f6d0 con 0x7ff1000a3450 2024-09-16T07:59:25.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.847+0000 7ff1074ac640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 -- 0x7ff10006f3d0 con 0x7ff100132520 2024-09-16T07:59:25.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.848+0000 7ff105248640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ff100132520 0x7ff100132930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:59:25.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.848+0000 7ff105248640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ff100132520 0x7ff100132930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40460/0 (socket says 172.21.15.27:40460) 2024-09-16T07:59:25.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.848+0000 7ff104a47640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7ff1000a3450 0x7ff1000a3860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:36128/0 (socket says 172.21.15.27:36128) 2024-09-16T07:59:25.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.848+0000 7ff105248640 1 -- 172.21.15.27:0/1219593805 learned_addr learned my addr 172.21.15.27:0/1219593805 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-09-16T07:59:25.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.848+0000 7ff105248640 1 -- 172.21.15.27:0/1219593805 >> v1:172.21.15.27:6789/0 conn(0x7ff1000a3450 legacy=0x7ff1000a3860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:59:25.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.848+0000 7ff105248640 1 -- 172.21.15.27:0/1219593805 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff10006f0c0 con 0x7ff100132520 2024-09-16T07:59:25.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.848+0000 7ff105248640 1 --2- 172.21.15.27:0/1219593805 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff100132520 0x7ff100132930 crc :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa7a3e5d7543b297 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:59:25.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.848+0000 7ff0f7fff640 1 -- 172.21.15.27:0/1219593805 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7ff0fc009e50 con 0x7ff100132520 2024-09-16T07:59:25.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.848+0000 7ff0f7fff640 1 -- 172.21.15.27:0/1219593805 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7ff0fc00e230 con 0x7ff100132520 2024-09-16T07:59:25.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.848+0000 7ff0f7fff640 1 -- 172.21.15.27:0/1219593805 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7ff0fc00e4d0 con 0x7ff100132520 2024-09-16T07:59:25.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.849+0000 7ff1074ac640 1 -- 172.21.15.27:0/1219593805 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff100132520 msgr2=0x7ff100132930 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:59:25.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.849+0000 7ff1074ac640 1 --2- 172.21.15.27:0/1219593805 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff100132520 0x7ff100132930 crc :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:59:25.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.849+0000 7ff1074ac640 1 -- 172.21.15.27:0/1219593805 shutdown_connections 2024-09-16T07:59:25.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.849+0000 7ff1074ac640 1 --2- 172.21.15.27:0/1219593805 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff100132520 0x7ff100132930 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:59:25.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.849+0000 7ff1074ac640 1 -- 172.21.15.27:0/1219593805 >> 172.21.15.27:0/1219593805 conn(0x7ff10006e490 msgr2=0x7ff10006e8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:59:25.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.849+0000 7ff1074ac640 1 -- 172.21.15.27:0/1219593805 shutdown_connections 2024-09-16T07:59:25.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.849+0000 7ff1074ac640 1 -- 172.21.15.27:0/1219593805 wait complete. 2024-09-16T07:59:25.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.849+0000 7ff1074ac640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T07:59:25.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.851+0000 7ff1074ac640 1 Processor -- start 2024-09-16T07:59:25.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.851+0000 7ff1074ac640 1 -- start start 2024-09-16T07:59:25.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.853+0000 7ff1074ac640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1000a3450 0x7ff1001ca510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:59:25.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.853+0000 7ff1074ac640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7ff100122eb0 con 0x7ff1000a3450 2024-09-16T07:59:25.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.853+0000 7ff105248640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1000a3450 0x7ff1001ca510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:59:25.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.853+0000 7ff105248640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1000a3450 0x7ff1001ca510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40464/0 (socket says 172.21.15.27:40464) 2024-09-16T07:59:25.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.853+0000 7ff105248640 1 -- 172.21.15.27:0/994029138 learned_addr learned my addr 172.21.15.27:0/994029138 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-09-16T07:59:25.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.853+0000 7ff105248640 1 -- 172.21.15.27:0/994029138 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1001253a0 con 0x7ff1000a3450 2024-09-16T07:59:25.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.853+0000 7ff105248640 1 --2- 172.21.15.27:0/994029138 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1000a3450 0x7ff1001ca510 crc :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:59:25.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.853+0000 7ff0f5ffb640 1 -- 172.21.15.27:0/994029138 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7ff0fc00edc0 con 0x7ff1000a3450 2024-09-16T07:59:25.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.853+0000 7ff0f5ffb640 1 -- 172.21.15.27:0/994029138 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7ff0fc015420 con 0x7ff1000a3450 2024-09-16T07:59:25.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.853+0000 7ff1074ac640 1 -- 172.21.15.27:0/994029138 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1001225c0 con 0x7ff1000a3450 2024-09-16T07:59:25.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.853+0000 7ff0f5ffb640 1 -- 172.21.15.27:0/994029138 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7ff0fc0156c0 con 0x7ff1000a3450 2024-09-16T07:59:25.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.854+0000 7ff1074ac640 1 -- 172.21.15.27:0/994029138 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1001250c0 con 0x7ff1000a3450 2024-09-16T07:59:25.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.854+0000 7ff1074ac640 1 -- 172.21.15.27:0/994029138 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7ff1000a3260 con 0x7ff1000a3450 2024-09-16T07:59:25.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.855+0000 7ff0f5ffb640 1 -- 172.21.15.27:0/994029138 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) ==== 50954+0+0 (crc 0 0 0) 0x7ff0fc015870 con 0x7ff1000a3450 2024-09-16T07:59:25.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.855+0000 7ff0f5ffb640 1 --2- 172.21.15.27:0/994029138 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7ff0e403e930 0x7ff0e4040e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:59:25.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.855+0000 7ff0f5ffb640 1 -- 172.21.15.27:0/994029138 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(228..228 src has 1..228) ==== 2700+0+0 (crc 0 0 0) 0x7ff0fc050e10 con 0x7ff1000a3450 2024-09-16T07:59:25.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.855+0000 7ff0f5ffb640 1 --2- 172.21.15.27:0/994029138 >> v2:172.21.15.27:6800/584480168 conn(0x7ff0e4043930 0x7ff0e4045de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:59:25.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.855+0000 7ff0f5ffb640 1 -- 172.21.15.27:0/994029138 --> v2:172.21.15.27:6800/584480168 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff0e4046470 con 0x7ff0e4043930 2024-09-16T07:59:25.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.855+0000 7ff0f5ffb640 1 -- 172.21.15.27:0/994029138 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_get_version_reply(handle=1 version=228) ==== 24+0+0 (crc 0 0 0) 0x7ff0fc053300 con 0x7ff1000a3450 2024-09-16T07:59:25.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.855+0000 7ff104a47640 1 --2- 172.21.15.27:0/994029138 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7ff0e403e930 0x7ff0e4040e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:59:25.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.856+0000 7ff105a49640 1 --2- 172.21.15.27:0/994029138 >> v2:172.21.15.27:6800/584480168 conn(0x7ff0e4043930 0x7ff0e4045de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-09-16T07:59:25.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.856+0000 7ff104a47640 1 --2- 172.21.15.27:0/994029138 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7ff0e403e930 0x7ff0e4040e40 crc :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:59:25.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.860+0000 7ff105a49640 1 --2- 172.21.15.27:0/994029138 >> v2:172.21.15.27:6800/584480168 conn(0x7ff0e4043930 0x7ff0e4045de0 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:59:25.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.861+0000 7ff0f5ffb640 1 -- 172.21.15.27:0/994029138 <== osd.1 v2:172.21.15.27:6800/584480168 1 ==== command_reply(tid 1: 0 ) ==== 8+0+5790 (crc 0 0 0) 0x7ff0e4046470 con 0x7ff0e4043930 2024-09-16T07:59:25.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.865+0000 7ff1074ac640 1 -- 172.21.15.27:0/994029138 --> v2:172.21.15.27:6800/584480168 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7ff10009c830 con 0x7ff0e4043930 2024-09-16T07:59:25.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.866+0000 7ff0f5ffb640 1 -- 172.21.15.27:0/994029138 <== osd.1 v2:172.21.15.27:6800/584480168 2 ==== command_reply(tid 2: 0 ) ==== 8+0+22 (crc 0 0 0) 0x7ff0f800a820 con 0x7ff0e4043930 2024-09-16T07:59:25.865 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-09-16T07:59:25.865 INFO:teuthology.orchestra.run.smithi027.stdout: "success": "" 2024-09-16T07:59:25.865 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-09-16T07:59:25.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.866+0000 7ff1074ac640 1 -- 172.21.15.27:0/994029138 >> v2:172.21.15.27:6800/584480168 conn(0x7ff0e4043930 msgr2=0x7ff0e4045de0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:59:25.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.866+0000 7ff1074ac640 1 --2- 172.21.15.27:0/994029138 >> v2:172.21.15.27:6800/584480168 conn(0x7ff0e4043930 0x7ff0e4045de0 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:59:25.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.867+0000 7ff1074ac640 1 -- 172.21.15.27:0/994029138 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7ff0e403e930 msgr2=0x7ff0e4040e40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:59:25.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.867+0000 7ff1074ac640 1 --2- 172.21.15.27:0/994029138 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7ff0e403e930 0x7ff0e4040e40 crc :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:59:25.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.867+0000 7ff1074ac640 1 -- 172.21.15.27:0/994029138 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1000a3450 msgr2=0x7ff1001ca510 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:59:25.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.867+0000 7ff1074ac640 1 --2- 172.21.15.27:0/994029138 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1000a3450 0x7ff1001ca510 crc :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:59:25.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.867+0000 7ff1074ac640 1 -- 172.21.15.27:0/994029138 shutdown_connections 2024-09-16T07:59:25.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.867+0000 7ff1074ac640 1 --2- 172.21.15.27:0/994029138 >> v2:172.21.15.27:6800/584480168 conn(0x7ff0e4043930 0x7ff0e4045de0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:59:25.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.867+0000 7ff1074ac640 1 --2- 172.21.15.27:0/994029138 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7ff0e403e930 0x7ff0e4040e40 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:59:25.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.867+0000 7ff1074ac640 1 --2- 172.21.15.27:0/994029138 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1000a3450 0x7ff1001ca510 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:59:25.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.867+0000 7ff1074ac640 1 -- 172.21.15.27:0/994029138 >> 172.21.15.27:0/994029138 conn(0x7ff10006e490 msgr2=0x7ff100135bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:59:25.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.867+0000 7ff1074ac640 1 -- 172.21.15.27:0/994029138 shutdown_connections 2024-09-16T07:59:25.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.867+0000 7ff1074ac640 1 -- 172.21.15.27:0/994029138 wait complete. 2024-09-16T07:59:25.877 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-09-16T07:59:25.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.989+0000 7f5661b91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T07:59:25.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.996+0000 7f5661b91640 1 Processor -- start 2024-09-16T07:59:25.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.996+0000 7f5661b91640 1 -- start start 2024-09-16T07:59:25.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.997+0000 7f5661b91640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f565c131300 0x7f565c131710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:59:25.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.997+0000 7f5661b91640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f565c09ce60 con 0x7f565c132030 2024-09-16T07:59:25.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.997+0000 7f5661b91640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 -- 0x7f565c09e010 con 0x7f565c131300 2024-09-16T07:59:25.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.998+0000 7f565b7fe640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f565c131300 0x7f565c131710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:59:25.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.998+0000 7f5653fff640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f565c132030 0x7f565c132440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:36134/0 (socket says 172.21.15.27:36134) 2024-09-16T07:59:25.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.998+0000 7f5653fff640 1 -- 172.21.15.27:0/102657755 learned_addr learned my addr 172.21.15.27:0/102657755 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-09-16T07:59:25.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.998+0000 7f565b7fe640 1 -- 172.21.15.27:0/102657755 >> v1:172.21.15.27:6789/0 conn(0x7f565c132030 legacy=0x7f565c132440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:59:25.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.998+0000 7f565b7fe640 1 -- 172.21.15.27:0/102657755 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f565c0a0aa0 con 0x7f565c131300 2024-09-16T07:59:25.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.998+0000 7f565b7fe640 1 --2- 172.21.15.27:0/102657755 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f565c131300 0x7f565c131710 crc :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8311d5308341c382 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:59:25.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.998+0000 7f565affd640 1 -- 172.21.15.27:0/102657755 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f564c009d60 con 0x7f565c131300 2024-09-16T07:59:25.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.998+0000 7f565affd640 1 -- 172.21.15.27:0/102657755 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f564c00e230 con 0x7f565c131300 2024-09-16T07:59:25.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.998+0000 7f565affd640 1 -- 172.21.15.27:0/102657755 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f564c00e4d0 con 0x7f565c131300 2024-09-16T07:59:26.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.999+0000 7f5661b91640 1 -- 172.21.15.27:0/102657755 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f565c131300 msgr2=0x7f565c131710 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:59:26.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.999+0000 7f5661b91640 1 --2- 172.21.15.27:0/102657755 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f565c131300 0x7f565c131710 crc :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:59:26.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.999+0000 7f5661b91640 1 -- 172.21.15.27:0/102657755 shutdown_connections 2024-09-16T07:59:26.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.999+0000 7f5661b91640 1 --2- 172.21.15.27:0/102657755 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f565c131300 0x7f565c131710 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:59:26.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.999+0000 7f5661b91640 1 -- 172.21.15.27:0/102657755 >> 172.21.15.27:0/102657755 conn(0x7f565c069d10 msgr2=0x7f565c09f200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:59:26.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.999+0000 7f5661b91640 1 -- 172.21.15.27:0/102657755 shutdown_connections 2024-09-16T07:59:26.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.999+0000 7f5661b91640 1 -- 172.21.15.27:0/102657755 wait complete. 2024-09-16T07:59:26.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:25.999+0000 7f5661b91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T07:59:26.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.001+0000 7f5661b91640 1 Processor -- start 2024-09-16T07:59:26.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.001+0000 7f5661b91640 1 -- start start 2024-09-16T07:59:26.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.003+0000 7f5661b91640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f565c131300 0x7f565c11f180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:59:26.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.003+0000 7f5661b91640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f565c073d90 con 0x7f565c131300 2024-09-16T07:59:26.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.003+0000 7f565b7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f565c131300 0x7f565c11f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:59:26.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.003+0000 7f565b7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f565c131300 0x7f565c11f180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40492/0 (socket says 172.21.15.27:40492) 2024-09-16T07:59:26.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.003+0000 7f565b7fe640 1 -- 172.21.15.27:0/471854979 learned_addr learned my addr 172.21.15.27:0/471854979 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-09-16T07:59:26.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.003+0000 7f565b7fe640 1 -- 172.21.15.27:0/471854979 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f565c121140 con 0x7f565c131300 2024-09-16T07:59:26.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.003+0000 7f565b7fe640 1 --2- 172.21.15.27:0/471854979 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f565c131300 0x7f565c11f180 crc :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:59:26.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.003+0000 7f5658ff9640 1 -- 172.21.15.27:0/471854979 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f564c014620 con 0x7f565c131300 2024-09-16T07:59:26.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.003+0000 7f5658ff9640 1 -- 172.21.15.27:0/471854979 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f564c014780 con 0x7f565c131300 2024-09-16T07:59:26.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.003+0000 7f5661b91640 1 -- 172.21.15.27:0/471854979 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f565c1213e0 con 0x7f565c131300 2024-09-16T07:59:26.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.003+0000 7f5658ff9640 1 -- 172.21.15.27:0/471854979 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f564c014a20 con 0x7f565c131300 2024-09-16T07:59:26.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.003+0000 7f5661b91640 1 -- 172.21.15.27:0/471854979 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f565c120440 con 0x7f565c131300 2024-09-16T07:59:26.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.004+0000 7f5661b91640 1 -- 172.21.15.27:0/471854979 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f565c130950 con 0x7f565c131300 2024-09-16T07:59:26.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.004+0000 7f5658ff9640 1 -- 172.21.15.27:0/471854979 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) ==== 50954+0+0 (crc 0 0 0) 0x7f564c014c20 con 0x7f565c131300 2024-09-16T07:59:26.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.005+0000 7f5658ff9640 1 --2- 172.21.15.27:0/471854979 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7f563c03e8e0 0x7f563c040df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:59:26.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.005+0000 7f5658ff9640 1 -- 172.21.15.27:0/471854979 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(228..228 src has 1..228) ==== 2700+0+0 (crc 0 0 0) 0x7f564c019040 con 0x7f565c131300 2024-09-16T07:59:26.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.005+0000 7f5653fff640 1 --2- 172.21.15.27:0/471854979 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7f563c03e8e0 0x7f563c040df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:59:26.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.006+0000 7f5653fff640 1 --2- 172.21.15.27:0/471854979 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7f563c03e8e0 0x7f563c040df0 crc :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:59:26.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.010+0000 7f5658ff9640 1 -- 172.21.15.27:0/471854979 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (crc 0 0 0) 0x7f564c011570 con 0x7f565c131300 2024-09-16T07:59:26.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.195+0000 7f5661b91640 1 -- 172.21.15.27:0/471854979 --> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) -- 0x7f565c074b50 con 0x7f563c03e8e0 2024-09-16T07:59:26.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.195+0000 7f5658ff9640 1 -- 172.21.15.27:0/471854979 <== mgr.4102 v2:172.21.15.187:6800/4038049593 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) ==== 41+0+0 (crc 0 0 0) 0x7f5644005930 con 0x7f563c03e8e0 2024-09-16T07:59:26.194 INFO:teuthology.orchestra.run.smithi027.stderr:instructed osd(s) 1 to deep-scrub 2024-09-16T07:59:26.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.198+0000 7f5661b91640 1 -- 172.21.15.27:0/471854979 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7f563c03e8e0 msgr2=0x7f563c040df0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:59:26.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.198+0000 7f5661b91640 1 --2- 172.21.15.27:0/471854979 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7f563c03e8e0 0x7f563c040df0 crc :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:59:26.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.198+0000 7f5661b91640 1 -- 172.21.15.27:0/471854979 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f565c131300 msgr2=0x7f565c11f180 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:59:26.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.198+0000 7f5661b91640 1 --2- 172.21.15.27:0/471854979 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f565c131300 0x7f565c11f180 crc :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:59:26.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.198+0000 7f5661b91640 1 -- 172.21.15.27:0/471854979 shutdown_connections 2024-09-16T07:59:26.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.198+0000 7f5661b91640 1 --2- 172.21.15.27:0/471854979 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7f563c03e8e0 0x7f563c040df0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:59:26.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.198+0000 7f5661b91640 1 --2- 172.21.15.27:0/471854979 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f565c131300 0x7f565c11f180 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:59:26.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.198+0000 7f5661b91640 1 -- 172.21.15.27:0/471854979 >> 172.21.15.27:0/471854979 conn(0x7f565c069d10 msgr2=0x7f565c1ba110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:59:26.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.198+0000 7f5661b91640 1 -- 172.21.15.27:0/471854979 shutdown_connections 2024-09-16T07:59:26.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.198+0000 7f5661b91640 1 -- 172.21.15.27:0/471854979 wait complete. 2024-09-16T07:59:26.214 INFO:tasks.ceph:Scrubbing osd.2 2024-09-16T07:59:26.214 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-09-16T07:59:26.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.325+0000 7f82a6446640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T07:59:26.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.335+0000 7f82a6446640 1 Processor -- start 2024-09-16T07:59:26.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.335+0000 7f82a6446640 1 -- start start 2024-09-16T07:59:26.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.337+0000 7f82a6446640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f82a0138720 0x7f82a013ab40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:59:26.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.337+0000 7f82a6446640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f82a0133040 con 0x7f82a0132520 2024-09-16T07:59:26.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.337+0000 7f82a6446640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 -- 0x7f82a006f6d0 con 0x7f82a0138720 2024-09-16T07:59:26.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.337+0000 7f8297fff640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f82a0138720 0x7f82a013ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:59:26.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.337+0000 7f8297fff640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f82a0138720 0x7f82a013ab40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40502/0 (socket says 172.21.15.27:40502) 2024-09-16T07:59:26.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.337+0000 7f829ffff640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f82a0132520 0x7f82a0132930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:36140/0 (socket says 172.21.15.27:36140) 2024-09-16T07:59:26.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.337+0000 7f8297fff640 1 -- 172.21.15.27:0/438528576 learned_addr learned my addr 172.21.15.27:0/438528576 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-09-16T07:59:26.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.337+0000 7f8297fff640 1 -- 172.21.15.27:0/438528576 >> v1:172.21.15.27:6789/0 conn(0x7f82a0132520 legacy=0x7f82a0132930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:59:26.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.337+0000 7f8297fff640 1 -- 172.21.15.27:0/438528576 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82a006f3d0 con 0x7f82a0138720 2024-09-16T07:59:26.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.337+0000 7f8297fff640 1 --2- 172.21.15.27:0/438528576 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f82a0138720 0x7f82a013ab40 crc :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93ec9d9b570e84ba server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:59:26.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.337+0000 7f829f7fe640 1 -- 172.21.15.27:0/438528576 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f8288009e50 con 0x7f82a0138720 2024-09-16T07:59:26.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.337+0000 7f829f7fe640 1 -- 172.21.15.27:0/438528576 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f828800e230 con 0x7f82a0138720 2024-09-16T07:59:26.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.337+0000 7f829f7fe640 1 -- 172.21.15.27:0/438528576 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f828800e4d0 con 0x7f82a0138720 2024-09-16T07:59:26.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.338+0000 7f82a6446640 1 -- 172.21.15.27:0/438528576 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f82a0138720 msgr2=0x7f82a013ab40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:59:26.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.338+0000 7f82a6446640 1 --2- 172.21.15.27:0/438528576 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f82a0138720 0x7f82a013ab40 crc :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:59:26.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.338+0000 7f82a6446640 1 -- 172.21.15.27:0/438528576 shutdown_connections 2024-09-16T07:59:26.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.338+0000 7f82a6446640 1 --2- 172.21.15.27:0/438528576 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f82a0138720 0x7f82a013ab40 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:59:26.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.338+0000 7f82a6446640 1 -- 172.21.15.27:0/438528576 >> 172.21.15.27:0/438528576 conn(0x7f82a006e490 msgr2=0x7f82a006e8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:59:26.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.338+0000 7f82a6446640 1 -- 172.21.15.27:0/438528576 shutdown_connections 2024-09-16T07:59:26.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.338+0000 7f82a6446640 1 -- 172.21.15.27:0/438528576 wait complete. 2024-09-16T07:59:26.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.339+0000 7f82a6446640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T07:59:26.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.341+0000 7f82a6446640 1 Processor -- start 2024-09-16T07:59:26.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.341+0000 7f82a6446640 1 -- start start 2024-09-16T07:59:26.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.342+0000 7f82a6446640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f82a0132520 0x7f82a011f610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:59:26.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.342+0000 7f82a6446640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f82a0132100 con 0x7f82a0132520 2024-09-16T07:59:26.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.342+0000 7f829ffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f82a0132520 0x7f82a011f610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:59:26.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.342+0000 7f829ffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f82a0132520 0x7f82a011f610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40518/0 (socket says 172.21.15.27:40518) 2024-09-16T07:59:26.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.342+0000 7f829ffff640 1 -- 172.21.15.27:0/3111700528 learned_addr learned my addr 172.21.15.27:0/3111700528 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-09-16T07:59:26.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.343+0000 7f829ffff640 1 -- 172.21.15.27:0/3111700528 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82a011f250 con 0x7f82a0132520 2024-09-16T07:59:26.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.343+0000 7f829ffff640 1 --2- 172.21.15.27:0/3111700528 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f82a0132520 0x7f82a011f610 crc :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:59:26.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.343+0000 7f829d7fa640 1 -- 172.21.15.27:0/3111700528 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f82900059a0 con 0x7f82a0132520 2024-09-16T07:59:26.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.343+0000 7f829d7fa640 1 -- 172.21.15.27:0/3111700528 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f8290005b00 con 0x7f82a0132520 2024-09-16T07:59:26.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.343+0000 7f82a6446640 1 -- 172.21.15.27:0/3111700528 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82a0120ea0 con 0x7f82a0132520 2024-09-16T07:59:26.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.343+0000 7f829d7fa640 1 -- 172.21.15.27:0/3111700528 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f8290005da0 con 0x7f82a0132520 2024-09-16T07:59:26.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.343+0000 7f82a6446640 1 -- 172.21.15.27:0/3111700528 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82a011eeb0 con 0x7f82a0132520 2024-09-16T07:59:26.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.344+0000 7f82a6446640 1 -- 172.21.15.27:0/3111700528 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f82a009c830 con 0x7f82a0132520 2024-09-16T07:59:26.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.344+0000 7f829d7fa640 1 -- 172.21.15.27:0/3111700528 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) ==== 50954+0+0 (crc 0 0 0) 0x7f82900070b0 con 0x7f82a0132520 2024-09-16T07:59:26.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.344+0000 7f829d7fa640 1 --2- 172.21.15.27:0/3111700528 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7f828003e930 0x7f8280040e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:59:26.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.344+0000 7f829d7fa640 1 -- 172.21.15.27:0/3111700528 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(228..228 src has 1..228) ==== 2700+0+0 (crc 0 0 0) 0x7f829004f4c0 con 0x7f82a0132520 2024-09-16T07:59:26.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.344+0000 7f829d7fa640 1 --2- 172.21.15.27:0/3111700528 >> v2:172.21.15.187:6809/4110688419 conn(0x7f8280043930 0x7f8280045de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:59:26.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.344+0000 7f829d7fa640 1 -- 172.21.15.27:0/3111700528 --> v2:172.21.15.187:6809/4110688419 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8280046470 con 0x7f8280043930 2024-09-16T07:59:26.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.344+0000 7f829d7fa640 1 -- 172.21.15.27:0/3111700528 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_get_version_reply(handle=1 version=228) ==== 24+0+0 (crc 0 0 0) 0x7f82900519b0 con 0x7f82a0132520 2024-09-16T07:59:26.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.345+0000 7f8297fff640 1 --2- 172.21.15.27:0/3111700528 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7f828003e930 0x7f8280040e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:59:26.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.346+0000 7f8297fff640 1 --2- 172.21.15.27:0/3111700528 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7f828003e930 0x7f8280040e40 crc :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:59:26.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.346+0000 7f82a49e3640 1 --2- 172.21.15.27:0/3111700528 >> v2:172.21.15.187:6809/4110688419 conn(0x7f8280043930 0x7f8280045de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-09-16T07:59:26.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.350+0000 7f82a49e3640 1 --2- 172.21.15.27:0/3111700528 >> v2:172.21.15.187:6809/4110688419 conn(0x7f8280043930 0x7f8280045de0 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:59:26.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.352+0000 7f829d7fa640 1 -- 172.21.15.27:0/3111700528 <== osd.2 v2:172.21.15.187:6809/4110688419 1 ==== command_reply(tid 1: 0 ) ==== 8+0+5790 (crc 0 0 0) 0x7f8280046470 con 0x7f8280043930 2024-09-16T07:59:26.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.356+0000 7f82a6446640 1 -- 172.21.15.27:0/3111700528 --> v2:172.21.15.187:6809/4110688419 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f82a009d380 con 0x7f8280043930 2024-09-16T07:59:26.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.358+0000 7f829d7fa640 1 -- 172.21.15.27:0/3111700528 <== osd.2 v2:172.21.15.187:6809/4110688419 2 ==== command_reply(tid 2: 0 ) ==== 8+0+22 (crc 0 0 0) 0x7f829800a820 con 0x7f8280043930 2024-09-16T07:59:26.356 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-09-16T07:59:26.356 INFO:teuthology.orchestra.run.smithi027.stdout: "success": "" 2024-09-16T07:59:26.356 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-09-16T07:59:26.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.358+0000 7f82a6446640 1 -- 172.21.15.27:0/3111700528 >> v2:172.21.15.187:6809/4110688419 conn(0x7f8280043930 msgr2=0x7f8280045de0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:59:26.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.358+0000 7f82a6446640 1 --2- 172.21.15.27:0/3111700528 >> v2:172.21.15.187:6809/4110688419 conn(0x7f8280043930 0x7f8280045de0 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:59:26.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.358+0000 7f82a6446640 1 -- 172.21.15.27:0/3111700528 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7f828003e930 msgr2=0x7f8280040e40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:59:26.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.358+0000 7f82a6446640 1 --2- 172.21.15.27:0/3111700528 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7f828003e930 0x7f8280040e40 crc :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:59:26.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.358+0000 7f82a6446640 1 -- 172.21.15.27:0/3111700528 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f82a0132520 msgr2=0x7f82a011f610 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:59:26.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.358+0000 7f82a6446640 1 --2- 172.21.15.27:0/3111700528 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f82a0132520 0x7f82a011f610 crc :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:59:26.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.358+0000 7f82a6446640 1 -- 172.21.15.27:0/3111700528 shutdown_connections 2024-09-16T07:59:26.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.358+0000 7f82a6446640 1 --2- 172.21.15.27:0/3111700528 >> v2:172.21.15.187:6809/4110688419 conn(0x7f8280043930 0x7f8280045de0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:59:26.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.358+0000 7f82a6446640 1 --2- 172.21.15.27:0/3111700528 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7f828003e930 0x7f8280040e40 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:59:26.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.358+0000 7f82a6446640 1 --2- 172.21.15.27:0/3111700528 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f82a0132520 0x7f82a011f610 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:59:26.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.358+0000 7f82a6446640 1 -- 172.21.15.27:0/3111700528 >> 172.21.15.27:0/3111700528 conn(0x7f82a006e490 msgr2=0x7f82a00a3200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:59:26.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.358+0000 7f82a6446640 1 -- 172.21.15.27:0/3111700528 shutdown_connections 2024-09-16T07:59:26.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.358+0000 7f82a6446640 1 -- 172.21.15.27:0/3111700528 wait complete. 2024-09-16T07:59:26.367 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-09-16T07:59:26.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.478+0000 7f05c5791640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T07:59:26.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.484+0000 7f05c5791640 1 Processor -- start 2024-09-16T07:59:26.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.484+0000 7f05c5791640 1 -- start start 2024-09-16T07:59:26.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.485+0000 7f05c5791640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f05c01330b0 0x7f05c01334c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:59:26.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.486+0000 7f05c5791640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f05c0072d70 con 0x7f05c0133a00 2024-09-16T07:59:26.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.486+0000 7f05c5791640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 -- 0x7f05c009bd40 con 0x7f05c01330b0 2024-09-16T07:59:26.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.486+0000 7f05beffd640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f05c01330b0 0x7f05c01334c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:59:26.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.486+0000 7f05beffd640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f05c01330b0 0x7f05c01334c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40530/0 (socket says 172.21.15.27:40530) 2024-09-16T07:59:26.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.486+0000 7f05beffd640 1 -- 172.21.15.27:0/1890231872 learned_addr learned my addr 172.21.15.27:0/1890231872 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-09-16T07:59:26.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.486+0000 7f05be7fc640 1 --1- 172.21.15.27:0/1890231872 >> v1:172.21.15.27:6789/0 conn(0x7f05c0133a00 0x7f05c012fea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:36154/0 (socket says 172.21.15.27:36154) 2024-09-16T07:59:26.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.486+0000 7f05beffd640 1 -- 172.21.15.27:0/1890231872 >> v1:172.21.15.27:6789/0 conn(0x7f05c0133a00 legacy=0x7f05c012fea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:59:26.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.486+0000 7f05beffd640 1 -- 172.21.15.27:0/1890231872 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05c009d3f0 con 0x7f05c01330b0 2024-09-16T07:59:26.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.486+0000 7f05beffd640 1 --2- 172.21.15.27:0/1890231872 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f05c01330b0 0x7f05c01334c0 crc :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55f21201373157b1 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:59:26.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.486+0000 7f05bdffb640 1 -- 172.21.15.27:0/1890231872 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f05b0009e50 con 0x7f05c01330b0 2024-09-16T07:59:26.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.486+0000 7f05bdffb640 1 -- 172.21.15.27:0/1890231872 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f05b000e230 con 0x7f05c01330b0 2024-09-16T07:59:26.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.486+0000 7f05bdffb640 1 -- 172.21.15.27:0/1890231872 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f05b000e4d0 con 0x7f05c01330b0 2024-09-16T07:59:26.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.487+0000 7f05c5791640 1 -- 172.21.15.27:0/1890231872 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f05c01330b0 msgr2=0x7f05c01334c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:59:26.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.487+0000 7f05c5791640 1 --2- 172.21.15.27:0/1890231872 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f05c01330b0 0x7f05c01334c0 crc :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:59:26.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.487+0000 7f05c5791640 1 -- 172.21.15.27:0/1890231872 shutdown_connections 2024-09-16T07:59:26.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.487+0000 7f05c5791640 1 --2- 172.21.15.27:0/1890231872 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f05c01330b0 0x7f05c01334c0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:59:26.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.487+0000 7f05c5791640 1 -- 172.21.15.27:0/1890231872 >> 172.21.15.27:0/1890231872 conn(0x7f05c00a25b0 msgr2=0x7f05c00a29c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:59:26.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.487+0000 7f05c5791640 1 -- 172.21.15.27:0/1890231872 shutdown_connections 2024-09-16T07:59:26.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.487+0000 7f05c5791640 1 -- 172.21.15.27:0/1890231872 wait complete. 2024-09-16T07:59:26.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.487+0000 7f05c5791640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T07:59:26.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.489+0000 7f05c5791640 1 Processor -- start 2024-09-16T07:59:26.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.489+0000 7f05c5791640 1 -- start start 2024-09-16T07:59:26.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.491+0000 7f05c5791640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f05c01330b0 0x7f05c011eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:59:26.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.491+0000 7f05c5791640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f05c0131de0 con 0x7f05c01330b0 2024-09-16T07:59:26.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.491+0000 7f05beffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f05c01330b0 0x7f05c011eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:59:26.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.491+0000 7f05beffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f05c01330b0 0x7f05c011eaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40546/0 (socket says 172.21.15.27:40546) 2024-09-16T07:59:26.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.491+0000 7f05beffd640 1 -- 172.21.15.27:0/284002297 learned_addr learned my addr 172.21.15.27:0/284002297 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-09-16T07:59:26.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.491+0000 7f05beffd640 1 -- 172.21.15.27:0/284002297 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05c0120520 con 0x7f05c01330b0 2024-09-16T07:59:26.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.491+0000 7f05beffd640 1 --2- 172.21.15.27:0/284002297 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f05c01330b0 0x7f05c011eaa0 crc :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:59:26.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.492+0000 7f05a77fe640 1 -- 172.21.15.27:0/284002297 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f05b000ee20 con 0x7f05c01330b0 2024-09-16T07:59:26.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.492+0000 7f05a77fe640 1 -- 172.21.15.27:0/284002297 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f05b0015420 con 0x7f05c01330b0 2024-09-16T07:59:26.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.492+0000 7f05c5791640 1 -- 172.21.15.27:0/284002297 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f05c011fce0 con 0x7f05c01330b0 2024-09-16T07:59:26.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.492+0000 7f05a77fe640 1 -- 172.21.15.27:0/284002297 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f05b00156c0 con 0x7f05c01330b0 2024-09-16T07:59:26.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.492+0000 7f05c5791640 1 -- 172.21.15.27:0/284002297 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f05c01213b0 con 0x7f05c01330b0 2024-09-16T07:59:26.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.493+0000 7f05a77fe640 1 -- 172.21.15.27:0/284002297 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) ==== 50954+0+0 (crc 0 0 0) 0x7f05b0015870 con 0x7f05c01330b0 2024-09-16T07:59:26.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.493+0000 7f05a77fe640 1 --2- 172.21.15.27:0/284002297 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7f05ac03e930 0x7f05ac040e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:59:26.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.493+0000 7f05a77fe640 1 -- 172.21.15.27:0/284002297 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(228..228 src has 1..228) ==== 2700+0+0 (crc 0 0 0) 0x7f05b0050c40 con 0x7f05c01330b0 2024-09-16T07:59:26.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.493+0000 7f05c5791640 1 -- 172.21.15.27:0/284002297 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f05c00a5580 con 0x7f05c01330b0 2024-09-16T07:59:26.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.493+0000 7f05be7fc640 1 --2- 172.21.15.27:0/284002297 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7f05ac03e930 0x7f05ac040e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:59:26.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.494+0000 7f05be7fc640 1 --2- 172.21.15.27:0/284002297 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7f05ac03e930 0x7f05ac040e40 crc :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:59:26.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.498+0000 7f05a77fe640 1 -- 172.21.15.27:0/284002297 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (crc 0 0 0) 0x7f05b0014c60 con 0x7f05c01330b0 2024-09-16T07:59:26.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.689+0000 7f05c5791640 1 -- 172.21.15.27:0/284002297 --> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) -- 0x7f05c0074ac0 con 0x7f05ac03e930 2024-09-16T07:59:26.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.690+0000 7f05a77fe640 1 -- 172.21.15.27:0/284002297 <== mgr.4102 v2:172.21.15.187:6800/4038049593 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) ==== 41+0+0 (crc 0 0 0) 0x7f05b4005870 con 0x7f05ac03e930 2024-09-16T07:59:26.688 INFO:teuthology.orchestra.run.smithi027.stderr:instructed osd(s) 2 to deep-scrub 2024-09-16T07:59:26.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.692+0000 7f05c5791640 1 -- 172.21.15.27:0/284002297 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7f05ac03e930 msgr2=0x7f05ac040e40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:59:26.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.692+0000 7f05c5791640 1 --2- 172.21.15.27:0/284002297 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7f05ac03e930 0x7f05ac040e40 crc :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:59:26.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.692+0000 7f05c5791640 1 -- 172.21.15.27:0/284002297 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f05c01330b0 msgr2=0x7f05c011eaa0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:59:26.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.692+0000 7f05c5791640 1 --2- 172.21.15.27:0/284002297 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f05c01330b0 0x7f05c011eaa0 crc :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:59:26.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.692+0000 7f05c5791640 1 -- 172.21.15.27:0/284002297 shutdown_connections 2024-09-16T07:59:26.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.692+0000 7f05c5791640 1 --2- 172.21.15.27:0/284002297 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7f05ac03e930 0x7f05ac040e40 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:59:26.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.692+0000 7f05c5791640 1 --2- 172.21.15.27:0/284002297 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f05c01330b0 0x7f05c011eaa0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:59:26.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.692+0000 7f05c5791640 1 -- 172.21.15.27:0/284002297 >> 172.21.15.27:0/284002297 conn(0x7f05c00a25b0 msgr2=0x7f05c00a42b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:59:26.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.692+0000 7f05c5791640 1 -- 172.21.15.27:0/284002297 shutdown_connections 2024-09-16T07:59:26.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.692+0000 7f05c5791640 1 -- 172.21.15.27:0/284002297 wait complete. 2024-09-16T07:59:26.703 INFO:tasks.ceph:Scrubbing osd.3 2024-09-16T07:59:26.703 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-09-16T07:59:26.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.813+0000 7f58f1888640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T07:59:26.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.823+0000 7f58f1888640 1 Processor -- start 2024-09-16T07:59:26.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.823+0000 7f58f1888640 1 -- start start 2024-09-16T07:59:26.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.825+0000 7f58f1888640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f58ec130470 0x7f58ec130880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:59:26.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.825+0000 7f58f1888640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f58ec0745e0 con 0x7f58ec133250 2024-09-16T07:59:26.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.825+0000 7f58f1888640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 -- 0x7f58ec0740f0 con 0x7f58ec130470 2024-09-16T07:59:26.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.825+0000 7f58eaffd640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f58ec130470 0x7f58ec130880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:59:26.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.825+0000 7f58eaffd640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f58ec130470 0x7f58ec130880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40560/0 (socket says 172.21.15.27:40560) 2024-09-16T07:59:26.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.825+0000 7f58ea7fc640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f58ec133250 0x7f58ec133660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:36170/0 (socket says 172.21.15.27:36170) 2024-09-16T07:59:26.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.825+0000 7f58eaffd640 1 -- 172.21.15.27:0/1806846275 learned_addr learned my addr 172.21.15.27:0/1806846275 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-09-16T07:59:26.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.825+0000 7f58eaffd640 1 -- 172.21.15.27:0/1806846275 >> v1:172.21.15.27:6789/0 conn(0x7f58ec133250 legacy=0x7f58ec133660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:59:26.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.825+0000 7f58eaffd640 1 -- 172.21.15.27:0/1806846275 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58ec0749a0 con 0x7f58ec130470 2024-09-16T07:59:26.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.826+0000 7f58eaffd640 1 --2- 172.21.15.27:0/1806846275 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f58ec130470 0x7f58ec130880 crc :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f4ac21becf123e7 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:59:26.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.826+0000 7f58e9ffb640 1 -- 172.21.15.27:0/1806846275 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f58dc009e50 con 0x7f58ec130470 2024-09-16T07:59:26.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.826+0000 7f58e9ffb640 1 -- 172.21.15.27:0/1806846275 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f58dc00e230 con 0x7f58ec130470 2024-09-16T07:59:26.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.826+0000 7f58e9ffb640 1 -- 172.21.15.27:0/1806846275 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f58dc00e4d0 con 0x7f58ec130470 2024-09-16T07:59:26.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.826+0000 7f58f1888640 1 -- 172.21.15.27:0/1806846275 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f58ec130470 msgr2=0x7f58ec130880 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:59:26.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.826+0000 7f58f1888640 1 --2- 172.21.15.27:0/1806846275 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f58ec130470 0x7f58ec130880 crc :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:59:26.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.826+0000 7f58f1888640 1 -- 172.21.15.27:0/1806846275 shutdown_connections 2024-09-16T07:59:26.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.826+0000 7f58f1888640 1 --2- 172.21.15.27:0/1806846275 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f58ec130470 0x7f58ec130880 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:59:26.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.826+0000 7f58f1888640 1 -- 172.21.15.27:0/1806846275 >> 172.21.15.27:0/1806846275 conn(0x7f58ec0690e0 msgr2=0x7f58ec0694f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:59:26.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.827+0000 7f58f1888640 1 -- 172.21.15.27:0/1806846275 shutdown_connections 2024-09-16T07:59:26.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.827+0000 7f58f1888640 1 -- 172.21.15.27:0/1806846275 wait complete. 2024-09-16T07:59:26.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.827+0000 7f58f1888640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T07:59:26.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.828+0000 7f58f1888640 1 Processor -- start 2024-09-16T07:59:26.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.829+0000 7f58f1888640 1 -- start start 2024-09-16T07:59:26.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.830+0000 7f58f1888640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f58ec130470 0x7f58ec11f580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:59:26.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.830+0000 7f58f1888640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f58ec122990 con 0x7f58ec130470 2024-09-16T07:59:26.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.830+0000 7f58eaffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f58ec130470 0x7f58ec11f580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:59:26.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.830+0000 7f58eaffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f58ec130470 0x7f58ec11f580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40562/0 (socket says 172.21.15.27:40562) 2024-09-16T07:59:26.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.830+0000 7f58eaffd640 1 -- 172.21.15.27:0/2498844558 learned_addr learned my addr 172.21.15.27:0/2498844558 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-09-16T07:59:26.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.831+0000 7f58eaffd640 1 -- 172.21.15.27:0/2498844558 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58ec120ed0 con 0x7f58ec130470 2024-09-16T07:59:26.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.831+0000 7f58eaffd640 1 --2- 172.21.15.27:0/2498844558 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f58ec130470 0x7f58ec11f580 crc :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:59:26.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.831+0000 7f58d3fff640 1 -- 172.21.15.27:0/2498844558 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f58dc00ee20 con 0x7f58ec130470 2024-09-16T07:59:26.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.831+0000 7f58d3fff640 1 -- 172.21.15.27:0/2498844558 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f58dc015420 con 0x7f58ec130470 2024-09-16T07:59:26.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.831+0000 7f58f1888640 1 -- 172.21.15.27:0/2498844558 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f58ec1217b0 con 0x7f58ec130470 2024-09-16T07:59:26.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.831+0000 7f58d3fff640 1 -- 172.21.15.27:0/2498844558 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f58dc0156c0 con 0x7f58ec130470 2024-09-16T07:59:26.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.831+0000 7f58f1888640 1 -- 172.21.15.27:0/2498844558 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58ec122f20 con 0x7f58ec130470 2024-09-16T07:59:26.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.832+0000 7f58f1888640 1 -- 172.21.15.27:0/2498844558 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f58ec122790 con 0x7f58ec130470 2024-09-16T07:59:26.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.832+0000 7f58d3fff640 1 -- 172.21.15.27:0/2498844558 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) ==== 50954+0+0 (crc 0 0 0) 0x7f58dc0158c0 con 0x7f58ec130470 2024-09-16T07:59:26.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.832+0000 7f58d3fff640 1 --2- 172.21.15.27:0/2498844558 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7f58d803e8e0 0x7f58d8040df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:59:26.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.832+0000 7f58d3fff640 1 -- 172.21.15.27:0/2498844558 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(228..228 src has 1..228) ==== 2700+0+0 (crc 0 0 0) 0x7f58dc050d10 con 0x7f58ec130470 2024-09-16T07:59:26.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.832+0000 7f58d3fff640 1 --2- 172.21.15.27:0/2498844558 >> v2:172.21.15.187:6811/1136904278 conn(0x7f58d80438e0 0x7f58d8045d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:59:26.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.832+0000 7f58d3fff640 1 -- 172.21.15.27:0/2498844558 --> v2:172.21.15.187:6811/1136904278 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f58d8046420 con 0x7f58d80438e0 2024-09-16T07:59:26.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.832+0000 7f58d3fff640 1 -- 172.21.15.27:0/2498844558 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_get_version_reply(handle=1 version=228) ==== 24+0+0 (crc 0 0 0) 0x7f58dc053280 con 0x7f58ec130470 2024-09-16T07:59:26.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.833+0000 7f58ea7fc640 1 --2- 172.21.15.27:0/2498844558 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7f58d803e8e0 0x7f58d8040df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:59:26.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.833+0000 7f58ea7fc640 1 --2- 172.21.15.27:0/2498844558 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7f58d803e8e0 0x7f58d8040df0 crc :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:59:26.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.834+0000 7f58eb7fe640 1 --2- 172.21.15.27:0/2498844558 >> v2:172.21.15.187:6811/1136904278 conn(0x7f58d80438e0 0x7f58d8045d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-09-16T07:59:26.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.838+0000 7f58eb7fe640 1 --2- 172.21.15.27:0/2498844558 >> v2:172.21.15.187:6811/1136904278 conn(0x7f58d80438e0 0x7f58d8045d90 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:59:26.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.839+0000 7f58d3fff640 1 -- 172.21.15.27:0/2498844558 <== osd.3 v2:172.21.15.187:6811/1136904278 1 ==== command_reply(tid 1: 0 ) ==== 8+0+5790 (crc 0 0 0) 0x7f58d8046420 con 0x7f58d80438e0 2024-09-16T07:59:26.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.843+0000 7f58f1888640 1 -- 172.21.15.27:0/2498844558 --> v2:172.21.15.187:6811/1136904278 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f58ec12fd90 con 0x7f58d80438e0 2024-09-16T07:59:26.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.845+0000 7f58d3fff640 1 -- 172.21.15.27:0/2498844558 <== osd.3 v2:172.21.15.187:6811/1136904278 2 ==== command_reply(tid 2: 0 ) ==== 8+0+22 (crc 0 0 0) 0x7f58ec12fd90 con 0x7f58d80438e0 2024-09-16T07:59:26.843 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-09-16T07:59:26.843 INFO:teuthology.orchestra.run.smithi027.stdout: "success": "" 2024-09-16T07:59:26.843 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-09-16T07:59:26.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.845+0000 7f58f1888640 1 -- 172.21.15.27:0/2498844558 >> v2:172.21.15.187:6811/1136904278 conn(0x7f58d80438e0 msgr2=0x7f58d8045d90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:59:26.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.845+0000 7f58f1888640 1 --2- 172.21.15.27:0/2498844558 >> v2:172.21.15.187:6811/1136904278 conn(0x7f58d80438e0 0x7f58d8045d90 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:59:26.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.845+0000 7f58f1888640 1 -- 172.21.15.27:0/2498844558 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7f58d803e8e0 msgr2=0x7f58d8040df0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:59:26.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.845+0000 7f58f1888640 1 --2- 172.21.15.27:0/2498844558 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7f58d803e8e0 0x7f58d8040df0 crc :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:59:26.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.845+0000 7f58f1888640 1 -- 172.21.15.27:0/2498844558 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f58ec130470 msgr2=0x7f58ec11f580 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:59:26.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.845+0000 7f58f1888640 1 --2- 172.21.15.27:0/2498844558 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f58ec130470 0x7f58ec11f580 crc :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:59:26.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.845+0000 7f58f1888640 1 -- 172.21.15.27:0/2498844558 shutdown_connections 2024-09-16T07:59:26.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.845+0000 7f58f1888640 1 --2- 172.21.15.27:0/2498844558 >> v2:172.21.15.187:6811/1136904278 conn(0x7f58d80438e0 0x7f58d8045d90 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:59:26.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.845+0000 7f58f1888640 1 --2- 172.21.15.27:0/2498844558 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7f58d803e8e0 0x7f58d8040df0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:59:26.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.845+0000 7f58f1888640 1 --2- 172.21.15.27:0/2498844558 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f58ec130470 0x7f58ec11f580 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:59:26.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.845+0000 7f58f1888640 1 -- 172.21.15.27:0/2498844558 >> 172.21.15.27:0/2498844558 conn(0x7f58ec0690e0 msgr2=0x7f58ec0a4f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:59:26.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.845+0000 7f58f1888640 1 -- 172.21.15.27:0/2498844558 shutdown_connections 2024-09-16T07:59:26.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.845+0000 7f58f1888640 1 -- 172.21.15.27:0/2498844558 wait complete. 2024-09-16T07:59:26.855 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-09-16T07:59:26.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.965+0000 7f6411551640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T07:59:26.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.972+0000 7f6411551640 1 Processor -- start 2024-09-16T07:59:26.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.972+0000 7f6411551640 1 -- start start 2024-09-16T07:59:26.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.974+0000 7f6411551640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f640c06d550 0x7f640c1386d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:59:26.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.974+0000 7f6411551640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f640c0a0d00 con 0x7f640c09ec50 2024-09-16T07:59:26.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.974+0000 7f6411551640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 -- 0x7f640c0a0a30 con 0x7f640c06d550 2024-09-16T07:59:26.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.974+0000 7f640affd640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f640c09ec50 0x7f640c06ce40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:36178/0 (socket says 172.21.15.27:36178) 2024-09-16T07:59:26.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.974+0000 7f640affd640 1 -- 172.21.15.27:0/4281351735 learned_addr learned my addr 172.21.15.27:0/4281351735 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-09-16T07:59:26.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.974+0000 7f640a7fc640 1 --2- 172.21.15.27:0/4281351735 >> v2:172.21.15.27:3300/0 conn(0x7f640c06d550 0x7f640c1386d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:59:26.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.974+0000 7f640a7fc640 1 -- 172.21.15.27:0/4281351735 >> v1:172.21.15.27:6789/0 conn(0x7f640c09ec50 legacy=0x7f640c06ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:59:26.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.974+0000 7f6409ffb640 1 -- 172.21.15.27:0/4281351735 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3060783694 0 0) 0x7f640c0a0d00 con 0x7f640c09ec50 2024-09-16T07:59:26.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.974+0000 7f640a7fc640 1 -- 172.21.15.27:0/4281351735 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f640c0710b0 con 0x7f640c06d550 2024-09-16T07:59:26.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.974+0000 7f640a7fc640 1 --2- 172.21.15.27:0/4281351735 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f640c06d550 0x7f640c1386d0 crc :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd65a22f61629368 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:59:26.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.974+0000 7f6409ffb640 1 -- 172.21.15.27:0/4281351735 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f6400009d90 con 0x7f640c06d550 2024-09-16T07:59:26.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.975+0000 7f6409ffb640 1 -- 172.21.15.27:0/4281351735 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f640000e230 con 0x7f640c06d550 2024-09-16T07:59:26.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.975+0000 7f6409ffb640 1 -- 172.21.15.27:0/4281351735 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f640000e4d0 con 0x7f640c06d550 2024-09-16T07:59:26.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.975+0000 7f6411551640 1 -- 172.21.15.27:0/4281351735 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f640c06d550 msgr2=0x7f640c1386d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:59:26.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.975+0000 7f6411551640 1 --2- 172.21.15.27:0/4281351735 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f640c06d550 0x7f640c1386d0 crc :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:59:26.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.975+0000 7f6411551640 1 -- 172.21.15.27:0/4281351735 shutdown_connections 2024-09-16T07:59:26.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.976+0000 7f6411551640 1 --2- 172.21.15.27:0/4281351735 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f640c06d550 0x7f640c1386d0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:59:26.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.976+0000 7f6411551640 1 -- 172.21.15.27:0/4281351735 >> 172.21.15.27:0/4281351735 conn(0x7f640c069d10 msgr2=0x7f640c09f200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:59:26.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.976+0000 7f6411551640 1 -- 172.21.15.27:0/4281351735 shutdown_connections 2024-09-16T07:59:26.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.976+0000 7f6411551640 1 -- 172.21.15.27:0/4281351735 wait complete. 2024-09-16T07:59:26.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.976+0000 7f6411551640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T07:59:26.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.978+0000 7f6411551640 1 Processor -- start 2024-09-16T07:59:26.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.978+0000 7f6411551640 1 -- start start 2024-09-16T07:59:26.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.979+0000 7f6411551640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f640c06d550 0x7f640c1212b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:59:26.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.979+0000 7f6411551640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f640c123200 con 0x7f640c06d550 2024-09-16T07:59:26.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.979+0000 7f640affd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f640c06d550 0x7f640c1212b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:59:26.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.980+0000 7f640affd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f640c06d550 0x7f640c1212b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40574/0 (socket says 172.21.15.27:40574) 2024-09-16T07:59:26.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.980+0000 7f640affd640 1 -- 172.21.15.27:0/1549348902 learned_addr learned my addr 172.21.15.27:0/1549348902 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-09-16T07:59:26.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.980+0000 7f640affd640 1 -- 172.21.15.27:0/1549348902 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f640c121090 con 0x7f640c06d550 2024-09-16T07:59:26.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.980+0000 7f640affd640 1 --2- 172.21.15.27:0/1549348902 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f640c06d550 0x7f640c1212b0 crc :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:59:26.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.980+0000 7f63f37fe640 1 -- 172.21.15.27:0/1549348902 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f63fc005a60 con 0x7f640c06d550 2024-09-16T07:59:26.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.980+0000 7f6411551640 1 -- 172.21.15.27:0/1549348902 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f640c124820 con 0x7f640c06d550 2024-09-16T07:59:26.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.980+0000 7f63f37fe640 1 -- 172.21.15.27:0/1549348902 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f63fc005bc0 con 0x7f640c06d550 2024-09-16T07:59:26.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.980+0000 7f63f37fe640 1 -- 172.21.15.27:0/1549348902 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f63fc005e60 con 0x7f640c06d550 2024-09-16T07:59:26.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.980+0000 7f6411551640 1 -- 172.21.15.27:0/1549348902 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f640c122be0 con 0x7f640c06d550 2024-09-16T07:59:26.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.981+0000 7f63f37fe640 1 -- 172.21.15.27:0/1549348902 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) ==== 50954+0+0 (crc 0 0 0) 0x7f63fc0070b0 con 0x7f640c06d550 2024-09-16T07:59:26.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.981+0000 7f6411551640 1 -- 172.21.15.27:0/1549348902 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f640c06ce40 con 0x7f640c06d550 2024-09-16T07:59:26.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.981+0000 7f63f37fe640 1 --2- 172.21.15.27:0/1549348902 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7f63f803e8e0 0x7f63f8040df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:59:26.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.981+0000 7f63f37fe640 1 -- 172.21.15.27:0/1549348902 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(228..228 src has 1..228) ==== 2700+0+0 (crc 0 0 0) 0x7f63fc04f6e0 con 0x7f640c06d550 2024-09-16T07:59:26.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.982+0000 7f640a7fc640 1 --2- 172.21.15.27:0/1549348902 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7f63f803e8e0 0x7f63f8040df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:59:26.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.982+0000 7f640a7fc640 1 --2- 172.21.15.27:0/1549348902 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7f63f803e8e0 0x7f63f8040df0 crc :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:59:26.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:26.987+0000 7f63f37fe640 1 -- 172.21.15.27:0/1549348902 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (crc 0 0 0) 0x7f63fc00dc10 con 0x7f640c06d550 2024-09-16T07:59:27.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:27.173+0000 7f6411551640 1 -- 172.21.15.27:0/1549348902 --> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) -- 0x7f640c06e0b0 con 0x7f63f803e8e0 2024-09-16T07:59:27.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:27.174+0000 7f63f37fe640 1 -- 172.21.15.27:0/1549348902 <== mgr.4102 v2:172.21.15.187:6800/4038049593 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) ==== 41+0+0 (crc 0 0 0) 0x7f640c06e0b0 con 0x7f63f803e8e0 2024-09-16T07:59:27.172 INFO:teuthology.orchestra.run.smithi027.stderr:instructed osd(s) 3 to deep-scrub 2024-09-16T07:59:27.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:27.176+0000 7f6411551640 1 -- 172.21.15.27:0/1549348902 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7f63f803e8e0 msgr2=0x7f63f8040df0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:59:27.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:27.176+0000 7f6411551640 1 --2- 172.21.15.27:0/1549348902 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7f63f803e8e0 0x7f63f8040df0 crc :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:59:27.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:27.176+0000 7f6411551640 1 -- 172.21.15.27:0/1549348902 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f640c06d550 msgr2=0x7f640c1212b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:59:27.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:27.176+0000 7f6411551640 1 --2- 172.21.15.27:0/1549348902 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f640c06d550 0x7f640c1212b0 crc :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:59:27.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:27.176+0000 7f6411551640 1 -- 172.21.15.27:0/1549348902 shutdown_connections 2024-09-16T07:59:27.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:27.176+0000 7f6411551640 1 --2- 172.21.15.27:0/1549348902 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7f63f803e8e0 0x7f63f8040df0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:59:27.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:27.176+0000 7f6411551640 1 --2- 172.21.15.27:0/1549348902 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f640c06d550 0x7f640c1212b0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:59:27.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:27.176+0000 7f6411551640 1 -- 172.21.15.27:0/1549348902 >> 172.21.15.27:0/1549348902 conn(0x7f640c069d10 msgr2=0x7f640c0a2600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:59:27.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:27.176+0000 7f6411551640 1 -- 172.21.15.27:0/1549348902 shutdown_connections 2024-09-16T07:59:27.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:27.176+0000 7f6411551640 1 -- 172.21.15.27:0/1549348902 wait complete. 2024-09-16T07:59:27.188 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T07:59:27.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:27.299+0000 7f0a54288640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T07:59:27.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:27.304+0000 7f0a54288640 1 Processor -- start 2024-09-16T07:59:27.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:27.304+0000 7f0a54288640 1 -- start start 2024-09-16T07:59:27.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:27.306+0000 7f0a54288640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f0a4c135380 0x7f0a4c1377a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:59:27.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:27.306+0000 7f0a54288640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0a4c09ce60 con 0x7f0a4c131300 2024-09-16T07:59:27.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:27.306+0000 7f0a54288640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 -- 0x7f0a4c09e010 con 0x7f0a4c135380 2024-09-16T07:59:27.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:27.306+0000 7f0a51823640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f0a4c135380 0x7f0a4c1377a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:59:27.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:27.306+0000 7f0a52024640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f0a4c131300 0x7f0a4c131710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:36180/0 (socket says 172.21.15.27:36180) 2024-09-16T07:59:27.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:27.306+0000 7f0a52024640 1 -- 172.21.15.27:0/2560505364 learned_addr learned my addr 172.21.15.27:0/2560505364 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-09-16T07:59:27.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:27.306+0000 7f0a51823640 1 --2- 172.21.15.27:0/2560505364 >> v2:172.21.15.27:3300/0 conn(0x7f0a4c135380 0x7f0a4c1377a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40582/0 (socket says 172.21.15.27:40582) 2024-09-16T07:59:27.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:27.306+0000 7f0a51823640 1 -- 172.21.15.27:0/2560505364 >> v1:172.21.15.27:6789/0 conn(0x7f0a4c131300 legacy=0x7f0a4c131710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:59:27.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:27.306+0000 7f0a51823640 1 -- 172.21.15.27:0/2560505364 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a4c131e20 con 0x7f0a4c135380 2024-09-16T07:59:27.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:27.306+0000 7f0a51022640 1 -- 172.21.15.27:0/2560505364 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 199438823 0 0) 0x7f0a4c09ce60 con 0x7f0a4c131300 2024-09-16T07:59:27.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:27.307+0000 7f0a51823640 1 --2- 172.21.15.27:0/2560505364 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0a4c135380 0x7f0a4c1377a0 crc :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39d645f75dbe9eab server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:59:27.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:27.307+0000 7f0a51022640 1 -- 172.21.15.27:0/2560505364 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f0a3c009d90 con 0x7f0a4c135380 2024-09-16T07:59:27.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:27.307+0000 7f0a51022640 1 -- 172.21.15.27:0/2560505364 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f0a3c00e230 con 0x7f0a4c135380 2024-09-16T07:59:27.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:27.307+0000 7f0a51022640 1 -- 172.21.15.27:0/2560505364 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f0a3c00e4d0 con 0x7f0a4c135380 2024-09-16T07:59:27.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:27.307+0000 7f0a54288640 1 -- 172.21.15.27:0/2560505364 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0a4c135380 msgr2=0x7f0a4c1377a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:59:27.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:27.307+0000 7f0a54288640 1 --2- 172.21.15.27:0/2560505364 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0a4c135380 0x7f0a4c1377a0 crc :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:59:27.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:27.308+0000 7f0a54288640 1 -- 172.21.15.27:0/2560505364 shutdown_connections 2024-09-16T07:59:27.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:27.308+0000 7f0a54288640 1 --2- 172.21.15.27:0/2560505364 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0a4c135380 0x7f0a4c1377a0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:59:27.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:27.308+0000 7f0a54288640 1 -- 172.21.15.27:0/2560505364 >> 172.21.15.27:0/2560505364 conn(0x7f0a4c069d10 msgr2=0x7f0a4c09f200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:59:27.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:27.308+0000 7f0a54288640 1 -- 172.21.15.27:0/2560505364 shutdown_connections 2024-09-16T07:59:27.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:27.308+0000 7f0a54288640 1 -- 172.21.15.27:0/2560505364 wait complete. 2024-09-16T07:59:27.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:27.308+0000 7f0a54288640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T07:59:27.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:27.310+0000 7f0a54288640 1 Processor -- start 2024-09-16T07:59:27.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:27.310+0000 7f0a54288640 1 -- start start 2024-09-16T07:59:27.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:27.311+0000 7f0a54288640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0a4c131300 0x7f0a4c1c7dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:59:27.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:27.311+0000 7f0a54288640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f0a4c1c7a00 con 0x7f0a4c131300 2024-09-16T07:59:27.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:27.311+0000 7f0a52024640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0a4c131300 0x7f0a4c1c7dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:59:27.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:27.311+0000 7f0a52024640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0a4c131300 0x7f0a4c1c7dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40598/0 (socket says 172.21.15.27:40598) 2024-09-16T07:59:27.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:27.311+0000 7f0a52024640 1 -- 172.21.15.27:0/4276236112 learned_addr learned my addr 172.21.15.27:0/4276236112 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-09-16T07:59:27.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:27.312+0000 7f0a52024640 1 -- 172.21.15.27:0/4276236112 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a4c06d930 con 0x7f0a4c131300 2024-09-16T07:59:27.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:27.312+0000 7f0a52024640 1 --2- 172.21.15.27:0/4276236112 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0a4c131300 0x7f0a4c1c7dc0 crc :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:59:27.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:27.312+0000 7f0a42ffd640 1 -- 172.21.15.27:0/4276236112 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f0a48004b80 con 0x7f0a4c131300 2024-09-16T07:59:27.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:27.312+0000 7f0a42ffd640 1 -- 172.21.15.27:0/4276236112 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f0a48004ce0 con 0x7f0a4c131300 2024-09-16T07:59:27.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:27.312+0000 7f0a54288640 1 -- 172.21.15.27:0/4276236112 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a4c121a40 con 0x7f0a4c131300 2024-09-16T07:59:27.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:27.312+0000 7f0a42ffd640 1 -- 172.21.15.27:0/4276236112 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f0a48007120 con 0x7f0a4c131300 2024-09-16T07:59:27.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:27.312+0000 7f0a54288640 1 -- 172.21.15.27:0/4276236112 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a4c121ca0 con 0x7f0a4c131300 2024-09-16T07:59:27.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:27.313+0000 7f0a42ffd640 1 -- 172.21.15.27:0/4276236112 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) ==== 50954+0+0 (crc 0 0 0) 0x7f0a480072d0 con 0x7f0a4c131300 2024-09-16T07:59:27.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:27.313+0000 7f0a42ffd640 1 --2- 172.21.15.27:0/4276236112 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7f0a3003e930 0x7f0a30040e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:59:27.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:27.313+0000 7f0a54288640 1 -- 172.21.15.27:0/4276236112 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0a4c09e990 con 0x7f0a4c131300 2024-09-16T07:59:27.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:27.313+0000 7f0a42ffd640 1 -- 172.21.15.27:0/4276236112 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(228..228 src has 1..228) ==== 2700+0+0 (crc 0 0 0) 0x7f0a4804f730 con 0x7f0a4c131300 2024-09-16T07:59:27.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:27.314+0000 7f0a51823640 1 --2- 172.21.15.27:0/4276236112 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7f0a3003e930 0x7f0a30040e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:59:27.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:27.314+0000 7f0a51823640 1 --2- 172.21.15.27:0/4276236112 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7f0a3003e930 0x7f0a30040e40 crc :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:59:27.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:27.319+0000 7f0a42ffd640 1 -- 172.21.15.27:0/4276236112 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (crc 0 0 0) 0x7f0a4800f420 con 0x7f0a4c131300 2024-09-16T07:59:27.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:27.498+0000 7f0a54288640 1 -- 172.21.15.27:0/4276236112 --> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f0a4c09db70 con 0x7f0a3003e930 2024-09-16T07:59:27.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:27.500+0000 7f0a42ffd640 1 -- 172.21.15.27:0/4276236112 <== mgr.4102 v2:172.21.15.187:6800/4038049593 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+31238 (crc 0 0 0) 0x7f0a3c00e4c0 con 0x7f0a3003e930 2024-09-16T07:59:27.498 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-16T07:59:27.499 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-09-16T07:59:27.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:27.502+0000 7f0a54288640 1 -- 172.21.15.27:0/4276236112 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7f0a3003e930 msgr2=0x7f0a30040e40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:59:27.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:27.502+0000 7f0a54288640 1 --2- 172.21.15.27:0/4276236112 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7f0a3003e930 0x7f0a30040e40 crc :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:59:27.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:27.502+0000 7f0a54288640 1 -- 172.21.15.27:0/4276236112 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0a4c131300 msgr2=0x7f0a4c1c7dc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:59:27.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:27.502+0000 7f0a54288640 1 --2- 172.21.15.27:0/4276236112 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0a4c131300 0x7f0a4c1c7dc0 crc :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:59:27.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:27.502+0000 7f0a54288640 1 -- 172.21.15.27:0/4276236112 shutdown_connections 2024-09-16T07:59:27.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:27.502+0000 7f0a54288640 1 --2- 172.21.15.27:0/4276236112 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7f0a3003e930 0x7f0a30040e40 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:59:27.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:27.502+0000 7f0a54288640 1 --2- 172.21.15.27:0/4276236112 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0a4c131300 0x7f0a4c1c7dc0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:59:27.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:27.502+0000 7f0a54288640 1 -- 172.21.15.27:0/4276236112 >> 172.21.15.27:0/4276236112 conn(0x7f0a4c069d10 msgr2=0x7f0a4c070600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:59:27.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:27.502+0000 7f0a54288640 1 -- 172.21.15.27:0/4276236112 shutdown_connections 2024-09-16T07:59:27.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:27.502+0000 7f0a54288640 1 -- 172.21.15.27:0/4276236112 wait complete. 2024-09-16T07:59:27.512 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":374,"stamp":"2024-09-16T07:59:27.148785+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":209190912,"kb_used":15568,"kb_used_data":15568,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":209175344,"statfs":{"total":214211493888,"available":214195552256,"internally_reserved":0,"allocated":15941632,"data_stored":15941632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.425322"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":645,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-09-16T07:59:18.795976+0000","last_change":"2024-09-16T07:54:26.740923+0000","last_active":"2024-09-16T07:59:18.795976+0000","last_peered":"2024-09-16T07:59:18.795976+0000","last_clean":"2024-09-16T07:59:18.795976+0000","last_became_active":"2024-09-16T07:54:24.784360+0000","last_became_peered":"2024-09-16T07:54:24.784360+0000","last_unstale":"2024-09-16T07:59:18.795976+0000","last_undegraded":"2024-09-16T07:59:18.795976+0000","last_fullsized":"2024-09-16T07:59:18.795976+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T07:54:23.687651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T07:54:23.687651+0000","last_clean_scrub_stamp":"2024-09-16T07:54:23.687651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":645,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-09-16T07:59:18.761001+0000","last_change":"2024-09-16T07:54:26.739464+0000","last_active":"2024-09-16T07:59:18.761001+0000","last_peered":"2024-09-16T07:59:18.761001+0000","last_clean":"2024-09-16T07:59:18.761001+0000","last_became_active":"2024-09-16T07:54:24.805913+0000","last_became_peered":"2024-09-16T07:54:24.805913+0000","last_unstale":"2024-09-16T07:59:18.761001+0000","last_undegraded":"2024-09-16T07:59:18.761001+0000","last_fullsized":"2024-09-16T07:59:18.761001+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T07:54:23.687651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T07:54:23.687651+0000","last_clean_scrub_stamp":"2024-09-16T07:54:23.687651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":645,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-09-16T07:59:18.782573+0000","last_change":"2024-09-16T07:54:26.735332+0000","last_active":"2024-09-16T07:59:18.782573+0000","last_peered":"2024-09-16T07:59:18.782573+0000","last_clean":"2024-09-16T07:59:18.782573+0000","last_became_active":"2024-09-16T07:54:24.773365+0000","last_became_peered":"2024-09-16T07:54:24.773365+0000","last_unstale":"2024-09-16T07:59:18.782573+0000","last_undegraded":"2024-09-16T07:59:18.782573+0000","last_fullsized":"2024-09-16T07:59:18.782573+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T07:54:23.687651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T07:54:23.687651+0000","last_clean_scrub_stamp":"2024-09-16T07:54:23.687651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":645,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-09-16T07:59:18.751250+0000","last_change":"2024-09-16T07:54:26.734682+0000","last_active":"2024-09-16T07:59:18.751250+0000","last_peered":"2024-09-16T07:59:18.751250+0000","last_clean":"2024-09-16T07:59:18.751250+0000","last_became_active":"2024-09-16T07:54:24.756974+0000","last_became_peered":"2024-09-16T07:54:24.756974+0000","last_unstale":"2024-09-16T07:59:18.751250+0000","last_undegraded":"2024-09-16T07:59:18.751250+0000","last_fullsized":"2024-09-16T07:59:18.751250+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T07:54:23.687651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T07:54:23.687651+0000","last_clean_scrub_stamp":"2024-09-16T07:54:23.687651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"15'2","reported_seq":647,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-09-16T07:59:18.780803+0000","last_change":"2024-09-16T07:54:26.741927+0000","last_active":"2024-09-16T07:59:18.780803+0000","last_peered":"2024-09-16T07:59:18.780803+0000","last_clean":"2024-09-16T07:59:18.780803+0000","last_became_active":"2024-09-16T07:54:24.794754+0000","last_became_peered":"2024-09-16T07:54:24.794754+0000","last_unstale":"2024-09-16T07:59:18.780803+0000","last_undegraded":"2024-09-16T07:59:18.780803+0000","last_fullsized":"2024-09-16T07:59:18.780803+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T07:54:23.687651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T07:54:23.687651+0000","last_clean_scrub_stamp":"2024-09-16T07:54:23.687651+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":645,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-09-16T07:59:18.758597+0000","last_change":"2024-09-16T07:54:26.734478+0000","last_active":"2024-09-16T07:59:18.758597+0000","last_peered":"2024-09-16T07:59:18.758597+0000","last_clean":"2024-09-16T07:59:18.758597+0000","last_became_active":"2024-09-16T07:54:24.774923+0000","last_became_peered":"2024-09-16T07:54:24.774923+0000","last_unstale":"2024-09-16T07:59:18.758597+0000","last_undegraded":"2024-09-16T07:59:18.758597+0000","last_fullsized":"2024-09-16T07:59:18.758597+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T07:54:23.687651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T07:54:23.687651+0000","last_clean_scrub_stamp":"2024-09-16T07:54:23.687651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":645,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-09-16T07:59:18.810530+0000","last_change":"2024-09-16T07:54:26.742127+0000","last_active":"2024-09-16T07:59:18.810530+0000","last_peered":"2024-09-16T07:59:18.810530+0000","last_clean":"2024-09-16T07:59:18.810530+0000","last_became_active":"2024-09-16T07:54:24.773472+0000","last_became_peered":"2024-09-16T07:54:24.773472+0000","last_unstale":"2024-09-16T07:59:18.810530+0000","last_undegraded":"2024-09-16T07:59:18.810530+0000","last_fullsized":"2024-09-16T07:59:18.810530+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T07:54:23.687651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T07:54:23.687651+0000","last_clean_scrub_stamp":"2024-09-16T07:54:23.687651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"13'1","reported_seq":646,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-09-16T07:59:18.805158+0000","last_change":"2024-09-16T07:54:26.739091+0000","last_active":"2024-09-16T07:59:18.805158+0000","last_peered":"2024-09-16T07:59:18.805158+0000","last_clean":"2024-09-16T07:59:18.805158+0000","last_became_active":"2024-09-16T07:54:24.801907+0000","last_became_peered":"2024-09-16T07:54:24.801907+0000","last_unstale":"2024-09-16T07:59:18.805158+0000","last_undegraded":"2024-09-16T07:59:18.805158+0000","last_fullsized":"2024-09-16T07:59:18.805158+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T07:54:23.687651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T07:54:23.687651+0000","last_clean_scrub_stamp":"2024-09-16T07:54:23.687651+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'52","reported_seq":717,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-09-16T07:59:18.755371+0000","last_change":"2024-09-16T07:54:22.755090+0000","last_active":"2024-09-16T07:59:18.755371+0000","last_peered":"2024-09-16T07:59:18.755371+0000","last_clean":"2024-09-16T07:59:18.755371+0000","last_became_active":"2024-09-16T07:54:22.736266+0000","last_became_peered":"2024-09-16T07:54:22.736266+0000","last_unstale":"2024-09-16T07:59:18.755371+0000","last_undegraded":"2024-09-16T07:59:18.755371+0000","last_fullsized":"2024-09-16T07:59:18.755371+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T07:54:21.678374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T07:54:21.678374+0000","last_clean_scrub_stamp":"2024-09-16T07:54:21.678374+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":214211493888,"available":214198435840,"internally_reserved":0,"allocated":13058048,"data_stored":13058048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53552873472,"available":53549346816,"internally_reserved":0,"allocated":3526656,"data_stored":3526656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705963,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":4340,"kb_used_data":4340,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52293388,"statfs":{"total":53552873472,"available":53548429312,"internally_reserved":0,"allocated":4444160,"data_stored":4444160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705963,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":3300,"kb_used_data":3300,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52294428,"statfs":{"total":53552873472,"available":53549494272,"internally_reserved":0,"allocated":3379200,"data_stored":3379200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705963,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":3376,"kb_used_data":3376,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52294352,"statfs":{"total":53552873472,"available":53549416448,"internally_reserved":0,"allocated":3457024,"data_stored":3457024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705963,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":4552,"kb_used_data":4552,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52293176,"statfs":{"total":53552873472,"available":53548212224,"internally_reserved":0,"allocated":4661248,"data_stored":4661248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":3,"total":53552873472,"available":53549346816,"internally_reserved":0,"allocated":3526656,"data_stored":3526656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53552873472,"available":53549064192,"internally_reserved":0,"allocated":3809280,"data_stored":3809280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53552873472,"available":53550006272,"internally_reserved":0,"allocated":2867200,"data_stored":2867200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53552873472,"available":53550018560,"internally_reserved":0,"allocated":2854912,"data_stored":2854912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":53552873472,"available":53549346816,"internally_reserved":0,"allocated":3526656,"data_stored":3526656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T07:59:27.515 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-09-16T07:54:23.687651+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=7, tm_min=54, tm_sec=23, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=7, tm_min=59, tm_sec=24, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T07:59:27.515 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-09-16T07:54:23.687651+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=7, tm_min=54, tm_sec=23, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=7, tm_min=59, tm_sec=24, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T07:59:27.515 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-09-16T07:54:23.687651+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=7, tm_min=54, tm_sec=23, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=7, tm_min=59, tm_sec=24, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T07:59:27.515 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-09-16T07:54:23.687651+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=7, tm_min=54, tm_sec=23, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=7, tm_min=59, tm_sec=24, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T07:59:27.516 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-09-16T07:54:23.687651+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=7, tm_min=54, tm_sec=23, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=7, tm_min=59, tm_sec=24, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T07:59:27.516 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-09-16T07:54:23.687651+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=7, tm_min=54, tm_sec=23, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=7, tm_min=59, tm_sec=24, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T07:59:27.516 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-09-16T07:54:23.687651+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=7, tm_min=54, tm_sec=23, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=7, tm_min=59, tm_sec=24, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T07:59:27.516 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-09-16T07:54:23.687651+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=7, tm_min=54, tm_sec=23, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=7, tm_min=59, tm_sec=24, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T07:59:27.516 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-09-16T07:54:21.678374+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=7, tm_min=54, tm_sec=21, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=7, tm_min=59, tm_sec=24, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T07:59:27.516 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-09-16T07:59:47.517 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T07:59:47.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:47.643+0000 7f5d39111640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T07:59:47.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:47.648+0000 7f5d39111640 1 Processor -- start 2024-09-16T07:59:47.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:47.648+0000 7f5d39111640 1 -- start start 2024-09-16T07:59:47.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:47.649+0000 7f5d39111640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f5d34138720 0x7f5d3413ab40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:59:47.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:47.649+0000 7f5d39111640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5d34133040 con 0x7f5d34132520 2024-09-16T07:59:47.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:47.649+0000 7f5d39111640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 -- 0x7f5d3406f6d0 con 0x7f5d34138720 2024-09-16T07:59:47.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:47.650+0000 7f5d32d9d640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f5d34132520 0x7f5d34132930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:38666/0 (socket says 172.21.15.27:38666) 2024-09-16T07:59:47.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:47.650+0000 7f5d32d9d640 1 -- 172.21.15.27:0/4163176893 learned_addr learned my addr 172.21.15.27:0/4163176893 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-09-16T07:59:47.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:47.650+0000 7f5d3259c640 1 --2- 172.21.15.27:0/4163176893 >> v2:172.21.15.27:3300/0 conn(0x7f5d34138720 0x7f5d3413ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:59:47.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:47.650+0000 7f5d3259c640 1 -- 172.21.15.27:0/4163176893 >> v1:172.21.15.27:6789/0 conn(0x7f5d34132520 legacy=0x7f5d34132930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:59:47.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:47.650+0000 7f5d3259c640 1 -- 172.21.15.27:0/4163176893 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d3406f3d0 con 0x7f5d34138720 2024-09-16T07:59:47.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:47.650+0000 7f5d31d9b640 1 -- 172.21.15.27:0/4163176893 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2255418823 0 0) 0x7f5d24001110 con 0x7f5d34132520 2024-09-16T07:59:47.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:47.650+0000 7f5d3259c640 1 --2- 172.21.15.27:0/4163176893 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d34138720 0x7f5d3413ab40 crc :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c2feb3d4b3ae741 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:59:47.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:47.650+0000 7f5d31d9b640 1 -- 172.21.15.27:0/4163176893 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f5d1c009d90 con 0x7f5d34138720 2024-09-16T07:59:47.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:47.650+0000 7f5d31d9b640 1 -- 172.21.15.27:0/4163176893 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f5d1c00e230 con 0x7f5d34138720 2024-09-16T07:59:47.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:47.650+0000 7f5d31d9b640 1 -- 172.21.15.27:0/4163176893 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f5d1c00e4d0 con 0x7f5d34138720 2024-09-16T07:59:47.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:47.651+0000 7f5d39111640 1 -- 172.21.15.27:0/4163176893 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d34138720 msgr2=0x7f5d3413ab40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:59:47.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:47.651+0000 7f5d39111640 1 --2- 172.21.15.27:0/4163176893 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d34138720 0x7f5d3413ab40 crc :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:59:47.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:47.651+0000 7f5d39111640 1 -- 172.21.15.27:0/4163176893 shutdown_connections 2024-09-16T07:59:47.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:47.651+0000 7f5d39111640 1 --2- 172.21.15.27:0/4163176893 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d34138720 0x7f5d3413ab40 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:59:47.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:47.651+0000 7f5d39111640 1 -- 172.21.15.27:0/4163176893 >> 172.21.15.27:0/4163176893 conn(0x7f5d3406e490 msgr2=0x7f5d3406e8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:59:47.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:47.651+0000 7f5d39111640 1 -- 172.21.15.27:0/4163176893 shutdown_connections 2024-09-16T07:59:47.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:47.651+0000 7f5d39111640 1 -- 172.21.15.27:0/4163176893 wait complete. 2024-09-16T07:59:47.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:47.651+0000 7f5d39111640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T07:59:47.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:47.653+0000 7f5d39111640 1 Processor -- start 2024-09-16T07:59:47.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:47.654+0000 7f5d39111640 1 -- start start 2024-09-16T07:59:47.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:47.655+0000 7f5d39111640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d34132520 0x7f5d3411f580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:59:47.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:47.655+0000 7f5d39111640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f5d34122c10 con 0x7f5d34132520 2024-09-16T07:59:47.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:47.655+0000 7f5d32d9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d34132520 0x7f5d3411f580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:59:47.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:47.655+0000 7f5d32d9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d34132520 0x7f5d3411f580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41340/0 (socket says 172.21.15.27:41340) 2024-09-16T07:59:47.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:47.655+0000 7f5d32d9d640 1 -- 172.21.15.27:0/1122958940 learned_addr learned my addr 172.21.15.27:0/1122958940 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-09-16T07:59:47.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:47.655+0000 7f5d32d9d640 1 -- 172.21.15.27:0/1122958940 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d3411f1c0 con 0x7f5d34132520 2024-09-16T07:59:47.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:47.656+0000 7f5d32d9d640 1 --2- 172.21.15.27:0/1122958940 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d34132520 0x7f5d3411f580 crc :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:59:47.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:47.656+0000 7f5d2b7fe640 1 -- 172.21.15.27:0/1122958940 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f5d24004b80 con 0x7f5d34132520 2024-09-16T07:59:47.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:47.656+0000 7f5d2b7fe640 1 -- 172.21.15.27:0/1122958940 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f5d24004ce0 con 0x7f5d34132520 2024-09-16T07:59:47.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:47.656+0000 7f5d39111640 1 -- 172.21.15.27:0/1122958940 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d34120790 con 0x7f5d34132520 2024-09-16T07:59:47.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:47.656+0000 7f5d2b7fe640 1 -- 172.21.15.27:0/1122958940 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f5d24007120 con 0x7f5d34132520 2024-09-16T07:59:47.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:47.656+0000 7f5d39111640 1 -- 172.21.15.27:0/1122958940 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d341220c0 con 0x7f5d34132520 2024-09-16T07:59:47.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:47.657+0000 7f5d39111640 1 -- 172.21.15.27:0/1122958940 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5d34120f90 con 0x7f5d34132520 2024-09-16T07:59:47.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:47.657+0000 7f5d2b7fe640 1 -- 172.21.15.27:0/1122958940 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) ==== 50954+0+0 (crc 0 0 0) 0x7f5d24007320 con 0x7f5d34132520 2024-09-16T07:59:47.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:47.657+0000 7f5d2b7fe640 1 --2- 172.21.15.27:0/1122958940 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7f5d1403e8e0 0x7f5d14040df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:59:47.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:47.657+0000 7f5d2b7fe640 1 -- 172.21.15.27:0/1122958940 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(228..228 src has 1..228) ==== 2700+0+0 (crc 0 0 0) 0x7f5d2404f6a0 con 0x7f5d34132520 2024-09-16T07:59:47.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:47.658+0000 7f5d3259c640 1 --2- 172.21.15.27:0/1122958940 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7f5d1403e8e0 0x7f5d14040df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:59:47.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:47.658+0000 7f5d3259c640 1 --2- 172.21.15.27:0/1122958940 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7f5d1403e8e0 0x7f5d14040df0 crc :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:59:47.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:47.663+0000 7f5d2b7fe640 1 -- 172.21.15.27:0/1122958940 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (crc 0 0 0) 0x7f5d2400a8d0 con 0x7f5d34132520 2024-09-16T07:59:47.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:47.847+0000 7f5d39111640 1 -- 172.21.15.27:0/1122958940 --> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f5d3411efc0 con 0x7f5d1403e8e0 2024-09-16T07:59:47.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:47.849+0000 7f5d2b7fe640 1 -- 172.21.15.27:0/1122958940 <== mgr.4102 v2:172.21.15.187:6800/4038049593 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+31239 (crc 0 0 0) 0x7f5d3411efc0 con 0x7f5d1403e8e0 2024-09-16T07:59:47.848 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-16T07:59:47.848 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-09-16T07:59:47.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:47.851+0000 7f5d39111640 1 -- 172.21.15.27:0/1122958940 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7f5d1403e8e0 msgr2=0x7f5d14040df0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:59:47.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:47.851+0000 7f5d39111640 1 --2- 172.21.15.27:0/1122958940 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7f5d1403e8e0 0x7f5d14040df0 crc :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:59:47.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:47.851+0000 7f5d39111640 1 -- 172.21.15.27:0/1122958940 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d34132520 msgr2=0x7f5d3411f580 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:59:47.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:47.851+0000 7f5d39111640 1 --2- 172.21.15.27:0/1122958940 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d34132520 0x7f5d3411f580 crc :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:59:47.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:47.851+0000 7f5d39111640 1 -- 172.21.15.27:0/1122958940 shutdown_connections 2024-09-16T07:59:47.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:47.851+0000 7f5d39111640 1 --2- 172.21.15.27:0/1122958940 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7f5d1403e8e0 0x7f5d14040df0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:59:47.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:47.851+0000 7f5d39111640 1 --2- 172.21.15.27:0/1122958940 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d34132520 0x7f5d3411f580 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:59:47.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:47.851+0000 7f5d39111640 1 -- 172.21.15.27:0/1122958940 >> 172.21.15.27:0/1122958940 conn(0x7f5d3406e490 msgr2=0x7f5d340a3190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:59:47.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:47.851+0000 7f5d39111640 1 -- 172.21.15.27:0/1122958940 shutdown_connections 2024-09-16T07:59:47.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T07:59:47.851+0000 7f5d39111640 1 -- 172.21.15.27:0/1122958940 wait complete. 2024-09-16T07:59:47.861 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":384,"stamp":"2024-09-16T07:59:47.153590+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":209190912,"kb_used":15568,"kb_used_data":15568,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":209175344,"statfs":{"total":214211493888,"available":214195552256,"internally_reserved":0,"allocated":15941632,"data_stored":15941632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002905"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":645,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-09-16T07:59:18.795976+0000","last_change":"2024-09-16T07:54:26.740923+0000","last_active":"2024-09-16T07:59:18.795976+0000","last_peered":"2024-09-16T07:59:18.795976+0000","last_clean":"2024-09-16T07:59:18.795976+0000","last_became_active":"2024-09-16T07:54:24.784360+0000","last_became_peered":"2024-09-16T07:54:24.784360+0000","last_unstale":"2024-09-16T07:59:18.795976+0000","last_undegraded":"2024-09-16T07:59:18.795976+0000","last_fullsized":"2024-09-16T07:59:18.795976+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T07:54:23.687651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T07:54:23.687651+0000","last_clean_scrub_stamp":"2024-09-16T07:54:23.687651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":645,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-09-16T07:59:18.761001+0000","last_change":"2024-09-16T07:54:26.739464+0000","last_active":"2024-09-16T07:59:18.761001+0000","last_peered":"2024-09-16T07:59:18.761001+0000","last_clean":"2024-09-16T07:59:18.761001+0000","last_became_active":"2024-09-16T07:54:24.805913+0000","last_became_peered":"2024-09-16T07:54:24.805913+0000","last_unstale":"2024-09-16T07:59:18.761001+0000","last_undegraded":"2024-09-16T07:59:18.761001+0000","last_fullsized":"2024-09-16T07:59:18.761001+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T07:54:23.687651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T07:54:23.687651+0000","last_clean_scrub_stamp":"2024-09-16T07:54:23.687651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":645,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-09-16T07:59:18.782573+0000","last_change":"2024-09-16T07:54:26.735332+0000","last_active":"2024-09-16T07:59:18.782573+0000","last_peered":"2024-09-16T07:59:18.782573+0000","last_clean":"2024-09-16T07:59:18.782573+0000","last_became_active":"2024-09-16T07:54:24.773365+0000","last_became_peered":"2024-09-16T07:54:24.773365+0000","last_unstale":"2024-09-16T07:59:18.782573+0000","last_undegraded":"2024-09-16T07:59:18.782573+0000","last_fullsized":"2024-09-16T07:59:18.782573+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T07:54:23.687651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T07:54:23.687651+0000","last_clean_scrub_stamp":"2024-09-16T07:54:23.687651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":645,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-09-16T07:59:18.751250+0000","last_change":"2024-09-16T07:54:26.734682+0000","last_active":"2024-09-16T07:59:18.751250+0000","last_peered":"2024-09-16T07:59:18.751250+0000","last_clean":"2024-09-16T07:59:18.751250+0000","last_became_active":"2024-09-16T07:54:24.756974+0000","last_became_peered":"2024-09-16T07:54:24.756974+0000","last_unstale":"2024-09-16T07:59:18.751250+0000","last_undegraded":"2024-09-16T07:59:18.751250+0000","last_fullsized":"2024-09-16T07:59:18.751250+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T07:54:23.687651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T07:54:23.687651+0000","last_clean_scrub_stamp":"2024-09-16T07:54:23.687651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"15'2","reported_seq":647,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-09-16T07:59:18.780803+0000","last_change":"2024-09-16T07:54:26.741927+0000","last_active":"2024-09-16T07:59:18.780803+0000","last_peered":"2024-09-16T07:59:18.780803+0000","last_clean":"2024-09-16T07:59:18.780803+0000","last_became_active":"2024-09-16T07:54:24.794754+0000","last_became_peered":"2024-09-16T07:54:24.794754+0000","last_unstale":"2024-09-16T07:59:18.780803+0000","last_undegraded":"2024-09-16T07:59:18.780803+0000","last_fullsized":"2024-09-16T07:59:18.780803+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T07:54:23.687651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T07:54:23.687651+0000","last_clean_scrub_stamp":"2024-09-16T07:54:23.687651+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":645,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-09-16T07:59:18.758597+0000","last_change":"2024-09-16T07:54:26.734478+0000","last_active":"2024-09-16T07:59:18.758597+0000","last_peered":"2024-09-16T07:59:18.758597+0000","last_clean":"2024-09-16T07:59:18.758597+0000","last_became_active":"2024-09-16T07:54:24.774923+0000","last_became_peered":"2024-09-16T07:54:24.774923+0000","last_unstale":"2024-09-16T07:59:18.758597+0000","last_undegraded":"2024-09-16T07:59:18.758597+0000","last_fullsized":"2024-09-16T07:59:18.758597+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T07:54:23.687651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T07:54:23.687651+0000","last_clean_scrub_stamp":"2024-09-16T07:54:23.687651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":645,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-09-16T07:59:18.810530+0000","last_change":"2024-09-16T07:54:26.742127+0000","last_active":"2024-09-16T07:59:18.810530+0000","last_peered":"2024-09-16T07:59:18.810530+0000","last_clean":"2024-09-16T07:59:18.810530+0000","last_became_active":"2024-09-16T07:54:24.773472+0000","last_became_peered":"2024-09-16T07:54:24.773472+0000","last_unstale":"2024-09-16T07:59:18.810530+0000","last_undegraded":"2024-09-16T07:59:18.810530+0000","last_fullsized":"2024-09-16T07:59:18.810530+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T07:54:23.687651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T07:54:23.687651+0000","last_clean_scrub_stamp":"2024-09-16T07:54:23.687651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"13'1","reported_seq":646,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-09-16T07:59:18.805158+0000","last_change":"2024-09-16T07:54:26.739091+0000","last_active":"2024-09-16T07:59:18.805158+0000","last_peered":"2024-09-16T07:59:18.805158+0000","last_clean":"2024-09-16T07:59:18.805158+0000","last_became_active":"2024-09-16T07:54:24.801907+0000","last_became_peered":"2024-09-16T07:54:24.801907+0000","last_unstale":"2024-09-16T07:59:18.805158+0000","last_undegraded":"2024-09-16T07:59:18.805158+0000","last_fullsized":"2024-09-16T07:59:18.805158+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T07:54:23.687651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T07:54:23.687651+0000","last_clean_scrub_stamp":"2024-09-16T07:54:23.687651+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'52","reported_seq":717,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-09-16T07:59:18.755371+0000","last_change":"2024-09-16T07:54:22.755090+0000","last_active":"2024-09-16T07:59:18.755371+0000","last_peered":"2024-09-16T07:59:18.755371+0000","last_clean":"2024-09-16T07:59:18.755371+0000","last_became_active":"2024-09-16T07:54:22.736266+0000","last_became_peered":"2024-09-16T07:54:22.736266+0000","last_unstale":"2024-09-16T07:59:18.755371+0000","last_undegraded":"2024-09-16T07:59:18.755371+0000","last_fullsized":"2024-09-16T07:59:18.755371+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T07:54:21.678374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T07:54:21.678374+0000","last_clean_scrub_stamp":"2024-09-16T07:54:21.678374+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":214211493888,"available":214198435840,"internally_reserved":0,"allocated":13058048,"data_stored":13058048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53552873472,"available":53549346816,"internally_reserved":0,"allocated":3526656,"data_stored":3526656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705963,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":4340,"kb_used_data":4340,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52293388,"statfs":{"total":53552873472,"available":53548429312,"internally_reserved":0,"allocated":4444160,"data_stored":4444160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705963,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":3300,"kb_used_data":3300,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52294428,"statfs":{"total":53552873472,"available":53549494272,"internally_reserved":0,"allocated":3379200,"data_stored":3379200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705963,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":3376,"kb_used_data":3376,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52294352,"statfs":{"total":53552873472,"available":53549416448,"internally_reserved":0,"allocated":3457024,"data_stored":3457024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705963,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":4552,"kb_used_data":4552,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52293176,"statfs":{"total":53552873472,"available":53548212224,"internally_reserved":0,"allocated":4661248,"data_stored":4661248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":3,"total":53552873472,"available":53549346816,"internally_reserved":0,"allocated":3526656,"data_stored":3526656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53552873472,"available":53549064192,"internally_reserved":0,"allocated":3809280,"data_stored":3809280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53552873472,"available":53550006272,"internally_reserved":0,"allocated":2867200,"data_stored":2867200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53552873472,"available":53550018560,"internally_reserved":0,"allocated":2854912,"data_stored":2854912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":53552873472,"available":53549346816,"internally_reserved":0,"allocated":3526656,"data_stored":3526656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T07:59:47.862 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-09-16T07:54:23.687651+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=7, tm_min=54, tm_sec=23, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=7, tm_min=59, tm_sec=24, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T07:59:47.862 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-09-16T07:54:23.687651+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=7, tm_min=54, tm_sec=23, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=7, tm_min=59, tm_sec=24, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T07:59:47.862 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-09-16T07:54:23.687651+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=7, tm_min=54, tm_sec=23, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=7, tm_min=59, tm_sec=24, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T07:59:47.863 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-09-16T07:54:23.687651+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=7, tm_min=54, tm_sec=23, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=7, tm_min=59, tm_sec=24, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T07:59:47.863 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-09-16T07:54:23.687651+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=7, tm_min=54, tm_sec=23, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=7, tm_min=59, tm_sec=24, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T07:59:47.863 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-09-16T07:54:23.687651+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=7, tm_min=54, tm_sec=23, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=7, tm_min=59, tm_sec=24, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T07:59:47.863 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-09-16T07:54:23.687651+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=7, tm_min=54, tm_sec=23, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=7, tm_min=59, tm_sec=24, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T07:59:47.863 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-09-16T07:54:23.687651+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=7, tm_min=54, tm_sec=23, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=7, tm_min=59, tm_sec=24, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T07:59:47.863 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-09-16T07:54:21.678374+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=7, tm_min=54, tm_sec=21, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=7, tm_min=59, tm_sec=24, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T07:59:47.864 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-09-16T08:00:07.864 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T08:00:07.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T08:00:07.989+0000 7f3f34b11640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T08:00:07.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T08:00:07.995+0000 7f3f34b11640 1 Processor -- start 2024-09-16T08:00:07.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T08:00:07.995+0000 7f3f34b11640 1 -- start start 2024-09-16T08:00:07.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T08:00:07.997+0000 7f3f34b11640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3f30138720 0x7f3f3013ab40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:00:07.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T08:00:07.997+0000 7f3f34b11640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3f30133040 con 0x7f3f30132520 2024-09-16T08:00:07.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T08:00:07.997+0000 7f3f34b11640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 -- 0x7f3f3006f6d0 con 0x7f3f30138720 2024-09-16T08:00:07.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T08:00:07.997+0000 7f3f2dd9b640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3f30138720 0x7f3f3013ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:00:07.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T08:00:07.997+0000 7f3f2e59c640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f3f30132520 0x7f3f30132930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:50004/0 (socket says 172.21.15.27:50004) 2024-09-16T08:00:07.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T08:00:07.997+0000 7f3f2dd9b640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3f30138720 0x7f3f3013ab40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35962/0 (socket says 172.21.15.27:35962) 2024-09-16T08:00:07.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T08:00:07.997+0000 7f3f2e59c640 1 -- 172.21.15.27:0/3970465207 learned_addr learned my addr 172.21.15.27:0/3970465207 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-09-16T08:00:07.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T08:00:07.998+0000 7f3f2dd9b640 1 -- 172.21.15.27:0/3970465207 >> v1:172.21.15.27:6789/0 conn(0x7f3f30132520 legacy=0x7f3f30132930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:00:07.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T08:00:07.998+0000 7f3f2dd9b640 1 -- 172.21.15.27:0/3970465207 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f3006f3d0 con 0x7f3f30138720 2024-09-16T08:00:07.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T08:00:07.998+0000 7f3f2dd9b640 1 --2- 172.21.15.27:0/3970465207 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f30138720 0x7f3f3013ab40 crc :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51be3640f6c04c3c server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:00:07.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T08:00:07.998+0000 7f3f2d59a640 1 -- 172.21.15.27:0/3970465207 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f3f18009d90 con 0x7f3f30138720 2024-09-16T08:00:07.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T08:00:07.998+0000 7f3f2d59a640 1 -- 172.21.15.27:0/3970465207 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f3f1800e230 con 0x7f3f30138720 2024-09-16T08:00:07.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T08:00:07.998+0000 7f3f2d59a640 1 -- 172.21.15.27:0/3970465207 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f3f1800e4d0 con 0x7f3f30138720 2024-09-16T08:00:08.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T08:00:07.999+0000 7f3f34b11640 1 -- 172.21.15.27:0/3970465207 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f30138720 msgr2=0x7f3f3013ab40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:00:08.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T08:00:07.999+0000 7f3f34b11640 1 --2- 172.21.15.27:0/3970465207 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f30138720 0x7f3f3013ab40 crc :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:00:08.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T08:00:07.999+0000 7f3f34b11640 1 -- 172.21.15.27:0/3970465207 shutdown_connections 2024-09-16T08:00:08.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T08:00:07.999+0000 7f3f34b11640 1 --2- 172.21.15.27:0/3970465207 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f30138720 0x7f3f3013ab40 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:00:08.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T08:00:07.999+0000 7f3f34b11640 1 -- 172.21.15.27:0/3970465207 >> 172.21.15.27:0/3970465207 conn(0x7f3f3006e490 msgr2=0x7f3f3006e8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T08:00:08.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T08:00:07.999+0000 7f3f34b11640 1 -- 172.21.15.27:0/3970465207 shutdown_connections 2024-09-16T08:00:08.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T08:00:07.999+0000 7f3f34b11640 1 -- 172.21.15.27:0/3970465207 wait complete. 2024-09-16T08:00:08.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T08:00:07.999+0000 7f3f34b11640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T08:00:08.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T08:00:08.001+0000 7f3f34b11640 1 Processor -- start 2024-09-16T08:00:08.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T08:00:08.001+0000 7f3f34b11640 1 -- start start 2024-09-16T08:00:08.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T08:00:08.003+0000 7f3f34b11640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f30132520 0x7f3f301c9f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:00:08.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T08:00:08.003+0000 7f3f34b11640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f3f300a03b0 con 0x7f3f30132520 2024-09-16T08:00:08.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T08:00:08.003+0000 7f3f2e59c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f30132520 0x7f3f301c9f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:00:08.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T08:00:08.003+0000 7f3f2e59c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f30132520 0x7f3f301c9f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35972/0 (socket says 172.21.15.27:35972) 2024-09-16T08:00:08.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T08:00:08.003+0000 7f3f2e59c640 1 -- 172.21.15.27:0/1953606995 learned_addr learned my addr 172.21.15.27:0/1953606995 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-09-16T08:00:08.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T08:00:08.003+0000 7f3f2e59c640 1 -- 172.21.15.27:0/1953606995 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f3009ce40 con 0x7f3f30132520 2024-09-16T08:00:08.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T08:00:08.003+0000 7f3f2e59c640 1 --2- 172.21.15.27:0/1953606995 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f30132520 0x7f3f301c9f00 crc :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:00:08.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T08:00:08.003+0000 7f3f26ffd640 1 -- 172.21.15.27:0/1953606995 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f3f200070b0 con 0x7f3f30132520 2024-09-16T08:00:08.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T08:00:08.003+0000 7f3f26ffd640 1 -- 172.21.15.27:0/1953606995 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f3f20007210 con 0x7f3f30132520 2024-09-16T08:00:08.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T08:00:08.003+0000 7f3f26ffd640 1 -- 172.21.15.27:0/1953606995 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f3f200074b0 con 0x7f3f30132520 2024-09-16T08:00:08.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T08:00:08.003+0000 7f3f34b11640 1 -- 172.21.15.27:0/1953606995 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f30120b90 con 0x7f3f30132520 2024-09-16T08:00:08.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T08:00:08.003+0000 7f3f34b11640 1 -- 172.21.15.27:0/1953606995 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f30120010 con 0x7f3f30132520 2024-09-16T08:00:08.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T08:00:08.005+0000 7f3f34b11640 1 -- 172.21.15.27:0/1953606995 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3f3006a300 con 0x7f3f30132520 2024-09-16T08:00:08.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T08:00:08.005+0000 7f3f26ffd640 1 -- 172.21.15.27:0/1953606995 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) ==== 50954+0+0 (crc 0 0 0) 0x7f3f20007660 con 0x7f3f30132520 2024-09-16T08:00:08.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T08:00:08.005+0000 7f3f26ffd640 1 --2- 172.21.15.27:0/1953606995 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7f3f1003e930 0x7f3f10040e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:00:08.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T08:00:08.005+0000 7f3f26ffd640 1 -- 172.21.15.27:0/1953606995 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(228..228 src has 1..228) ==== 2700+0+0 (crc 0 0 0) 0x7f3f2004fee0 con 0x7f3f30132520 2024-09-16T08:00:08.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T08:00:08.006+0000 7f3f2dd9b640 1 --2- 172.21.15.27:0/1953606995 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7f3f1003e930 0x7f3f10040e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:00:08.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T08:00:08.006+0000 7f3f2dd9b640 1 --2- 172.21.15.27:0/1953606995 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7f3f1003e930 0x7f3f10040e40 crc :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:00:08.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T08:00:08.010+0000 7f3f26ffd640 1 -- 172.21.15.27:0/1953606995 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (crc 0 0 0) 0x7f3f200096b0 con 0x7f3f30132520 2024-09-16T08:00:08.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T08:00:08.194+0000 7f3f34b11640 1 -- 172.21.15.27:0/1953606995 --> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f3f300a0780 con 0x7f3f1003e930 2024-09-16T08:00:08.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T08:00:08.195+0000 7f3f26ffd640 1 -- 172.21.15.27:0/1953606995 <== mgr.4102 v2:172.21.15.187:6800/4038049593 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+31243 (crc 0 0 0) 0x7f3f300a0780 con 0x7f3f1003e930 2024-09-16T08:00:08.195 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-16T08:00:08.196 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-09-16T08:00:08.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T08:00:08.198+0000 7f3f34b11640 1 -- 172.21.15.27:0/1953606995 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7f3f1003e930 msgr2=0x7f3f10040e40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:00:08.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T08:00:08.198+0000 7f3f34b11640 1 --2- 172.21.15.27:0/1953606995 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7f3f1003e930 0x7f3f10040e40 crc :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:00:08.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T08:00:08.198+0000 7f3f34b11640 1 -- 172.21.15.27:0/1953606995 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f30132520 msgr2=0x7f3f301c9f00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:00:08.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T08:00:08.198+0000 7f3f34b11640 1 --2- 172.21.15.27:0/1953606995 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f30132520 0x7f3f301c9f00 crc :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:00:08.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T08:00:08.198+0000 7f3f34b11640 1 -- 172.21.15.27:0/1953606995 shutdown_connections 2024-09-16T08:00:08.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T08:00:08.198+0000 7f3f34b11640 1 --2- 172.21.15.27:0/1953606995 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7f3f1003e930 0x7f3f10040e40 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:00:08.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T08:00:08.198+0000 7f3f34b11640 1 --2- 172.21.15.27:0/1953606995 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f30132520 0x7f3f301c9f00 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:00:08.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T08:00:08.198+0000 7f3f34b11640 1 -- 172.21.15.27:0/1953606995 >> 172.21.15.27:0/1953606995 conn(0x7f3f3006e490 msgr2=0x7f3f301bd840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T08:00:08.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T08:00:08.198+0000 7f3f34b11640 1 -- 172.21.15.27:0/1953606995 shutdown_connections 2024-09-16T08:00:08.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T08:00:08.198+0000 7f3f34b11640 1 -- 172.21.15.27:0/1953606995 wait complete. 2024-09-16T08:00:08.208 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":394,"stamp":"2024-09-16T08:00:07.158390+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":209190912,"kb_used":12884,"kb_used_data":12884,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":209178028,"statfs":{"total":214211493888,"available":214198300672,"internally_reserved":0,"allocated":13193216,"data_stored":13193216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002880"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":656,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T07:59:26.200802+0000","last_change":"2024-09-16T07:59:26.218878+0000","last_active":"2024-09-16T07:59:26.200802+0000","last_peered":"2024-09-16T07:59:26.200802+0000","last_clean":"2024-09-16T07:59:26.200802+0000","last_became_active":"2024-09-16T07:54:24.784360+0000","last_became_peered":"2024-09-16T07:54:24.784360+0000","last_unstale":"2024-09-16T07:59:26.200802+0000","last_undegraded":"2024-09-16T07:59:26.200802+0000","last_fullsized":"2024-09-16T07:59:26.200802+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T07:59:26.218596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T07:59:26.218596+0000","last_clean_scrub_stamp":"2024-09-16T07:54:23.687651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":656,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T07:59:26.694728+0000","last_change":"2024-09-16T07:59:26.710052+0000","last_active":"2024-09-16T07:59:26.694728+0000","last_peered":"2024-09-16T07:59:26.694728+0000","last_clean":"2024-09-16T07:59:26.694728+0000","last_became_active":"2024-09-16T07:54:24.805913+0000","last_became_peered":"2024-09-16T07:54:24.805913+0000","last_unstale":"2024-09-16T07:59:26.694728+0000","last_undegraded":"2024-09-16T07:59:26.694728+0000","last_fullsized":"2024-09-16T07:59:26.694728+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T07:59:26.709864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T07:59:26.709864+0000","last_clean_scrub_stamp":"2024-09-16T07:54:23.687651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":656,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T07:59:27.179317+0000","last_change":"2024-09-16T07:59:27.193066+0000","last_active":"2024-09-16T07:59:27.179317+0000","last_peered":"2024-09-16T07:59:27.179317+0000","last_clean":"2024-09-16T07:59:27.179317+0000","last_became_active":"2024-09-16T07:54:24.773365+0000","last_became_peered":"2024-09-16T07:54:24.773365+0000","last_unstale":"2024-09-16T07:59:27.179317+0000","last_undegraded":"2024-09-16T07:59:27.179317+0000","last_fullsized":"2024-09-16T07:59:27.179317+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T07:59:27.192856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T07:59:27.192856+0000","last_clean_scrub_stamp":"2024-09-16T07:54:23.687651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":656,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T07:59:26.692800+0000","last_change":"2024-09-16T07:59:26.708168+0000","last_active":"2024-09-16T07:59:26.692800+0000","last_peered":"2024-09-16T07:59:26.692800+0000","last_clean":"2024-09-16T07:59:26.692800+0000","last_became_active":"2024-09-16T07:54:24.756974+0000","last_became_peered":"2024-09-16T07:54:24.756974+0000","last_unstale":"2024-09-16T07:59:26.692800+0000","last_undegraded":"2024-09-16T07:59:26.692800+0000","last_fullsized":"2024-09-16T07:59:26.692800+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T07:59:26.707947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T07:59:26.707947+0000","last_clean_scrub_stamp":"2024-09-16T07:54:23.687651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"15'2","reported_seq":658,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T07:59:25.711222+0000","last_change":"2024-09-16T07:59:25.736870+0000","last_active":"2024-09-16T07:59:25.711222+0000","last_peered":"2024-09-16T07:59:25.711222+0000","last_clean":"2024-09-16T07:59:25.711222+0000","last_became_active":"2024-09-16T07:54:24.794754+0000","last_became_peered":"2024-09-16T07:54:24.794754+0000","last_unstale":"2024-09-16T07:59:25.711222+0000","last_undegraded":"2024-09-16T07:59:25.711222+0000","last_fullsized":"2024-09-16T07:59:25.711222+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"15'2","last_scrub_stamp":"2024-09-16T07:59:25.736670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T07:59:25.736670+0000","last_clean_scrub_stamp":"2024-09-16T07:54:23.687651+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":656,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T07:59:26.197963+0000","last_change":"2024-09-16T07:59:26.210926+0000","last_active":"2024-09-16T07:59:26.197963+0000","last_peered":"2024-09-16T07:59:26.197963+0000","last_clean":"2024-09-16T07:59:26.197963+0000","last_became_active":"2024-09-16T07:54:24.774923+0000","last_became_peered":"2024-09-16T07:54:24.774923+0000","last_unstale":"2024-09-16T07:59:26.197963+0000","last_undegraded":"2024-09-16T07:59:26.197963+0000","last_fullsized":"2024-09-16T07:59:26.197963+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T07:59:26.210744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T07:59:26.210744+0000","last_clean_scrub_stamp":"2024-09-16T07:54:23.687651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":656,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T07:59:25.710069+0000","last_change":"2024-09-16T07:59:25.725482+0000","last_active":"2024-09-16T07:59:25.710069+0000","last_peered":"2024-09-16T07:59:25.710069+0000","last_clean":"2024-09-16T07:59:25.710069+0000","last_became_active":"2024-09-16T07:54:24.773472+0000","last_became_peered":"2024-09-16T07:54:24.773472+0000","last_unstale":"2024-09-16T07:59:25.710069+0000","last_undegraded":"2024-09-16T07:59:25.710069+0000","last_fullsized":"2024-09-16T07:59:25.710069+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T07:59:25.725298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T07:59:25.725298+0000","last_clean_scrub_stamp":"2024-09-16T07:54:23.687651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"13'1","reported_seq":657,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T07:59:26.200568+0000","last_change":"2024-09-16T07:59:26.219160+0000","last_active":"2024-09-16T07:59:26.200568+0000","last_peered":"2024-09-16T07:59:26.200568+0000","last_clean":"2024-09-16T07:59:26.200568+0000","last_became_active":"2024-09-16T07:54:24.801907+0000","last_became_peered":"2024-09-16T07:54:24.801907+0000","last_unstale":"2024-09-16T07:59:26.200568+0000","last_undegraded":"2024-09-16T07:59:26.200568+0000","last_fullsized":"2024-09-16T07:59:26.200568+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"13'1","last_scrub_stamp":"2024-09-16T07:59:26.218970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T07:59:26.218970+0000","last_clean_scrub_stamp":"2024-09-16T07:54:23.687651+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'52","reported_seq":728,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T07:59:27.176729+0000","last_change":"2024-09-16T07:59:27.208944+0000","last_active":"2024-09-16T07:59:27.176729+0000","last_peered":"2024-09-16T07:59:27.176729+0000","last_clean":"2024-09-16T07:59:27.176729+0000","last_became_active":"2024-09-16T07:54:22.736266+0000","last_became_peered":"2024-09-16T07:54:22.736266+0000","last_unstale":"2024-09-16T07:59:27.176729+0000","last_undegraded":"2024-09-16T07:59:27.176729+0000","last_fullsized":"2024-09-16T07:59:27.176729+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"12'52","last_scrub_stamp":"2024-09-16T07:59:27.208711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T07:59:27.208711+0000","last_clean_scrub_stamp":"2024-09-16T07:54:21.678374+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":214211493888,"available":214198300672,"internally_reserved":0,"allocated":13193216,"data_stored":13193216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53552873472,"available":53549318144,"internally_reserved":0,"allocated":3555328,"data_stored":3555328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705993,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":3472,"kb_used_data":3472,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52294256,"statfs":{"total":53552873472,"available":53549318144,"internally_reserved":0,"allocated":3555328,"data_stored":3555328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705993,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":2836,"kb_used_data":2836,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52294892,"statfs":{"total":53552873472,"available":53549969408,"internally_reserved":0,"allocated":2904064,"data_stored":2904064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705993,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":2828,"kb_used_data":2828,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52294900,"statfs":{"total":53552873472,"available":53549977600,"internally_reserved":0,"allocated":2895872,"data_stored":2895872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705993,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":3748,"kb_used_data":3748,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52293980,"statfs":{"total":53552873472,"available":53549035520,"internally_reserved":0,"allocated":3837952,"data_stored":3837952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":3,"total":53552873472,"available":53549318144,"internally_reserved":0,"allocated":3555328,"data_stored":3555328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53552873472,"available":53549035520,"internally_reserved":0,"allocated":3837952,"data_stored":3837952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53552873472,"available":53549977600,"internally_reserved":0,"allocated":2895872,"data_stored":2895872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53552873472,"available":53549969408,"internally_reserved":0,"allocated":2904064,"data_stored":2904064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":53552873472,"available":53549318144,"internally_reserved":0,"allocated":3555328,"data_stored":3555328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T08:00:08.210 DEBUG:teuthology.orchestra.run.smithi027:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-09-16T08:00:08.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T08:00:08.317+0000 7f1453991640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T08:00:08.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T08:00:08.326+0000 7f1453991640 1 Processor -- start 2024-09-16T08:00:08.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T08:00:08.326+0000 7f1453991640 1 -- start start 2024-09-16T08:00:08.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T08:00:08.327+0000 7f1453991640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f144c0718e0 0x7f144c071cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:00:08.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T08:00:08.327+0000 7f1453991640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f144c0a4cb0 con 0x7f144c072230 2024-09-16T08:00:08.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T08:00:08.327+0000 7f1453991640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 -- 0x7f144c074bb0 con 0x7f144c0718e0 2024-09-16T08:00:08.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T08:00:08.328+0000 7f145172d640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f144c0718e0 0x7f144c071cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:00:08.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T08:00:08.328+0000 7f1450f2c640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f144c072230 0x7f144c133170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:33494/0 (socket says 172.21.15.27:33494) 2024-09-16T08:00:08.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T08:00:08.328+0000 7f145172d640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f144c0718e0 0x7f144c071cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54210/0 (socket says 172.21.15.27:54210) 2024-09-16T08:00:08.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T08:00:08.328+0000 7f1450f2c640 1 -- 172.21.15.27:0/3552020346 learned_addr learned my addr 172.21.15.27:0/3552020346 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-09-16T08:00:08.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T08:00:08.328+0000 7f145172d640 1 -- 172.21.15.27:0/3552020346 >> v1:172.21.15.27:6789/0 conn(0x7f144c072230 legacy=0x7f144c133170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:00:08.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T08:00:08.328+0000 7f145172d640 1 -- 172.21.15.27:0/3552020346 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f144c09bbd0 con 0x7f144c0718e0 2024-09-16T08:00:08.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T08:00:08.328+0000 7f143ffff640 1 -- 172.21.15.27:0/3552020346 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3355232087 0 0) 0x7f144c0a4cb0 con 0x7f144c072230 2024-09-16T08:00:08.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T08:00:08.328+0000 7f145172d640 1 --2- 172.21.15.27:0/3552020346 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f144c0718e0 0x7f144c071cf0 crc :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4bf48b611804418c server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:00:08.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T08:00:08.328+0000 7f143ffff640 1 -- 172.21.15.27:0/3552020346 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f1448009d90 con 0x7f144c0718e0 2024-09-16T08:00:08.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T08:00:08.328+0000 7f143ffff640 1 -- 172.21.15.27:0/3552020346 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f144800e230 con 0x7f144c0718e0 2024-09-16T08:00:08.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T08:00:08.328+0000 7f143ffff640 1 -- 172.21.15.27:0/3552020346 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f144800e4d0 con 0x7f144c0718e0 2024-09-16T08:00:08.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T08:00:08.329+0000 7f1453991640 1 -- 172.21.15.27:0/3552020346 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f144c0718e0 msgr2=0x7f144c071cf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:00:08.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T08:00:08.329+0000 7f1453991640 1 --2- 172.21.15.27:0/3552020346 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f144c0718e0 0x7f144c071cf0 crc :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:00:08.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T08:00:08.329+0000 7f1453991640 1 -- 172.21.15.27:0/3552020346 shutdown_connections 2024-09-16T08:00:08.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T08:00:08.329+0000 7f1453991640 1 --2- 172.21.15.27:0/3552020346 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f144c0718e0 0x7f144c071cf0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:00:08.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T08:00:08.329+0000 7f1453991640 1 -- 172.21.15.27:0/3552020346 >> 172.21.15.27:0/3552020346 conn(0x7f144c0a2650 msgr2=0x7f144c0a2a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T08:00:08.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T08:00:08.329+0000 7f1453991640 1 -- 172.21.15.27:0/3552020346 shutdown_connections 2024-09-16T08:00:08.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T08:00:08.329+0000 7f1453991640 1 -- 172.21.15.27:0/3552020346 wait complete. 2024-09-16T08:00:08.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T08:00:08.329+0000 7f1453991640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-16T08:00:08.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T08:00:08.331+0000 7f1453991640 1 Processor -- start 2024-09-16T08:00:08.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T08:00:08.331+0000 7f1453991640 1 -- start start 2024-09-16T08:00:08.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T08:00:08.333+0000 7f1453991640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f144c0718e0 0x7f144c1214d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:00:08.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T08:00:08.333+0000 7f1453991640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f144c124bc0 con 0x7f144c0718e0 2024-09-16T08:00:08.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T08:00:08.333+0000 7f145172d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f144c0718e0 0x7f144c1214d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:00:08.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T08:00:08.333+0000 7f145172d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f144c0718e0 0x7f144c1214d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54224/0 (socket says 172.21.15.27:54224) 2024-09-16T08:00:08.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T08:00:08.333+0000 7f145172d640 1 -- 172.21.15.27:0/2062061295 learned_addr learned my addr 172.21.15.27:0/2062061295 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-09-16T08:00:08.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T08:00:08.333+0000 7f145172d640 1 -- 172.21.15.27:0/2062061295 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f144c1210a0 con 0x7f144c0718e0 2024-09-16T08:00:08.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T08:00:08.333+0000 7f145172d640 1 --2- 172.21.15.27:0/2062061295 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f144c0718e0 0x7f144c1214d0 crc :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:00:08.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T08:00:08.333+0000 7f143dffb640 1 -- 172.21.15.27:0/2062061295 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f1448014620 con 0x7f144c0718e0 2024-09-16T08:00:08.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T08:00:08.334+0000 7f143dffb640 1 -- 172.21.15.27:0/2062061295 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f1448014780 con 0x7f144c0718e0 2024-09-16T08:00:08.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T08:00:08.334+0000 7f1453991640 1 -- 172.21.15.27:0/2062061295 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f144c120d90 con 0x7f144c0718e0 2024-09-16T08:00:08.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T08:00:08.334+0000 7f143dffb640 1 -- 172.21.15.27:0/2062061295 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f1448014a20 con 0x7f144c0718e0 2024-09-16T08:00:08.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T08:00:08.334+0000 7f1453991640 1 -- 172.21.15.27:0/2062061295 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f144c123eb0 con 0x7f144c0718e0 2024-09-16T08:00:08.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T08:00:08.335+0000 7f143dffb640 1 -- 172.21.15.27:0/2062061295 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) ==== 50954+0+0 (crc 0 0 0) 0x7f1448014c20 con 0x7f144c0718e0 2024-09-16T08:00:08.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T08:00:08.335+0000 7f1453991640 1 -- 172.21.15.27:0/2062061295 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f144c071cf0 con 0x7f144c0718e0 2024-09-16T08:00:08.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T08:00:08.335+0000 7f143dffb640 1 --2- 172.21.15.27:0/2062061295 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7f143003e8e0 0x7f1430040df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:00:08.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T08:00:08.335+0000 7f143dffb640 1 -- 172.21.15.27:0/2062061295 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(228..228 src has 1..228) ==== 2700+0+0 (crc 0 0 0) 0x7f1448019040 con 0x7f144c0718e0 2024-09-16T08:00:08.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T08:00:08.335+0000 7f1450f2c640 1 --2- 172.21.15.27:0/2062061295 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7f143003e8e0 0x7f1430040df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:00:08.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T08:00:08.336+0000 7f1450f2c640 1 --2- 172.21.15.27:0/2062061295 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7f143003e8e0 0x7f1430040df0 crc :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:00:08.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T08:00:08.340+0000 7f143dffb640 1 -- 172.21.15.27:0/2062061295 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (crc 0 0 0) 0x7f1448011380 con 0x7f144c0718e0 2024-09-16T08:00:08.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T08:00:08.525+0000 7f1453991640 1 -- 172.21.15.27:0/2062061295 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) -- 0x7f144c1235a0 con 0x7f144c0718e0 2024-09-16T08:00:08.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T08:00:08.528+0000 7f143dffb640 1 -- 172.21.15.27:0/2062061295 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v2)=0 v2) ==== 123+0+0 (crc 0 0 0) 0x7f1448008f20 con 0x7f144c0718e0 2024-09-16T08:00:08.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T08:00:08.529+0000 7f143dffb640 1 -- 172.21.15.27:0/2062061295 <== mon.0 v2:172.21.15.27:3300/0 8 ==== config(1 keys) ==== 35+0+0 (crc 0 0 0) 0x7f1448011380 con 0x7f144c0718e0 2024-09-16T08:00:08.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T08:00:08.531+0000 7f1453991640 1 -- 172.21.15.27:0/2062061295 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7f143003e8e0 msgr2=0x7f1430040df0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:00:08.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T08:00:08.531+0000 7f1453991640 1 --2- 172.21.15.27:0/2062061295 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7f143003e8e0 0x7f1430040df0 crc :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:00:08.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T08:00:08.531+0000 7f1453991640 1 -- 172.21.15.27:0/2062061295 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f144c0718e0 msgr2=0x7f144c1214d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:00:08.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T08:00:08.531+0000 7f1453991640 1 --2- 172.21.15.27:0/2062061295 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f144c0718e0 0x7f144c1214d0 crc :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:00:08.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T08:00:08.531+0000 7f1453991640 1 -- 172.21.15.27:0/2062061295 shutdown_connections 2024-09-16T08:00:08.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T08:00:08.531+0000 7f1453991640 1 --2- 172.21.15.27:0/2062061295 >> [v2:172.21.15.187:6800/4038049593,v1:172.21.15.187:6801/4038049593] conn(0x7f143003e8e0 0x7f1430040df0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:00:08.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T08:00:08.531+0000 7f1453991640 1 --2- 172.21.15.27:0/2062061295 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f144c0718e0 0x7f144c1214d0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:00:08.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T08:00:08.531+0000 7f1453991640 1 -- 172.21.15.27:0/2062061295 >> 172.21.15.27:0/2062061295 conn(0x7f144c0a2650 msgr2=0x7f144c0a11b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T08:00:08.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T08:00:08.531+0000 7f1453991640 1 -- 172.21.15.27:0/2062061295 shutdown_connections 2024-09-16T08:00:08.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-16T08:00:08.531+0000 7f1453991640 1 -- 172.21.15.27:0/2062061295 wait complete. 2024-09-16T08:00:08.545 INFO:teuthology.misc:Shutting down mds daemons... 2024-09-16T08:00:08.546 INFO:teuthology.misc:Shutting down osd daemons... 2024-09-16T08:00:08.546 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-09-16T08:00:08.546 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T08:00:08.633 INFO:tasks.ceph.osd.0:Stopped 2024-09-16T08:00:08.634 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-09-16T08:00:08.634 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T08:00:08.708 INFO:tasks.ceph.osd.1:Stopped 2024-09-16T08:00:08.709 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-09-16T08:00:08.709 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T08:00:08.784 INFO:tasks.ceph.osd.2:Stopped 2024-09-16T08:00:08.784 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-09-16T08:00:08.784 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T08:00:08.847 INFO:tasks.ceph.osd.3:Stopped 2024-09-16T08:00:08.848 INFO:teuthology.misc:Shutting down mgr daemons... 2024-09-16T08:00:08.848 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-09-16T08:00:08.848 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T08:00:08.877 INFO:tasks.ceph.mgr.x:Stopped 2024-09-16T08:00:08.878 INFO:teuthology.misc:Shutting down mon daemons... 2024-09-16T08:00:08.878 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-16T08:00:08.878 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T08:00:08.890 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~0s 2024-09-16T08:00:08.891 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~0s 2024-09-16T08:00:08.891 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~0s 2024-09-16T08:00:08.891 INFO:tasks.ceph.mon.a:Stopped 2024-09-16T08:00:08.891 INFO:tasks.ceph:Checking cluster log for badness... 2024-09-16T08:00:08.891 DEBUG:teuthology.orchestra.run.smithi027:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'reached quota' | egrep -v 'but it is still running' | egrep -v 'overall HEALTH_' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(PG_DEGRADED\)' | head -n 1 2024-09-16T08:00:08.967 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi027.front.sepia.ceph.com 2024-09-16T08:00:08.967 DEBUG:teuthology.orchestra.run.smithi027:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-09-16T08:00:09.157 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi027.front.sepia.ceph.com 2024-09-16T08:00:09.157 DEBUG:teuthology.orchestra.run.smithi027:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-09-16T08:00:09.281 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi187.front.sepia.ceph.com 2024-09-16T08:00:09.281 DEBUG:teuthology.orchestra.run.smithi187:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-09-16T08:00:09.360 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi187.front.sepia.ceph.com 2024-09-16T08:00:09.361 DEBUG:teuthology.orchestra.run.smithi187:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-09-16T08:00:09.482 INFO:tasks.ceph:Archiving mon data... 2024-09-16T08:00:09.483 DEBUG:teuthology.misc:Transferring archived files from smithi027:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/matan-2024-09-16_07:15:55-crimson-rados-wip-matanb-crimson-testing-59543-distro-crimson-smithi/7906886/data/mon.a.tgz 2024-09-16T08:00:09.483 DEBUG:teuthology.orchestra.run.smithi027:> mktemp 2024-09-16T08:00:09.500 INFO:teuthology.orchestra.run.smithi027.stdout:/tmp/tmp.FEN1MvDyOK 2024-09-16T08:00:09.501 DEBUG:teuthology.orchestra.run.smithi027:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.FEN1MvDyOK 2024-09-16T08:00:09.647 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod 0666 /tmp/tmp.FEN1MvDyOK 2024-09-16T08:00:09.733 DEBUG:teuthology.orchestra.remote:smithi027:/tmp/tmp.FEN1MvDyOK is 376KB 2024-09-16T08:00:09.805 DEBUG:teuthology.orchestra.run.smithi027:> rm -fr /tmp/tmp.FEN1MvDyOK 2024-09-16T08:00:09.820 INFO:tasks.ceph:Cleaning ceph cluster... 2024-09-16T08:00:09.821 DEBUG:teuthology.orchestra.run.smithi027:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-16T08:00:09.863 DEBUG:teuthology.orchestra.run.smithi187:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-16T08:00:09.942 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-09-16T08:00:09.959 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-09-16T08:00:09.959 INFO:tasks.ceph:Archiving crash dumps... 2024-09-16T08:00:09.961 DEBUG:teuthology.misc:Transferring archived files from smithi027:/var/lib/ceph/crash to /home/teuthworker/archive/matan-2024-09-16_07:15:55-crimson-rados-wip-matanb-crimson-testing-59543-distro-crimson-smithi/7906886/remote/smithi027/crash 2024-09-16T08:00:09.962 DEBUG:teuthology.orchestra.run.smithi027:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-16T08:00:10.007 DEBUG:teuthology.misc:Transferring archived files from smithi187:/var/lib/ceph/crash to /home/teuthworker/archive/matan-2024-09-16_07:15:55-crimson-rados-wip-matanb-crimson-testing-59543-distro-crimson-smithi/7906886/remote/smithi187/crash 2024-09-16T08:00:10.008 DEBUG:teuthology.orchestra.run.smithi187:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-16T08:00:10.046 INFO:tasks.ceph:Compressing logs... 2024-09-16T08:00:10.046 DEBUG:teuthology.orchestra.run.smithi027:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-16T08:00:10.051 DEBUG:teuthology.orchestra.run.smithi187:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-16T08:00:10.074 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-09-16T08:00:10.074 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31216.log 2024-09-16T08:00:10.075 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29046.log 2024-09-16T08:00:10.075 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.31216.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29926.log 2024-09-16T08:00:10.094 INFO:teuthology.orchestra.run.smithi027.stderr: 85.3% -- replaced with /var/log/ceph/ceph-client.admin.31216.log.gz 2024-09-16T08:00:10.094 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.29046.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28510.log 2024-09-16T08:00:10.094 INFO:teuthology.orchestra.run.smithi027.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.29046.log.gz 2024-09-16T08:00:10.094 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.29926.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-09-16T08:00:10.094 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.28510.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29196.log 2024-09-16T08:00:10.095 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-osd.1.log: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.28510.log.gz 2024-09-16T08:00:10.095 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28424.log 2024-09-16T08:00:10.095 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29154.log 2024-09-16T08:00:10.095 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.29196.log: /var/log/ceph/ceph-client.admin.28424.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31048.log 2024-09-16T08:00:10.095 INFO:teuthology.orchestra.run.smithi027.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.29196.log.gz 2024-09-16T08:00:10.095 INFO:teuthology.orchestra.run.smithi027.stderr: 84.8% -- replaced with /var/log/ceph/ceph-client.admin.28424.log.gz 2024-09-16T08:00:10.095 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.29154.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31384.log 2024-09-16T08:00:10.095 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-09-16T08:00:10.096 INFO:teuthology.orchestra.run.smithi027.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.29154.log.gz 2024-09-16T08:00:10.096 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.31048.log: /var/log/ceph/ceph-client.admin.31384.log: gzip -5 --verbose -- 85.5% /var/log/ceph/ceph-client.admin.28599.log -- replaced with /var/log/ceph/ceph-client.admin.31048.log.gz 2024-09-16T08:00:10.096 INFO:teuthology.orchestra.run.smithi027.stderr: 2024-09-16T08:00:10.096 INFO:teuthology.orchestra.run.smithi027.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.31384.log.gz 2024-09-16T08:00:10.096 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28692.log 2024-09-16T08:00:10.096 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28650.log 2024-09-16T08:00:10.096 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.28599.log: /var/log/ceph/ceph-client.admin.28692.log: gzip -5 --verbose -- 86.6% -- replaced with /var/log/ceph/ceph-client.admin.28599.log.gz 2024-09-16T08:00:10.097 INFO:teuthology.orchestra.run.smithi027.stderr: /var/log/ceph/ceph-client.admin.31344.log 2024-09-16T08:00:10.097 INFO:teuthology.orchestra.run.smithi027.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.28692.log.gz 2024-09-16T08:00:10.097 INFO:teuthology.orchestra.run.smithi027.stderr: 88.9%/var/log/ceph/ceph-client.admin.28650.log: -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-09-16T08:00:10.097 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29457.log 2024-09-16T08:00:10.097 INFO:teuthology.orchestra.run.smithi027.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.28650.log.gz 2024-09-16T08:00:10.097 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.31344.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.27390.log 2024-09-16T08:00:10.097 INFO:teuthology.orchestra.run.smithi027.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.31344.log.gz 2024-09-16T08:00:10.097 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.29457.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31174.log 2024-09-16T08:00:10.097 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph.tmp-client.admin.27390.log: 66.7% -- replaced with /var/log/ceph/ceph.tmp-client.admin.27390.log.gz 2024-09-16T08:00:10.098 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29045.log 2024-09-16T08:00:10.098 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.31174.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31090.log 2024-09-16T08:00:10.098 INFO:teuthology.orchestra.run.smithi027.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.31174.log.gz 2024-09-16T08:00:10.098 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.29045.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31132.log 2024-09-16T08:00:10.098 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.31090.log: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.29045.log.gz 2024-09-16T08:00:10.098 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29238.log 2024-09-16T08:00:10.098 INFO:teuthology.orchestra.run.smithi027.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.31090.log.gz/var/log/ceph/ceph-client.admin.31132.log: 2024-09-16T08:00:10.098 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-09-16T08:00:10.099 INFO:teuthology.orchestra.run.smithi027.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.31132.log.gz 2024-09-16T08:00:10.099 INFO:teuthology.orchestra.run.smithi027.stderr:gzip/var/log/ceph/ceph-client.admin.29238.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.31258.log 2024-09-16T08:00:10.099 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-mon.a.log: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.29238.log.gz 2024-09-16T08:00:10.099 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28734.log 2024-09-16T08:00:10.099 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.31258.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29043.log 2024-09-16T08:00:10.099 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.28734.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28877.log 2024-09-16T08:00:10.099 INFO:teuthology.orchestra.run.smithi027.stderr: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.31258.log.gz 2024-09-16T08:00:10.099 INFO:teuthology.orchestra.run.smithi027.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.28734.log.gz/var/log/ceph/ceph-client.admin.29043.log: 2024-09-16T08:00:10.099 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29044.log 2024-09-16T08:00:10.100 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.28877.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28452.log 2024-09-16T08:00:10.100 INFO:teuthology.orchestra.run.smithi027.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.29043.log.gz 2024-09-16T08:00:10.100 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.29044.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.28877.log.gz 2024-09-16T08:00:10.100 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-09-16T08:00:10.100 INFO:teuthology.orchestra.run.smithi027.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.29044.log.gz 2024-09-16T08:00:10.100 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28776.log 2024-09-16T08:00:10.100 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.28452.log: /var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30785.log 2024-09-16T08:00:10.101 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.28776.log: 84.7% -- replaced with /var/log/ceph/ceph-client.admin.28452.log.gz 2024-09-16T08:00:10.101 INFO:teuthology.orchestra.run.smithi027.stderr: 92.3% -- replaced with /var/log/ceph/ceph-client.admin.29457.log.gz 2024-09-16T08:00:10.101 INFO:teuthology.orchestra.run.smithi027.stderr:gzip 85.7% -- replaced with /var/log/ceph/ceph-client.admin.28776.log.gz 2024-09-16T08:00:10.101 INFO:teuthology.orchestra.run.smithi027.stderr: -5/var/log/ceph/ceph-client.admin.30785.log: --verbose -- /var/log/ceph/ceph-client.admin.30964.log 2024-09-16T08:00:10.101 INFO:teuthology.orchestra.run.smithi027.stderr: 87.3% -- replaced with /var/log/ceph/ceph.log.gz 2024-09-16T08:00:10.101 INFO:teuthology.orchestra.run.smithi027.stderr: 85.8%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28566.log 2024-09-16T08:00:10.101 INFO:teuthology.orchestra.run.smithi027.stderr: -- replaced with /var/log/ceph/ceph-client.admin.30785.log.gz 2024-09-16T08:00:10.101 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.30964.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28876.log 2024-09-16T08:00:10.102 INFO:teuthology.orchestra.run.smithi027.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.30964.log.gz 2024-09-16T08:00:10.102 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.28566.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31301.log 2024-09-16T08:00:10.102 INFO:teuthology.orchestra.run.smithi027.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.28566.log.gz 2024-09-16T08:00:10.102 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.28876.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28875.log 86.7% -- replaced with /var/log/ceph/ceph-client.admin.28876.log.gz 2024-09-16T08:00:10.102 INFO:teuthology.orchestra.run.smithi027.stderr: 2024-09-16T08:00:10.102 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.31301.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30922.log 2024-09-16T08:00:10.102 INFO:teuthology.orchestra.run.smithi027.stderr: 85.3% -- replaced with /var/log/ceph/ceph-client.admin.31301.log.gz 2024-09-16T08:00:10.102 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.28875.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30827.log 93.5% -- replaced with /var/log/ceph/ceph-client.admin.29926.log.gz 2024-09-16T08:00:10.103 INFO:teuthology.orchestra.run.smithi027.stderr: 2024-09-16T08:00:10.103 INFO:teuthology.orchestra.run.smithi027.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.28875.log.gz 2024-09-16T08:00:10.103 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.30922.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31006.log 2024-09-16T08:00:10.103 INFO:teuthology.orchestra.run.smithi027.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.30922.log.gz 2024-09-16T08:00:10.103 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.30827.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28268.log 2024-09-16T08:00:10.103 INFO:teuthology.orchestra.run.smithi027.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.30827.log.gz 2024-09-16T08:00:10.103 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.31006.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30877.log 2024-09-16T08:00:10.103 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.28268.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28189.log 2024-09-16T08:00:10.104 INFO:teuthology.orchestra.run.smithi027.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.31006.log.gz 2024-09-16T08:00:10.104 INFO:teuthology.orchestra.run.smithi027.stderr: 84.5% -- replaced with /var/log/ceph/ceph-client.admin.28268.log.gz 2024-09-16T08:00:10.104 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.30877.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30742.log 2024-09-16T08:00:10.104 INFO:teuthology.orchestra.run.smithi027.stderr: 85.9%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28878.log 2024-09-16T08:00:10.104 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.28189.log: -- replaced with /var/log/ceph/ceph-client.admin.30877.log.gz 2024-09-16T08:00:10.104 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.30742.log: 84.3% -- replaced with /var/log/ceph/ceph-client.admin.28189.log.gz 2024-09-16T08:00:10.104 INFO:teuthology.orchestra.run.smithi027.stderr: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.30742.log.gz 2024-09-16T08:00:10.104 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.28878.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.28878.log.gz 2024-09-16T08:00:10.132 INFO:teuthology.orchestra.run.smithi187.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27938.log 2024-09-16T08:00:10.132 INFO:teuthology.orchestra.run.smithi187.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-09-16T08:00:10.133 INFO:teuthology.orchestra.run.smithi187.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27978.log 2024-09-16T08:00:10.133 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/ceph-client.admin.27938.log: 85.1% -- replaced with /var/log/ceph/ceph-client.admin.27938.log.gz 2024-09-16T08:00:10.133 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-09-16T08:00:10.133 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/ceph-client.admin.27978.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27889.log 2024-09-16T08:00:10.133 INFO:teuthology.orchestra.run.smithi187.stderr: 84.9% -- replaced with /var/log/ceph/ceph-client.admin.27978.log.gz 2024-09-16T08:00:10.133 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27847.log 2024-09-16T08:00:10.133 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/ceph-client.admin.27889.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-09-16T08:00:10.133 INFO:teuthology.orchestra.run.smithi187.stderr: 85.4% -- replaced with /var/log/ceph/ceph-client.admin.27889.log.gz 2024-09-16T08:00:10.134 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/ceph-client.admin.27847.log: 85.1% -- replaced with /var/log/ceph/ceph-client.admin.27847.log.gz 2024-09-16T08:00:10.162 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/ceph-osd.3.log: 92.4% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-09-16T08:00:10.355 INFO:teuthology.orchestra.run.smithi027.stderr: 91.3% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-09-16T08:00:11.041 INFO:teuthology.orchestra.run.smithi187.stderr: 94.3% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-09-16T08:00:11.068 INFO:teuthology.orchestra.run.smithi187.stderr: 94.3% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-09-16T08:00:11.070 INFO:teuthology.orchestra.run.smithi187.stderr: 2024-09-16T08:00:11.070 INFO:teuthology.orchestra.run.smithi187.stderr:real 0m0.951s 2024-09-16T08:00:11.071 INFO:teuthology.orchestra.run.smithi187.stderr:user 0m1.807s 2024-09-16T08:00:11.071 INFO:teuthology.orchestra.run.smithi187.stderr:sys 0m0.094s 2024-09-16T08:00:11.105 INFO:teuthology.orchestra.run.smithi027.stderr: 94.4% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-09-16T08:00:11.273 INFO:teuthology.orchestra.run.smithi027.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-09-16T08:00:11.275 INFO:teuthology.orchestra.run.smithi027.stderr: 2024-09-16T08:00:11.275 INFO:teuthology.orchestra.run.smithi027.stderr:real 0m1.211s 2024-09-16T08:00:11.276 INFO:teuthology.orchestra.run.smithi027.stderr:user 0m2.457s 2024-09-16T08:00:11.276 INFO:teuthology.orchestra.run.smithi027.stderr:sys 0m0.128s 2024-09-16T08:00:11.276 INFO:tasks.ceph:Archiving logs... 2024-09-16T08:00:11.276 DEBUG:teuthology.misc:Transferring archived files from smithi027:/var/log/ceph to /home/teuthworker/archive/matan-2024-09-16_07:15:55-crimson-rados-wip-matanb-crimson-testing-59543-distro-crimson-smithi/7906886/remote/smithi027/log 2024-09-16T08:00:11.277 DEBUG:teuthology.orchestra.run.smithi027:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-16T08:00:11.695 DEBUG:teuthology.misc:Transferring archived files from smithi187:/var/log/ceph to /home/teuthworker/archive/matan-2024-09-16_07:15:55-crimson-rados-wip-matanb-crimson-testing-59543-distro-crimson-smithi/7906886/remote/smithi187/log 2024-09-16T08:00:11.695 DEBUG:teuthology.orchestra.run.smithi187:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-16T08:00:11.993 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-16T08:00:12.001 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-16T08:00:12.002 DEBUG:teuthology.orchestra.run.smithi027:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-16T08:00:12.004 DEBUG:teuthology.orchestra.run.smithi187:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-16T08:00:12.091 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-16T08:00:12.100 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-16T08:00:12.100 DEBUG:teuthology.orchestra.run.smithi027:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-16T08:00:12.103 DEBUG:teuthology.orchestra.run.smithi187:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-16T08:00:12.115 INFO:teuthology.orchestra.run.smithi027.stderr:bash: line 1: ntpq: command not found 2024-09-16T08:00:12.119 INFO:teuthology.orchestra.run.smithi187.stderr:bash: line 1: ntpq: command not found 2024-09-16T08:00:12.120 INFO:teuthology.orchestra.run.smithi027.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-16T08:00:12.120 INFO:teuthology.orchestra.run.smithi027.stdout:=============================================================================== 2024-09-16T08:00:12.120 INFO:teuthology.orchestra.run.smithi027.stdout:^* hv01.front.sepia.ceph.com 3 6 377 42 +1893us[+1905us] +/- 45ms 2024-09-16T08:00:12.120 INFO:teuthology.orchestra.run.smithi027.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 41 -4416us[-4416us] +/- 63ms 2024-09-16T08:00:12.120 INFO:teuthology.orchestra.run.smithi027.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 42 +1902us[+1902us] +/- 65ms 2024-09-16T08:00:12.121 INFO:teuthology.orchestra.run.smithi027.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T08:00:12.124 INFO:teuthology.orchestra.run.smithi187.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-16T08:00:12.124 INFO:teuthology.orchestra.run.smithi187.stdout:=============================================================================== 2024-09-16T08:00:12.124 INFO:teuthology.orchestra.run.smithi187.stdout:^* hv01.front.sepia.ceph.com 3 6 377 41 +1914us[+1939us] +/- 45ms 2024-09-16T08:00:12.124 INFO:teuthology.orchestra.run.smithi187.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 42 -4421us[-4396us] +/- 63ms 2024-09-16T08:00:12.124 INFO:teuthology.orchestra.run.smithi187.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 41 +1909us[+1909us] +/- 65ms 2024-09-16T08:00:12.124 INFO:teuthology.orchestra.run.smithi187.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T08:00:12.125 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-16T08:00:12.144 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-16T08:00:12.145 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-16T08:00:12.197 DEBUG:teuthology.orchestra.run.smithi027:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-16T08:00:12.261 DEBUG:teuthology.orchestra.run.smithi187:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-16T08:00:12.325 DEBUG:teuthology.orchestra.run.smithi027:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-16T08:00:12.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T08:00:12.354 DEBUG:teuthology.orchestra.run.smithi187:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-16T08:00:12.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T08:00:12.383 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-16T08:00:12.396 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-16T08:00:12.420 INFO:teuthology.task.internal:Duration was 793.379776 seconds 2024-09-16T08:00:12.420 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-16T08:00:12.471 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-16T08:00:12.471 DEBUG:teuthology.orchestra.run.smithi027:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-16T08:00:12.475 DEBUG:teuthology.orchestra.run.smithi187:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-16T08:00:12.512 INFO:teuthology.orchestra.run.smithi027.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-16T08:00:12.514 INFO:teuthology.orchestra.run.smithi187.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-16T08:00:12.775 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-16T08:00:12.775 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi027.front.sepia.ceph.com 2024-09-16T08:00:12.775 DEBUG:teuthology.orchestra.run.smithi027:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-16T08:00:12.797 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi187.front.sepia.ceph.com 2024-09-16T08:00:12.797 DEBUG:teuthology.orchestra.run.smithi187:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-16T08:00:12.842 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-16T08:00:12.842 DEBUG:teuthology.orchestra.run.smithi027:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-16T08:00:12.845 DEBUG:teuthology.orchestra.run.smithi187:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-16T08:00:12.908 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-16T08:00:12.908 DEBUG:teuthology.orchestra.run.smithi027:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-16T08:00:12.914 DEBUG:teuthology.orchestra.run.smithi187:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-16T08:00:13.046 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-16T08:00:13.061 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-16T08:00:13.062 DEBUG:teuthology.orchestra.run.smithi027:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-16T08:00:13.088 DEBUG:teuthology.orchestra.run.smithi187:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-16T08:00:13.113 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-16T08:00:13.123 DEBUG:teuthology.orchestra.run.smithi027:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-16T08:00:13.131 DEBUG:teuthology.orchestra.run.smithi187:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-16T08:00:13.152 INFO:teuthology.orchestra.run.smithi027.stdout:kernel.core_pattern = core 2024-09-16T08:00:13.176 INFO:teuthology.orchestra.run.smithi187.stdout:kernel.core_pattern = core 2024-09-16T08:00:13.221 DEBUG:teuthology.orchestra.run.smithi027:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-16T08:00:13.242 WARNING:teuthology.task.internal:Found coredumps on ubuntu@smithi027.front.sepia.ceph.com, flagging run as failed 2024-09-16T08:00:13.242 DEBUG:teuthology.orchestra.run.smithi187:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-16T08:00:13.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T08:00:13.276 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-16T08:00:13.286 INFO:teuthology.task.internal:Transferring archived files... 2024-09-16T08:00:13.286 DEBUG:teuthology.misc:Transferring archived files from smithi027:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/matan-2024-09-16_07:15:55-crimson-rados-wip-matanb-crimson-testing-59543-distro-crimson-smithi/7906886/remote/smithi027 2024-09-16T08:00:13.287 DEBUG:teuthology.orchestra.run.smithi027:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-16T08:00:14.472 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 has been restored 2024-09-16T08:00:14.472 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 has been restored 2024-09-16T08:00:14.472 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a has been restored 2024-09-16T08:00:20.637 INFO:teuthology.task.internal:Transferring binaries for coredumps... 2024-09-16T08:00:20.926 INFO:teuthology.task.internal: core looks like: /home/teuthworker/archive/matan-2024-09-16_07:15:55-crimson-rados-wip-matanb-crimson-testing-59543-distro-crimson-smithi/7906886/remote/smithi027/coredump/1726473561.29926.core.gz: gzip compressed data, was "1726473561.29926.core", last modified: Mon Sep 16 08:00:13 2024, max compression, original size modulo 2^32 141279232 2024-09-16T08:00:20.926 INFO:teuthology.task.internal:core is compressed, try accessing gzip file ... 2024-09-16T08:00:21.526 INFO:teuthology.task.internal: core looks like: /tmp/tmpf7xrp74p: ELF 64-bit LSB core file, x86-64, version 1 (SYSV), SVR4-style, from 'ceph_test_rados_api_aio_pp', real uid: 1000, effective uid: 1000, real gid: 1333, effective gid: 1333, execfn: '/usr/bin/ceph_test_rados_api_aio_pp', platform: 'x86_64' 2024-09-16T08:00:21.541 INFO:teuthology.task.internal: dump_program: ceph_test_rados_api_aio_pp 2024-09-16T08:00:21.542 DEBUG:teuthology.orchestra.run.smithi027:> which ceph_test_rados_api_aio_pp 2024-09-16T08:00:21.569 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph_test_rados_api_aio_pp 2024-09-16T08:00:21.632 DEBUG:teuthology.orchestra.remote:smithi027:/usr/bin/ceph_test_rados_api_aio_pp is 1MB 2024-09-16T08:00:21.740 DEBUG:teuthology.orchestra.run.smithi027:> mktemp 2024-09-16T08:00:21.758 INFO:teuthology.orchestra.run.smithi027.stdout:/tmp/tmp.W1TDB52jHC 2024-09-16T08:00:21.758 DEBUG:teuthology.orchestra.run.smithi027:> sudo cp /usr/bin/ceph_test_rados_api_aio_pp.debug /tmp/tmp.W1TDB52jHC 2024-09-16T08:00:21.825 INFO:teuthology.orchestra.run.smithi027.stderr:cp: cannot stat '/usr/bin/ceph_test_rados_api_aio_pp.debug': No such file or directory 2024-09-16T08:00:21.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T08:00:21.834 ERROR:teuthology.run_tasks:Manager failed: internal.archive Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 160, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/task/internal/__init__.py", line 420, in archive fetch_binaries_for_coredumps(path, rem) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/task/internal/__init__.py", line 363, in fetch_binaries_for_coredumps remote.get_file(debug_path, coredump_path) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/remote.py", line 622, in get_file self.run(args=args) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi027 with status 1: 'sudo cp /usr/bin/ceph_test_rados_api_aio_pp.debug /tmp/tmp.W1TDB52jHC' 2024-09-16T08:00:21.844 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-16T08:00:21.858 INFO:teuthology.task.internal:Not uploading archives. 2024-09-16T08:00:21.858 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-16T08:00:21.905 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-16T08:00:21.905 DEBUG:teuthology.orchestra.run.smithi027:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-16T08:00:21.911 DEBUG:teuthology.orchestra.run.smithi187:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-16T08:00:21.927 INFO:teuthology.orchestra.run.smithi027.stdout: 265324 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Sep 16 08:00 /home/ubuntu/cephtest 2024-09-16T08:00:21.927 INFO:teuthology.orchestra.run.smithi027.stdout: 403614 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Sep 16 07:54 /home/ubuntu/cephtest/mnt.0 2024-09-16T08:00:21.927 INFO:teuthology.orchestra.run.smithi027.stdout: 403615 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Sep 16 07:55 /home/ubuntu/cephtest/mnt.0/client.0 2024-09-16T08:00:21.927 INFO:teuthology.orchestra.run.smithi027.stdout: 418244 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 16 07:55 /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-16T08:00:21.928 INFO:teuthology.orchestra.run.smithi027.stdout: 265325 4 drwxr-xr-x 6 ubuntu ubuntu 4096 Sep 16 08:00 /home/ubuntu/cephtest/archive 2024-09-16T08:00:21.928 INFO:teuthology.orchestra.run.smithi027.stdout: 265330 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 16 08:00 /home/ubuntu/cephtest/archive/syslog 2024-09-16T08:00:21.928 INFO:teuthology.orchestra.run.smithi027.stdout: 265372 4 -rw-rw-rw- 1 ubuntu ubuntu 29 Sep 16 07:46 /home/ubuntu/cephtest/archive/syslog/kern.log.gz 2024-09-16T08:00:21.928 INFO:teuthology.orchestra.run.smithi027.stdout: 265332 192 -rw-r--r-- 1 ubuntu ubuntu 194229 Sep 16 08:00 /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-16T08:00:21.928 INFO:teuthology.orchestra.run.smithi027.stdout: 265331 4 -rw-rw-rw- 1 ubuntu ubuntu 29 Sep 16 07:46 /home/ubuntu/cephtest/archive/syslog/misc.log.gz 2024-09-16T08:00:21.928 INFO:teuthology.orchestra.run.smithi027.stdout: 265329 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 16 07:59 /home/ubuntu/cephtest/archive/coredump 2024-09-16T08:00:21.928 INFO:teuthology.orchestra.run.smithi027.stdout: 269683 90988 -rw------- 1 ubuntu ubuntu 141279232 Sep 16 07:59 /home/ubuntu/cephtest/archive/coredump/1726473561.29926.core 2024-09-16T08:00:21.928 INFO:teuthology.orchestra.run.smithi027.stdout: 418245 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 16 07:55 /home/ubuntu/cephtest/archive/unit_test_xml_report 2024-09-16T08:00:21.929 INFO:teuthology.orchestra.run.smithi027.stdout: 403613 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 16 08:00 /home/ubuntu/cephtest/archive/audit 2024-09-16T08:00:21.929 INFO:teuthology.orchestra.run.smithi027.stdout: 403617 120 -rw------- 1 ubuntu ubuntu 121547 Sep 16 08:00 /home/ubuntu/cephtest/archive/audit/audit.log.gz 2024-09-16T08:00:21.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T08:00:21.930 INFO:teuthology.orchestra.run.smithi027.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2024-09-16T08:00:21.930 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/task/internal/__init__.py", line 48, in base yield File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 160, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/task/internal/__init__.py", line 420, in archive fetch_binaries_for_coredumps(path, rem) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/task/internal/__init__.py", line 363, in fetch_binaries_for_coredumps remote.get_file(debug_path, coredump_path) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/remote.py", line 622, in get_file self.run(args=args) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi027 with status 1: 'sudo cp /usr/bin/ceph_test_rados_api_aio_pp.debug /tmp/tmp.W1TDB52jHC' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 160, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.10/contextlib.py", line 153, in __exit__ self.gen.throw(typ, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/task/internal/__init__.py", line 53, in base run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi027 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2024-09-16T08:00:21.940 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-16T08:00:21.954 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-16T08:00:22.005 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: CommandFailedError: Command failed on smithi027 with status 1: 'sudo cp /usr/bin/ceph_test_rados_api_aio_pp.debug /tmp/tmp.W1TDB52jHC' 2024-09-16T08:00:22.009 INFO:teuthology.run:Summary data: description: crimson-rados/basic/{clusters/fixed-2 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/seastore tasks/rados_api_tests} duration: 793.3797760009766 failure_reason: 'Command failed (workunit test rados/test.sh) on smithi027 with status 134: ''mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=0325fd3c11a225c0fd9eb38fab88aad1370ad779 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 CRIMSON_COMPAT=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh --crimson''' flavor: crimson owner: scheduled_matan@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=cd4f55a4386a422d8fb332ae19e0f572 status: fail success: false 2024-09-16T08:00:22.009 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-16T08:00:22.016 INFO:teuthology.orchestra.run.smithi187.stdout: 262169 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Sep 16 08:00 /home/ubuntu/cephtest 2024-09-16T08:00:22.016 INFO:teuthology.orchestra.run.smithi187.stdout: 262170 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Sep 16 08:00 /home/ubuntu/cephtest/archive 2024-09-16T08:00:22.017 INFO:teuthology.orchestra.run.smithi187.stdout: 262172 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 16 08:00 /home/ubuntu/cephtest/archive/syslog 2024-09-16T08:00:22.017 INFO:teuthology.orchestra.run.smithi187.stdout: 265369 4 -rw-rw-rw- 1 ubuntu ubuntu 29 Sep 16 07:46 /home/ubuntu/cephtest/archive/syslog/kern.log.gz 2024-09-16T08:00:22.017 INFO:teuthology.orchestra.run.smithi187.stdout: 265325 188 -rw-r--r-- 1 ubuntu ubuntu 191673 Sep 16 08:00 /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-16T08:00:22.017 INFO:teuthology.orchestra.run.smithi187.stdout: 265324 4 -rw-rw-rw- 1 ubuntu ubuntu 29 Sep 16 07:46 /home/ubuntu/cephtest/archive/syslog/misc.log.gz 2024-09-16T08:00:22.017 INFO:teuthology.orchestra.run.smithi187.stdout: 405791 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 16 08:00 /home/ubuntu/cephtest/archive/audit 2024-09-16T08:00:22.017 INFO:teuthology.orchestra.run.smithi187.stdout: 406099 116 -rw------- 1 ubuntu ubuntu 115219 Sep 16 08:00 /home/ubuntu/cephtest/archive/audit/audit.log.gz 2024-09-16T08:00:22.018 INFO:teuthology.orchestra.run.smithi187.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2024-09-16T08:00:22.096 INFO:teuthology.run:FAIL