2023-01-24T14:17:34.540 INFO:root:teuthology version: 0.0.1.dev22+g8e1bba1 2023-01-24T14:17:34.541 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --owner scheduled_rzarzynski@teuthology --archive /home/teuthworker/archive/rzarzynski-2023-01-24_14:08:39-crimson-rados-main-distro-default-smithi/7135310 --name rzarzynski-2023-01-24_14:08:39-crimson-rados-main-distro-default-smithi --description crimson-rados/basic/{centos_latest clusters/fixed-2 crimson_qa_overrides deploy/ceph tasks/rados_api_tests} -- /home/teuthworker/archive/rzarzynski-2023-01-24_14:08:39-crimson-rados-main-distro-default-smithi/7135310/orig.config.yaml 2023-01-24T14:17:34.571 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2023-01-24T14:17:34.628 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/rzarzynski-2023-01-24_14:08:39-crimson-rados-main-distro-default-smithi/7135310 branch: main description: crimson-rados/basic/{centos_latest clusters/fixed-2 crimson_qa_overrides deploy/ceph tasks/rados_api_tests} email: null first_in_suite: false job_id: '7135310' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: rzarzynski-2023-01-24_14:08:39-crimson-rados-main-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 8.stream overrides: admin_socket: branch: main ceph: conf: client: debug ms: 1 global: ms client mode: crc ms cluster mode: crc ms mon client mode: crc ms mon cluster mode: crc ms mon service mode: crc ms service mode: crc mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon warn on pool no app: false osd: debug ms: 1 debug osd: 20 osd blocked scrub grace period: 3600 osd class default list: '*' osd class load list: '*' osd shutdown pgref assert: true flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - reached quota - but it is still running - overall HEALTH_ - \(POOL_FULL\) - \(SMALLER_PGP_NUM\) - \(CACHE_POOL_NO_HIT_SET\) - \(CACHE_POOL_NEAR_FULL\) - \(POOL_APP_NOT_ENABLED\) - \(PG_AVAILABILITY\) - \(PG_DEGRADED\) sha1: 1a671c198559f54389cd5afa22085a0c7b29a1ef ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: flavor: crimson sha1: 1a671c198559f54389cd5afa22085a0c7b29a1ef selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: main env: CRIMSON_COMPAT: '1' sha1: 510284b66513490445619d1430aa869868c71a09 owner: scheduled_rzarzynski@teuthology priority: 60 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - osd.0 - osd.1 - client.0 - node-exporter.a - - mgr.x - osd.2 - osd.3 - client.1 - prometheus.a - node-exporter.b sha1: 1a671c198559f54389cd5afa22085a0c7b29a1ef sleep_before_teardown: 0 suite: crimson-rados suite_branch: main suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_510284b66513490445619d1430aa869868c71a09/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 510284b66513490445619d1430aa869868c71a09 targets: smithi027.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCyyKjOLmVFo1yODYdugRvmA4PwF5LOyll0nMHXIdtr5y8mJCfNBjSVho1aO7Lj0wsEcUYvj6sJubW/NQF8tRXEyewW7ueIkPWYZhpn+F4XU9/I/3spaTpil9gzzUo09gyWbpoPrRcp+KfD2Yd41gbkXVahA/ckqjKFzWt9WoI2O9yf9xG5A3Iqrk+X2NINWRjZvl/OBN2eG2U7NqtHIUuAcrrNMHviybyW3iCx6JIG+tyhBEsfz9ynrs81Z3VXpqzxYbMwZzUy6jR0rK7s8DOmlJrtneeuV1vP7ZQL529Mot1eEAIC6tu7+DDADy9VKa42zxGV+BLj90yzKCxjoEG3N+g7k1XPynu8+NpThviPHHu/xkJtIM8ARF1FihI+zqmmEthyGxBEu4hhP6UI7gcdSfCXj7Z/Xe+vHEJDRGvfRD5Qu6UBtXovn/OsEdASb63WB6MmtIm9H9iVq7dKgMqIlhsSkyPJcBCof8o+YIZOVhGM9/vLOM0OghreMPERH0s= smithi035.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC0U35SOhGJEcCl3gsD7NS9turfhFdIoVqcbI+HP7UJ9k9QXoGr8FaC86jZTiqo53O8iPTfHawLYATSYEb2fHnIcqPAjGOesshjL5uJW2eQ3UN9LcDMpAD2VLbqHY64AvfgTGnnOP7uvBt9Aq1pmGneE6f0uzaee3PWKKr7DqtZpMbEPzSAzwsy/4+IQmdDRMpopzHRbFhI1sYUe0Z/tPazDvreeINHFtBx+T8M764LG0YKHWvLdkoE6k12BDv0lPlZbd4lwKrDmiejQZIZL+3RczovsnH2eidLdDj0aYGVdFRYfUBY5kNaU08TlKe/TJ1zD2P2/KemmjxHErqMJ6wOHqZ+iZylsczlruglSqZKtNeVECl+S0wj5Fk6EtKGkzEfWs1jRiKS9U1Zota+jdLLT7STBXHl42v1FMB1xsc4FvIr5sAk7wOHq7nauvXeZTpoLpn11VqDPKUmjWGt7JklTkMa8zjhIkHMdu7D3qtBj+SDhg/eX2n5wSvVnlVhOe8= tasks: - install: null - ceph: conf: mon: mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 paxos service trim min: 10 osd: debug monc: 20 flavor: crimson - workunit: clients: client.0: - rados/test.sh - rados/test_pool_quota.sh teuthology_branch: main teuthology_sha1: 8e1bba12e584b7ae912e92e94afe9c075ab51fa3 timestamp: 2023-01-24_14:08:39 tube: smithi user: rzarzynski verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.17735 2023-01-24T14:17:34.628 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_510284b66513490445619d1430aa869868c71a09/qa; will attempt to use it 2023-01-24T14:17:34.629 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_510284b66513490445619d1430aa869868c71a09/qa/tasks 2023-01-24T14:17:34.629 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-01-24T14:17:34.631 INFO:teuthology.task.internal:Checking packages... 2023-01-24T14:17:34.659 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'crimson' and ceph hash '1a671c198559f54389cd5afa22085a0c7b29a1ef' 2023-01-24T14:17:34.659 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-01-24T14:17:34.659 INFO:teuthology.packaging:ref: None 2023-01-24T14:17:34.660 INFO:teuthology.packaging:tag: None 2023-01-24T14:17:34.660 INFO:teuthology.packaging:branch: main 2023-01-24T14:17:34.660 INFO:teuthology.packaging:sha1: 1a671c198559f54389cd5afa22085a0c7b29a1ef 2023-01-24T14:17:34.661 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F8%2Fx86_64&ref=main 2023-01-24T14:17:34.870 INFO:teuthology.task.internal:Found packages for ceph version 18.0.0-1954.g1a671c19 2023-01-24T14:17:34.871 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-01-24T14:17:34.878 INFO:teuthology.task.internal:no buildpackages task found 2023-01-24T14:17:34.878 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-01-24T14:17:34.885 INFO:teuthology.task.internal:Saving configuration 2023-01-24T14:17:34.897 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-01-24T14:17:34.903 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-01-24T14:17:34.927 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi027.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rzarzynski-2023-01-24_14:08:39-crimson-rados-main-distro-default-smithi/7135310', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-01-24 14:11:24.877107', 'locked_by': 'scheduled_rzarzynski@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBM08cQ29l0KKToemdTWrwZSrNxY4MImwnEbwMOAm1FLzNukZdy2LdIpu6aMkgOxs3ySrbGq/IWpuqlDzIpfUNZ0='} 2023-01-24T14:17:34.949 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi035.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rzarzynski-2023-01-24_14:08:39-crimson-rados-main-distro-default-smithi/7135310', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-01-24 14:11:24.875697', 'locked_by': 'scheduled_rzarzynski@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNRI5FndwbQDXAqWvDs27pCIkL5TKRDzA15ef8qfnC2mYwYK/VaBpetN8OFmM8Dk1aZMfwoqQ9U97Hh7LwPcSV8='} 2023-01-24T14:17:34.949 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-01-24T14:17:34.956 INFO:teuthology.task.internal:roles: ubuntu@smithi027.front.sepia.ceph.com - ['mon.a', 'osd.0', 'osd.1', 'client.0', 'node-exporter.a'] 2023-01-24T14:17:34.956 INFO:teuthology.task.internal:roles: ubuntu@smithi035.front.sepia.ceph.com - ['mgr.x', 'osd.2', 'osd.3', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-24T14:17:34.956 INFO:teuthology.run_tasks:Running task console_log... 2023-01-24T14:17:35.025 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fd92dd735e0>, signals=[15]) 2023-01-24T14:17:35.026 INFO:teuthology.run_tasks:Running task internal.connect... 2023-01-24T14:17:35.033 INFO:teuthology.task.internal:Opening connections... 2023-01-24T14:17:35.033 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi027.front.sepia.ceph.com 2023-01-24T14:17:35.035 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-01-24T14:17:35.105 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi035.front.sepia.ceph.com 2023-01-24T14:17:35.106 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi035.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-01-24T14:17:35.177 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-01-24T14:17:35.185 DEBUG:teuthology.orchestra.run.smithi027:> uname -m 2023-01-24T14:17:35.207 INFO:teuthology.orchestra.run.smithi027.stdout:x86_64 2023-01-24T14:17:35.208 DEBUG:teuthology.orchestra.run.smithi027:> cat /etc/os-release 2023-01-24T14:17:35.267 INFO:teuthology.orchestra.run.smithi027.stdout:NAME="CentOS Stream" 2023-01-24T14:17:35.267 INFO:teuthology.orchestra.run.smithi027.stdout:VERSION="8" 2023-01-24T14:17:35.268 INFO:teuthology.orchestra.run.smithi027.stdout:ID="centos" 2023-01-24T14:17:35.268 INFO:teuthology.orchestra.run.smithi027.stdout:ID_LIKE="rhel fedora" 2023-01-24T14:17:35.268 INFO:teuthology.orchestra.run.smithi027.stdout:VERSION_ID="8" 2023-01-24T14:17:35.268 INFO:teuthology.orchestra.run.smithi027.stdout:PLATFORM_ID="platform:el8" 2023-01-24T14:17:35.268 INFO:teuthology.orchestra.run.smithi027.stdout:PRETTY_NAME="CentOS Stream 8" 2023-01-24T14:17:35.269 INFO:teuthology.orchestra.run.smithi027.stdout:ANSI_COLOR="0;31" 2023-01-24T14:17:35.269 INFO:teuthology.orchestra.run.smithi027.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2023-01-24T14:17:35.269 INFO:teuthology.orchestra.run.smithi027.stdout:HOME_URL="https://centos.org/" 2023-01-24T14:17:35.269 INFO:teuthology.orchestra.run.smithi027.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-01-24T14:17:35.269 INFO:teuthology.orchestra.run.smithi027.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2023-01-24T14:17:35.270 INFO:teuthology.orchestra.run.smithi027.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-01-24T14:17:35.270 INFO:teuthology.lock.ops:Updating smithi027.front.sepia.ceph.com on lock server 2023-01-24T14:17:35.286 DEBUG:teuthology.orchestra.run.smithi035:> uname -m 2023-01-24T14:17:35.308 INFO:teuthology.orchestra.run.smithi035.stdout:x86_64 2023-01-24T14:17:35.308 DEBUG:teuthology.orchestra.run.smithi035:> cat /etc/os-release 2023-01-24T14:17:35.368 INFO:teuthology.orchestra.run.smithi035.stdout:NAME="CentOS Stream" 2023-01-24T14:17:35.369 INFO:teuthology.orchestra.run.smithi035.stdout:VERSION="8" 2023-01-24T14:17:35.369 INFO:teuthology.orchestra.run.smithi035.stdout:ID="centos" 2023-01-24T14:17:35.369 INFO:teuthology.orchestra.run.smithi035.stdout:ID_LIKE="rhel fedora" 2023-01-24T14:17:35.369 INFO:teuthology.orchestra.run.smithi035.stdout:VERSION_ID="8" 2023-01-24T14:17:35.370 INFO:teuthology.orchestra.run.smithi035.stdout:PLATFORM_ID="platform:el8" 2023-01-24T14:17:35.370 INFO:teuthology.orchestra.run.smithi035.stdout:PRETTY_NAME="CentOS Stream 8" 2023-01-24T14:17:35.370 INFO:teuthology.orchestra.run.smithi035.stdout:ANSI_COLOR="0;31" 2023-01-24T14:17:35.370 INFO:teuthology.orchestra.run.smithi035.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2023-01-24T14:17:35.370 INFO:teuthology.orchestra.run.smithi035.stdout:HOME_URL="https://centos.org/" 2023-01-24T14:17:35.371 INFO:teuthology.orchestra.run.smithi035.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-01-24T14:17:35.371 INFO:teuthology.orchestra.run.smithi035.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2023-01-24T14:17:35.371 INFO:teuthology.orchestra.run.smithi035.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-01-24T14:17:35.371 INFO:teuthology.lock.ops:Updating smithi035.front.sepia.ceph.com on lock server 2023-01-24T14:17:35.386 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-01-24T14:17:35.396 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-01-24T14:17:35.402 INFO:teuthology.task.internal:Checking for old test directory... 2023-01-24T14:17:35.402 DEBUG:teuthology.orchestra.run.smithi027:> test '!' -e /home/ubuntu/cephtest 2023-01-24T14:17:35.405 DEBUG:teuthology.orchestra.run.smithi035:> test '!' -e /home/ubuntu/cephtest 2023-01-24T14:17:35.427 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-01-24T14:17:35.434 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-01-24T14:17:35.434 DEBUG:teuthology.orchestra.run.smithi027:> test -z $(ls -A /var/lib/ceph) 2023-01-24T14:17:35.463 DEBUG:teuthology.orchestra.run.smithi035:> test -z $(ls -A /var/lib/ceph) 2023-01-24T14:17:35.496 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-01-24T14:17:35.545 INFO:teuthology.run_tasks:Running task kernel... 2023-01-24T14:17:35.561 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-01-24T14:17:35.562 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mgr.x': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-01-24T14:17:35.562 DEBUG:teuthology.orchestra.run.smithi027:> test -f /run/.containerenv -o -f /.dockerenv 2023-01-24T14:17:35.563 DEBUG:teuthology.orchestra.run.smithi035:> test -f /run/.containerenv -o -f /.dockerenv 2023-01-24T14:17:35.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-01-24T14:17:35.582 DEBUG:teuthology.orchestra.run.smithi027:> uname -r 2023-01-24T14:17:35.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-01-24T14:17:35.583 DEBUG:teuthology.orchestra.run.smithi035:> uname -r 2023-01-24T14:17:35.641 INFO:teuthology.orchestra.run.smithi027.stdout:4.18.0-408.el8.x86_64 2023-01-24T14:17:35.641 INFO:teuthology.task.kernel:Running kernel on smithi027: 4.18.0-408.el8.x86_64 2023-01-24T14:17:35.642 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum install -y kernel 2023-01-24T14:17:35.643 INFO:teuthology.orchestra.run.smithi035.stdout:4.18.0-408.el8.x86_64 2023-01-24T14:17:35.643 INFO:teuthology.task.kernel:Running kernel on smithi035: 4.18.0-408.el8.x86_64 2023-01-24T14:17:35.643 DEBUG:teuthology.orchestra.run.smithi035:> sudo yum install -y kernel 2023-01-24T14:17:39.235 INFO:teuthology.orchestra.run.smithi035.stdout:CentOS-8 - AppStream 34 MB/s | 27 MB 00:00 2023-01-24T14:17:39.357 INFO:teuthology.orchestra.run.smithi027.stdout:CentOS-8 - AppStream 31 MB/s | 27 MB 00:00 2023-01-24T14:17:45.519 INFO:teuthology.orchestra.run.smithi035.stdout:CentOS-8 - BaseOS 29 MB/s | 27 MB 00:00 2023-01-24T14:17:45.599 INFO:teuthology.orchestra.run.smithi027.stdout:CentOS-8 - BaseOS 31 MB/s | 27 MB 00:00 2023-01-24T14:17:49.968 INFO:teuthology.orchestra.run.smithi035.stdout:CentOS-8 - Extras 175 kB/s | 18 kB 00:00 2023-01-24T14:17:50.232 INFO:teuthology.orchestra.run.smithi027.stdout:CentOS-8 - Extras 162 kB/s | 18 kB 00:00 2023-01-24T14:17:50.370 INFO:teuthology.orchestra.run.smithi035.stdout:CentOS-8 - PowerTools 21 MB/s | 5.6 MB 00:00 2023-01-24T14:17:50.659 INFO:teuthology.orchestra.run.smithi027.stdout:CentOS-8 - PowerTools 20 MB/s | 5.6 MB 00:00 2023-01-24T14:17:51.678 INFO:teuthology.orchestra.run.smithi035.stdout:Copr repo for python3-asyncssh owned by ceph 25 kB/s | 3.5 kB 00:00 2023-01-24T14:17:51.942 INFO:teuthology.orchestra.run.smithi027.stdout:Copr repo for python3-asyncssh owned by ceph 27 kB/s | 3.5 kB 00:00 2023-01-24T14:17:54.145 INFO:teuthology.orchestra.run.smithi035.stdout:Extra Packages for Enterprise Linux 5.7 MB/s | 13 MB 00:02 2023-01-24T14:17:55.817 INFO:teuthology.orchestra.run.smithi027.stdout:Extra Packages for Enterprise Linux 3.6 MB/s | 13 MB 00:03 2023-01-24T14:17:57.711 INFO:teuthology.orchestra.run.smithi035.stdout:lab-extras 464 kB/s | 24 kB 00:00 2023-01-24T14:17:59.267 INFO:teuthology.orchestra.run.smithi027.stdout:lab-extras 400 kB/s | 24 kB 00:00 2023-01-24T14:18:01.001 INFO:teuthology.orchestra.run.smithi035.stdout:Package kernel-4.18.0-408.el8.x86_64 is already installed. 2023-01-24T14:18:01.002 INFO:teuthology.orchestra.run.smithi035.stdout:Package kernel-4.18.0-305.3.1.el8.x86_64 is already installed. 2023-01-24T14:18:01.162 INFO:teuthology.orchestra.run.smithi035.stdout:Dependencies resolved. 2023-01-24T14:18:01.163 INFO:teuthology.orchestra.run.smithi035.stdout:================================================================================ 2023-01-24T14:18:01.163 INFO:teuthology.orchestra.run.smithi035.stdout: Package Arch Version Repository Size 2023-01-24T14:18:01.163 INFO:teuthology.orchestra.run.smithi035.stdout:================================================================================ 2023-01-24T14:18:01.164 INFO:teuthology.orchestra.run.smithi035.stdout:Installing: 2023-01-24T14:18:01.164 INFO:teuthology.orchestra.run.smithi035.stdout: kernel x86_64 4.18.0-448.el8 CentOS-BaseOS 9.2 M 2023-01-24T14:18:01.164 INFO:teuthology.orchestra.run.smithi035.stdout: kernel-core x86_64 4.18.0-448.el8 CentOS-BaseOS 41 M 2023-01-24T14:18:01.164 INFO:teuthology.orchestra.run.smithi035.stdout: kernel-modules x86_64 4.18.0-448.el8 CentOS-BaseOS 34 M 2023-01-24T14:18:01.164 INFO:teuthology.orchestra.run.smithi035.stdout: 2023-01-24T14:18:01.165 INFO:teuthology.orchestra.run.smithi035.stdout:Transaction Summary 2023-01-24T14:18:01.165 INFO:teuthology.orchestra.run.smithi035.stdout:================================================================================ 2023-01-24T14:18:01.165 INFO:teuthology.orchestra.run.smithi035.stdout:Install 3 Packages 2023-01-24T14:18:01.165 INFO:teuthology.orchestra.run.smithi035.stdout: 2023-01-24T14:18:01.166 INFO:teuthology.orchestra.run.smithi035.stdout:Total download size: 85 M 2023-01-24T14:18:01.166 INFO:teuthology.orchestra.run.smithi035.stdout:Installed size: 94 M 2023-01-24T14:18:01.166 INFO:teuthology.orchestra.run.smithi035.stdout:Downloading Packages: 2023-01-24T14:18:01.538 INFO:teuthology.orchestra.run.smithi035.stdout:(1/3): kernel-4.18.0-448.el8.x86_64.rpm 25 MB/s | 9.2 MB 00:00 2023-01-24T14:18:02.222 INFO:teuthology.orchestra.run.smithi035.stdout:(2/3): kernel-modules-4.18.0-448.el8.x86_64.rpm 32 MB/s | 34 MB 00:01 2023-01-24T14:18:02.566 INFO:teuthology.orchestra.run.smithi027.stdout:Package kernel-4.18.0-408.el8.x86_64 is already installed. 2023-01-24T14:18:02.566 INFO:teuthology.orchestra.run.smithi027.stdout:Package kernel-4.18.0-305.3.1.el8.x86_64 is already installed. 2023-01-24T14:18:02.722 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2023-01-24T14:18:02.723 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2023-01-24T14:18:02.723 INFO:teuthology.orchestra.run.smithi027.stdout: Package Arch Version Repository Size 2023-01-24T14:18:02.723 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2023-01-24T14:18:02.724 INFO:teuthology.orchestra.run.smithi027.stdout:Installing: 2023-01-24T14:18:02.724 INFO:teuthology.orchestra.run.smithi027.stdout: kernel x86_64 4.18.0-448.el8 CentOS-BaseOS 9.2 M 2023-01-24T14:18:02.724 INFO:teuthology.orchestra.run.smithi027.stdout: kernel-core x86_64 4.18.0-448.el8 CentOS-BaseOS 41 M 2023-01-24T14:18:02.724 INFO:teuthology.orchestra.run.smithi027.stdout: kernel-modules x86_64 4.18.0-448.el8 CentOS-BaseOS 34 M 2023-01-24T14:18:02.724 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-24T14:18:02.725 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2023-01-24T14:18:02.725 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2023-01-24T14:18:02.725 INFO:teuthology.orchestra.run.smithi027.stdout:Install 3 Packages 2023-01-24T14:18:02.725 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-24T14:18:02.726 INFO:teuthology.orchestra.run.smithi027.stdout:Total download size: 85 M 2023-01-24T14:18:02.726 INFO:teuthology.orchestra.run.smithi027.stdout:Installed size: 94 M 2023-01-24T14:18:02.726 INFO:teuthology.orchestra.run.smithi027.stdout:Downloading Packages: 2023-01-24T14:18:02.797 INFO:teuthology.orchestra.run.smithi035.stdout:(3/3): kernel-core-4.18.0-448.el8.x86_64.rpm 25 MB/s | 41 MB 00:01 2023-01-24T14:18:02.800 INFO:teuthology.orchestra.run.smithi035.stdout:-------------------------------------------------------------------------------- 2023-01-24T14:18:02.800 INFO:teuthology.orchestra.run.smithi035.stdout:Total 52 MB/s | 85 MB 00:01 2023-01-24T14:18:02.985 INFO:teuthology.orchestra.run.smithi027.stdout:(1/3): kernel-4.18.0-448.el8.x86_64.rpm 35 MB/s | 9.2 MB 00:00 2023-01-24T14:18:03.460 INFO:teuthology.orchestra.run.smithi035.stdout:Running transaction check 2023-01-24T14:18:03.844 INFO:teuthology.orchestra.run.smithi027.stdout:(2/3): kernel-modules-4.18.0-448.el8.x86_64.rpm 30 MB/s | 34 MB 00:01 2023-01-24T14:18:03.876 INFO:teuthology.orchestra.run.smithi035.stdout:Transaction check succeeded. 2023-01-24T14:18:03.876 INFO:teuthology.orchestra.run.smithi035.stdout:Running transaction test 2023-01-24T14:18:04.377 INFO:teuthology.orchestra.run.smithi027.stdout:(3/3): kernel-core-4.18.0-448.el8.x86_64.rpm 25 MB/s | 41 MB 00:01 2023-01-24T14:18:04.380 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2023-01-24T14:18:04.381 INFO:teuthology.orchestra.run.smithi027.stdout:Total 51 MB/s | 85 MB 00:01 2023-01-24T14:18:05.046 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction check 2023-01-24T14:18:05.459 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction check succeeded. 2023-01-24T14:18:05.460 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction test 2023-01-24T14:18:05.865 INFO:teuthology.orchestra.run.smithi035.stdout:Transaction test succeeded. 2023-01-24T14:18:05.903 INFO:teuthology.orchestra.run.smithi035.stdout:Running transaction 2023-01-24T14:18:07.696 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction test succeeded. 2023-01-24T14:18:07.725 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction 2023-01-24T14:18:08.771 INFO:teuthology.orchestra.run.smithi035.stdout: Preparing : 1/1 2023-01-24T14:18:09.535 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : kernel-core-4.18.0-448.el8.x86_64 1/3 2023-01-24T14:18:10.617 INFO:teuthology.orchestra.run.smithi027.stdout: Preparing : 1/1 2023-01-24T14:18:11.258 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : kernel-core-4.18.0-448.el8.x86_64 1/3 2023-01-24T14:18:13.511 INFO:teuthology.orchestra.run.smithi035.stdout: Running scriptlet: kernel-core-4.18.0-448.el8.x86_64 1/3 2023-01-24T14:18:13.926 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : kernel-modules-4.18.0-448.el8.x86_64 2/3 2023-01-24T14:18:15.243 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: kernel-core-4.18.0-448.el8.x86_64 1/3 2023-01-24T14:18:15.700 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : kernel-modules-4.18.0-448.el8.x86_64 2/3 2023-01-24T14:18:17.486 INFO:teuthology.orchestra.run.smithi035.stdout: Running scriptlet: kernel-modules-4.18.0-448.el8.x86_64 2/3 2023-01-24T14:18:17.729 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : kernel-4.18.0-448.el8.x86_64 3/3 2023-01-24T14:18:19.250 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: kernel-modules-4.18.0-448.el8.x86_64 2/3 2023-01-24T14:18:19.536 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : kernel-4.18.0-448.el8.x86_64 3/3 2023-01-24T14:18:41.359 INFO:teuthology.orchestra.run.smithi035.stdout: Running scriptlet: kernel-core-4.18.0-448.el8.x86_64 3/3 2023-01-24T14:18:41.386 INFO:teuthology.orchestra.run.smithi035.stdout: Running scriptlet: kernel-modules-4.18.0-448.el8.x86_64 3/3 2023-01-24T14:18:41.810 INFO:teuthology.orchestra.run.smithi035.stdout: Running scriptlet: kernel-4.18.0-448.el8.x86_64 3/3 2023-01-24T14:18:41.810 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : kernel-4.18.0-448.el8.x86_64 1/3 2023-01-24T14:18:41.810 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : kernel-core-4.18.0-448.el8.x86_64 2/3 2023-01-24T14:18:42.515 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : kernel-modules-4.18.0-448.el8.x86_64 3/3 2023-01-24T14:18:42.516 INFO:teuthology.orchestra.run.smithi035.stdout: 2023-01-24T14:18:42.516 INFO:teuthology.orchestra.run.smithi035.stdout:Installed: 2023-01-24T14:18:42.516 INFO:teuthology.orchestra.run.smithi035.stdout: kernel-4.18.0-448.el8.x86_64 kernel-core-4.18.0-448.el8.x86_64 2023-01-24T14:18:42.516 INFO:teuthology.orchestra.run.smithi035.stdout: kernel-modules-4.18.0-448.el8.x86_64 2023-01-24T14:18:42.516 INFO:teuthology.orchestra.run.smithi035.stdout: 2023-01-24T14:18:42.517 INFO:teuthology.orchestra.run.smithi035.stdout:Complete! 2023-01-24T14:18:42.626 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: kernel-core-4.18.0-448.el8.x86_64 3/3 2023-01-24T14:18:42.646 DEBUG:teuthology.orchestra.run.smithi035:> rpm -q kernel --last | head -n 1 2023-01-24T14:18:42.651 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: kernel-modules-4.18.0-448.el8.x86_64 3/3 2023-01-24T14:18:42.890 INFO:teuthology.orchestra.run.smithi035.stdout:kernel-4.18.0-448.el8.x86_64 Tue 24 Jan 2023 02:18:17 PM UTC 2023-01-24T14:18:42.891 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-448.el8.x86_64 2023-01-24T14:18:42.892 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-408.el8.x86_64 Expected: 4.18.0-448.el8.x86_64 2023-01-24T14:18:42.892 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-01-24T14:18:42.892 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-01-24T14:18:42.892 INFO:teuthology.task.kernel:Installing distro kernel on mgr.x... 2023-01-24T14:18:42.893 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi035.front.sepia.ceph.com, path=None, version=distro) 2023-01-24T14:18:42.893 DEBUG:teuthology.orchestra.run.smithi035:> rpm -q kernel --last | head -n 1 2023-01-24T14:18:43.079 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: kernel-4.18.0-448.el8.x86_64 3/3 2023-01-24T14:18:43.079 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : kernel-4.18.0-448.el8.x86_64 1/3 2023-01-24T14:18:43.079 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : kernel-core-4.18.0-448.el8.x86_64 2/3 2023-01-24T14:18:43.089 INFO:teuthology.orchestra.run.smithi035.stdout:kernel-4.18.0-448.el8.x86_64 Tue 24 Jan 2023 02:18:17 PM UTC 2023-01-24T14:18:43.091 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-448.el8.x86_64 2023-01-24T14:18:43.092 DEBUG:teuthology.orchestra.run.smithi035:> sudo rpm -qi grub2-tools 2023-01-24T14:18:43.126 INFO:teuthology.orchestra.run.smithi035.stdout:Name : grub2-tools 2023-01-24T14:18:43.127 INFO:teuthology.orchestra.run.smithi035.stdout:Epoch : 1 2023-01-24T14:18:43.127 INFO:teuthology.orchestra.run.smithi035.stdout:Version : 2.02 2023-01-24T14:18:43.127 INFO:teuthology.orchestra.run.smithi035.stdout:Release : 129.el8 2023-01-24T14:18:43.127 INFO:teuthology.orchestra.run.smithi035.stdout:Architecture: x86_64 2023-01-24T14:18:43.127 INFO:teuthology.orchestra.run.smithi035.stdout:Install Date: Sun 28 Aug 2022 01:03:50 PM UTC 2023-01-24T14:18:43.128 INFO:teuthology.orchestra.run.smithi035.stdout:Group : System Environment/Base 2023-01-24T14:18:43.128 INFO:teuthology.orchestra.run.smithi035.stdout:Size : 9265586 2023-01-24T14:18:43.128 INFO:teuthology.orchestra.run.smithi035.stdout:License : GPLv3+ 2023-01-24T14:18:43.128 INFO:teuthology.orchestra.run.smithi035.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2023-01-24T14:18:43.128 INFO:teuthology.orchestra.run.smithi035.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2023-01-24T14:18:43.129 INFO:teuthology.orchestra.run.smithi035.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2023-01-24T14:18:43.129 INFO:teuthology.orchestra.run.smithi035.stdout:Build Host : kbuilder.bsys.centos.org 2023-01-24T14:18:43.129 INFO:teuthology.orchestra.run.smithi035.stdout:Relocations : (not relocatable) 2023-01-24T14:18:43.129 INFO:teuthology.orchestra.run.smithi035.stdout:Packager : CentOS BuildSystem 2023-01-24T14:18:43.129 INFO:teuthology.orchestra.run.smithi035.stdout:Vendor : CentOS 2023-01-24T14:18:43.130 INFO:teuthology.orchestra.run.smithi035.stdout:URL : http://www.gnu.org/software/grub/ 2023-01-24T14:18:43.130 INFO:teuthology.orchestra.run.smithi035.stdout:Summary : Support tools for GRUB. 2023-01-24T14:18:43.130 INFO:teuthology.orchestra.run.smithi035.stdout:Description : 2023-01-24T14:18:43.130 INFO:teuthology.orchestra.run.smithi035.stdout: 2023-01-24T14:18:43.130 INFO:teuthology.orchestra.run.smithi035.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-01-24T14:18:43.131 INFO:teuthology.orchestra.run.smithi035.stdout:customizable bootloader with modular architecture. It supports a rich 2023-01-24T14:18:43.131 INFO:teuthology.orchestra.run.smithi035.stdout:variety of kernel formats, file systems, computer architectures and 2023-01-24T14:18:43.131 INFO:teuthology.orchestra.run.smithi035.stdout:hardware devices. 2023-01-24T14:18:43.131 INFO:teuthology.orchestra.run.smithi035.stdout: 2023-01-24T14:18:43.131 INFO:teuthology.orchestra.run.smithi035.stdout:This subpackage provides tools for support of all platforms. 2023-01-24T14:18:43.132 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-01-24T14:18:43.133 INFO:teuthology.task.kernel:Updating grub on smithi035 to boot 4.18.0-448.el8.x86_64 2023-01-24T14:18:43.133 DEBUG:teuthology.orchestra.run.smithi035:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-01-24T14:18:44.032 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : kernel-modules-4.18.0-448.el8.x86_64 3/3 2023-01-24T14:18:44.032 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-24T14:18:44.033 INFO:teuthology.orchestra.run.smithi027.stdout:Installed: 2023-01-24T14:18:44.033 INFO:teuthology.orchestra.run.smithi027.stdout: kernel-4.18.0-448.el8.x86_64 kernel-core-4.18.0-448.el8.x86_64 2023-01-24T14:18:44.033 INFO:teuthology.orchestra.run.smithi027.stdout: kernel-modules-4.18.0-448.el8.x86_64 2023-01-24T14:18:44.033 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-24T14:18:44.034 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2023-01-24T14:18:44.338 DEBUG:teuthology.orchestra.run.smithi027:> rpm -q kernel --last | head -n 1 2023-01-24T14:18:44.478 INFO:teuthology.orchestra.run.smithi035.stderr:Generating grub configuration file ... 2023-01-24T14:18:44.543 INFO:teuthology.orchestra.run.smithi027.stdout:kernel-4.18.0-448.el8.x86_64 Tue 24 Jan 2023 02:18:19 PM UTC 2023-01-24T14:18:44.545 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-448.el8.x86_64 2023-01-24T14:18:44.545 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-408.el8.x86_64 Expected: 4.18.0-448.el8.x86_64 2023-01-24T14:18:44.546 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-01-24T14:18:44.546 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-01-24T14:18:44.546 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2023-01-24T14:18:44.546 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi027.front.sepia.ceph.com, path=None, version=distro) 2023-01-24T14:18:44.547 DEBUG:teuthology.orchestra.run.smithi027:> rpm -q kernel --last | head -n 1 2023-01-24T14:18:44.742 INFO:teuthology.orchestra.run.smithi027.stdout:kernel-4.18.0-448.el8.x86_64 Tue 24 Jan 2023 02:18:19 PM UTC 2023-01-24T14:18:44.743 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-448.el8.x86_64 2023-01-24T14:18:44.744 DEBUG:teuthology.orchestra.run.smithi027:> sudo rpm -qi grub2-tools 2023-01-24T14:18:44.777 INFO:teuthology.orchestra.run.smithi027.stdout:Name : grub2-tools 2023-01-24T14:18:44.778 INFO:teuthology.orchestra.run.smithi027.stdout:Epoch : 1 2023-01-24T14:18:44.778 INFO:teuthology.orchestra.run.smithi027.stdout:Version : 2.02 2023-01-24T14:18:44.778 INFO:teuthology.orchestra.run.smithi027.stdout:Release : 129.el8 2023-01-24T14:18:44.778 INFO:teuthology.orchestra.run.smithi027.stdout:Architecture: x86_64 2023-01-24T14:18:44.779 INFO:teuthology.orchestra.run.smithi027.stdout:Install Date: Sun 28 Aug 2022 01:03:50 PM UTC 2023-01-24T14:18:44.779 INFO:teuthology.orchestra.run.smithi027.stdout:Group : System Environment/Base 2023-01-24T14:18:44.779 INFO:teuthology.orchestra.run.smithi027.stdout:Size : 9265586 2023-01-24T14:18:44.779 INFO:teuthology.orchestra.run.smithi027.stdout:License : GPLv3+ 2023-01-24T14:18:44.779 INFO:teuthology.orchestra.run.smithi027.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2023-01-24T14:18:44.779 INFO:teuthology.orchestra.run.smithi027.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2023-01-24T14:18:44.780 INFO:teuthology.orchestra.run.smithi027.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2023-01-24T14:18:44.780 INFO:teuthology.orchestra.run.smithi027.stdout:Build Host : kbuilder.bsys.centos.org 2023-01-24T14:18:44.780 INFO:teuthology.orchestra.run.smithi027.stdout:Relocations : (not relocatable) 2023-01-24T14:18:44.780 INFO:teuthology.orchestra.run.smithi027.stdout:Packager : CentOS BuildSystem 2023-01-24T14:18:44.781 INFO:teuthology.orchestra.run.smithi027.stdout:Vendor : CentOS 2023-01-24T14:18:44.781 INFO:teuthology.orchestra.run.smithi027.stdout:URL : http://www.gnu.org/software/grub/ 2023-01-24T14:18:44.781 INFO:teuthology.orchestra.run.smithi027.stdout:Summary : Support tools for GRUB. 2023-01-24T14:18:44.781 INFO:teuthology.orchestra.run.smithi027.stdout:Description : 2023-01-24T14:18:44.781 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-24T14:18:44.782 INFO:teuthology.orchestra.run.smithi027.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-01-24T14:18:44.782 INFO:teuthology.orchestra.run.smithi027.stdout:customizable bootloader with modular architecture. It supports a rich 2023-01-24T14:18:44.782 INFO:teuthology.orchestra.run.smithi027.stdout:variety of kernel formats, file systems, computer architectures and 2023-01-24T14:18:44.782 INFO:teuthology.orchestra.run.smithi027.stdout:hardware devices. 2023-01-24T14:18:44.782 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-24T14:18:44.783 INFO:teuthology.orchestra.run.smithi027.stdout:This subpackage provides tools for support of all platforms. 2023-01-24T14:18:44.784 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-01-24T14:18:44.784 INFO:teuthology.task.kernel:Updating grub on smithi027 to boot 4.18.0-448.el8.x86_64 2023-01-24T14:18:44.784 DEBUG:teuthology.orchestra.run.smithi027:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-01-24T14:18:45.861 INFO:teuthology.orchestra.run.smithi027.stderr:Generating grub configuration file ... 2023-01-24T14:18:46.948 INFO:teuthology.orchestra.run.smithi035.stderr:done 2023-01-24T14:18:46.950 DEBUG:teuthology.orchestra.run.smithi035:> mktemp 2023-01-24T14:18:47.012 INFO:teuthology.orchestra.run.smithi035.stdout:/tmp/tmp.DLpje5LKdQ 2023-01-24T14:18:47.012 DEBUG:teuthology.orchestra.run.smithi035:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.DLpje5LKdQ 2023-01-24T14:18:47.056 DEBUG:teuthology.orchestra.run.smithi035:> sudo chmod 0666 /tmp/tmp.DLpje5LKdQ 2023-01-24T14:18:47.212 DEBUG:teuthology.orchestra.remote:smithi035:/tmp/tmp.DLpje5LKdQ is 6KB 2023-01-24T14:18:47.261 DEBUG:teuthology.orchestra.run.smithi035:> rm -fr /tmp/tmp.DLpje5LKdQ 2023-01-24T14:18:47.278 DEBUG:teuthology.orchestra.run.smithi035:> sudo /bin/ls /boot/loader/entries || true 2023-01-24T14:18:47.350 INFO:teuthology.orchestra.run.smithi035.stdout:2ff282740cf84d4fa02ff60ed95fbbff-0-rescue.conf 2023-01-24T14:18:47.350 INFO:teuthology.orchestra.run.smithi035.stdout:2ff282740cf84d4fa02ff60ed95fbbff-4.18.0-305.3.1.el8.x86_64.conf 2023-01-24T14:18:47.350 INFO:teuthology.orchestra.run.smithi035.stdout:2ff282740cf84d4fa02ff60ed95fbbff-4.18.0-408.el8.x86_64.conf 2023-01-24T14:18:47.350 INFO:teuthology.orchestra.run.smithi035.stdout:2ff282740cf84d4fa02ff60ed95fbbff-4.18.0-448.el8.x86_64.conf 2023-01-24T14:18:47.351 DEBUG:teuthology.orchestra.run.smithi035:> sudo grub2-set-default 2ff282740cf84d4fa02ff60ed95fbbff-4.18.0-448.el8.x86_64 2023-01-24T14:18:47.525 DEBUG:teuthology.orchestra.run.smithi035:> sudo shutdown -r now 2023-01-24T14:18:48.213 INFO:teuthology.orchestra.run.smithi027.stderr:done 2023-01-24T14:18:48.215 DEBUG:teuthology.orchestra.run.smithi027:> mktemp 2023-01-24T14:18:48.250 INFO:teuthology.orchestra.run.smithi027.stdout:/tmp/tmp.50BnVj9uFu 2023-01-24T14:18:48.250 DEBUG:teuthology.orchestra.run.smithi027:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.50BnVj9uFu 2023-01-24T14:18:48.335 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod 0666 /tmp/tmp.50BnVj9uFu 2023-01-24T14:18:48.438 DEBUG:teuthology.orchestra.remote:smithi027:/tmp/tmp.50BnVj9uFu is 6KB 2023-01-24T14:18:48.487 DEBUG:teuthology.orchestra.run.smithi027:> rm -fr /tmp/tmp.50BnVj9uFu 2023-01-24T14:18:48.504 DEBUG:teuthology.orchestra.run.smithi027:> sudo /bin/ls /boot/loader/entries || true 2023-01-24T14:18:48.592 INFO:teuthology.orchestra.run.smithi027.stdout:2ff282740cf84d4fa02ff60ed95fbbff-0-rescue.conf 2023-01-24T14:18:48.592 INFO:teuthology.orchestra.run.smithi027.stdout:2ff282740cf84d4fa02ff60ed95fbbff-4.18.0-305.3.1.el8.x86_64.conf 2023-01-24T14:18:48.593 INFO:teuthology.orchestra.run.smithi027.stdout:2ff282740cf84d4fa02ff60ed95fbbff-4.18.0-408.el8.x86_64.conf 2023-01-24T14:18:48.593 INFO:teuthology.orchestra.run.smithi027.stdout:2ff282740cf84d4fa02ff60ed95fbbff-4.18.0-448.el8.x86_64.conf 2023-01-24T14:18:48.594 DEBUG:teuthology.orchestra.run.smithi027:> sudo grub2-set-default 2ff282740cf84d4fa02ff60ed95fbbff-4.18.0-448.el8.x86_64 2023-01-24T14:18:48.709 DEBUG:teuthology.orchestra.run.smithi027:> sudo shutdown -r now 2023-01-24T14:19:17.546 INFO:teuthology.task.kernel:Checking client mgr.x for new kernel version... 2023-01-24T14:19:17.547 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi035.front.sepia.ceph.com' 2023-01-24T14:19:17.547 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi035.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-01-24T14:19:18.713 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2023-01-24T14:19:18.714 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2023-01-24T14:19:18.715 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-01-24T14:19:37.192 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.27 2023-01-24T14:20:07.214 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2023-01-24T14:20:07.215 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-01-24T14:20:10.284 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.27 2023-01-24T14:20:17.556 DEBUG:teuthology.orchestra.remote:timed out 2023-01-24T14:20:40.308 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2023-01-24T14:20:40.309 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-01-24T14:20:47.564 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi035.front.sepia.ceph.com' 2023-01-24T14:20:47.565 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi035.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-01-24T14:20:47.830 DEBUG:teuthology.orchestra.run.smithi035:> true 2023-01-24T14:20:48.128 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi035.front.sepia.ceph.com' 2023-01-24T14:20:48.128 INFO:teuthology.task.kernel:Checking kernel version of mgr.x, want "4.18.0-448.el8.x86_64"... 2023-01-24T14:20:48.128 DEBUG:teuthology.orchestra.run.smithi035:> uname -r 2023-01-24T14:20:48.150 INFO:teuthology.orchestra.run.smithi035.stdout:4.18.0-448.el8.x86_64 2023-01-24T14:20:48.151 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-448.el8.x86_64 vs 4.18.0-448.el8.x86_64 2023-01-24T14:20:48.151 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-01-24T14:20:48.152 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-01-24T14:20:49.153 INFO:teuthology.task.kernel:Enabling kdb on mgr.x... 2023-01-24T14:20:49.154 DEBUG:teuthology.orchestra.run.smithi035:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-01-24T14:20:49.254 INFO:teuthology.orchestra.run.smithi035.stdout:ttyS1 2023-01-24T14:20:49.273 DEBUG:teuthology.parallel:result is None 2023-01-24T14:20:59.169 DEBUG:teuthology.orchestra.run.smithi027:> true 2023-01-24T14:21:02.240 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi027.front.sepia.ceph.com' 2023-01-24T14:21:02.241 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "4.18.0-448.el8.x86_64"... 2023-01-24T14:21:02.241 DEBUG:teuthology.orchestra.run.smithi027:> uname -r 2023-01-24T14:21:02.260 INFO:teuthology.orchestra.run.smithi027.stdout:4.18.0-448.el8.x86_64 2023-01-24T14:21:02.260 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-448.el8.x86_64 vs 4.18.0-448.el8.x86_64 2023-01-24T14:21:02.261 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-01-24T14:21:02.261 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-01-24T14:21:03.262 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2023-01-24T14:21:03.262 DEBUG:teuthology.orchestra.run.smithi027:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-01-24T14:21:03.374 INFO:teuthology.orchestra.run.smithi027.stdout:ttyS1 2023-01-24T14:21:03.405 DEBUG:teuthology.parallel:result is None 2023-01-24T14:21:03.405 INFO:teuthology.run_tasks:Running task internal.base... 2023-01-24T14:21:03.413 INFO:teuthology.task.internal:Creating test directory... 2023-01-24T14:21:03.414 DEBUG:teuthology.orchestra.run.smithi027:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-01-24T14:21:03.416 DEBUG:teuthology.orchestra.run.smithi035:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-01-24T14:21:03.438 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-01-24T14:21:03.444 INFO:teuthology.run_tasks:Running task internal.archive... 2023-01-24T14:21:03.454 INFO:teuthology.task.internal:Creating archive directory... 2023-01-24T14:21:03.454 DEBUG:teuthology.orchestra.run.smithi027:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-01-24T14:21:03.477 DEBUG:teuthology.orchestra.run.smithi035:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-01-24T14:21:03.510 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-01-24T14:21:03.517 INFO:teuthology.task.internal:Enabling coredump saving... 2023-01-24T14:21:03.517 DEBUG:teuthology.orchestra.run.smithi027:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-01-24T14:21:03.542 DEBUG:teuthology.orchestra.run.smithi035:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-01-24T14:21:03.590 INFO:teuthology.orchestra.run.smithi035.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-01-24T14:21:03.591 INFO:teuthology.orchestra.run.smithi027.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-01-24T14:21:03.601 INFO:teuthology.orchestra.run.smithi035.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-01-24T14:21:03.603 INFO:teuthology.orchestra.run.smithi027.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-01-24T14:21:03.604 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-01-24T14:21:03.611 INFO:teuthology.task.internal:Configuring sudo... 2023-01-24T14:21:03.611 DEBUG:teuthology.orchestra.run.smithi027:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-01-24T14:21:03.614 DEBUG:teuthology.orchestra.run.smithi035:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-01-24T14:21:03.645 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-01-24T14:21:03.653 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-01-24T14:21:03.654 DEBUG:teuthology.orchestra.run.smithi027:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-01-24T14:21:03.683 DEBUG:teuthology.orchestra.run.smithi035:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-01-24T14:21:03.704 DEBUG:teuthology.orchestra.run.smithi027:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-01-24T14:21:03.790 DEBUG:teuthology.orchestra.run.smithi027:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-01-24T14:21:03.833 DEBUG:teuthology.orchestra.run.smithi027:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-01-24T14:21:03.927 DEBUG:teuthology.orchestra.run.smithi027:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-01-24T14:21:03.963 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2023-01-24T14:21:03.964 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-01-24T14:21:04.107 DEBUG:teuthology.orchestra.run.smithi035:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-01-24T14:21:04.158 DEBUG:teuthology.orchestra.run.smithi035:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-01-24T14:21:04.212 DEBUG:teuthology.orchestra.run.smithi035:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-01-24T14:21:04.310 DEBUG:teuthology.orchestra.run.smithi035:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-01-24T14:21:04.345 DEBUG:teuthology.orchestra.run.smithi035:> set -ex 2023-01-24T14:21:04.346 DEBUG:teuthology.orchestra.run.smithi035:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-01-24T14:21:04.414 DEBUG:teuthology.orchestra.run.smithi027:> sudo service rsyslog restart 2023-01-24T14:21:04.417 DEBUG:teuthology.orchestra.run.smithi035:> sudo service rsyslog restart 2023-01-24T14:21:04.470 INFO:teuthology.orchestra.run.smithi027.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-01-24T14:21:04.516 INFO:teuthology.orchestra.run.smithi035.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-01-24T14:21:04.862 INFO:teuthology.run_tasks:Running task internal.timer... 2023-01-24T14:21:04.871 INFO:teuthology.task.internal:Starting timer... 2023-01-24T14:21:04.871 INFO:teuthology.run_tasks:Running task pcp... 2023-01-24T14:21:04.880 INFO:teuthology.run_tasks:Running task selinux... 2023-01-24T14:21:04.889 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2023-01-24T14:21:04.889 DEBUG:teuthology.orchestra.run.smithi027:> sudo service auditd rotate 2023-01-24T14:21:04.986 INFO:teuthology.orchestra.run.smithi027.stdout:Rotating logs: 2023-01-24T14:21:04.988 DEBUG:teuthology.orchestra.run.smithi035:> sudo service auditd rotate 2023-01-24T14:21:05.040 INFO:teuthology.orchestra.run.smithi035.stdout:Rotating logs: 2023-01-24T14:21:05.042 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-01-24T14:21:05.042 DEBUG:teuthology.orchestra.run.smithi027:> /usr/sbin/getenforce 2023-01-24T14:21:05.071 INFO:teuthology.orchestra.run.smithi027.stdout:Permissive 2023-01-24T14:21:05.072 DEBUG:teuthology.orchestra.run.smithi035:> /usr/sbin/getenforce 2023-01-24T14:21:05.099 INFO:teuthology.orchestra.run.smithi035.stdout:Permissive 2023-01-24T14:21:05.100 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi027.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi035.front.sepia.ceph.com': 'permissive'} 2023-01-24T14:21:05.100 DEBUG:teuthology.orchestra.run.smithi027:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|comm="ksmtuned"\|comm="sssd"\|comm="sss_cache"\|context=system_u:system_r:NetworkManager_dispatcher_t:s0\)' 2023-01-24T14:21:05.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-01-24T14:21:05.141 DEBUG:teuthology.orchestra.run.smithi035:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|comm="ksmtuned"\|comm="sssd"\|comm="sss_cache"\|context=system_u:system_r:NetworkManager_dispatcher_t:s0\)' 2023-01-24T14:21:05.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-01-24T14:21:05.170 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-01-24T14:21:05.171 DEBUG:teuthology.orchestra.run.smithi027:> sudo /usr/sbin/setenforce permissive 2023-01-24T14:21:05.209 DEBUG:teuthology.orchestra.run.smithi035:> sudo /usr/sbin/setenforce permissive 2023-01-24T14:21:05.246 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-01-24T14:21:05.256 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main was just updated or references a specific commit; assuming it is current 2023-01-24T14:21:05.256 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-01-24T14:21:05.272 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-01-24T14:21:05.274 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi027.front.sepia.ceph.com,smithi035.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-01-24T14:27:47.238 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi027.front.sepia.ceph.com'), Remote(name='ubuntu@smithi035.front.sepia.ceph.com')] 2023-01-24T14:27:47.239 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2023-01-24T14:27:47.240 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-01-24T14:27:47.309 DEBUG:teuthology.orchestra.run.smithi027:> true 2023-01-24T14:27:47.336 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi035.front.sepia.ceph.com' 2023-01-24T14:27:47.337 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi035.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-01-24T14:27:47.408 DEBUG:teuthology.orchestra.run.smithi035:> true 2023-01-24T14:27:47.437 INFO:teuthology.run_tasks:Running task clock... 2023-01-24T14:27:47.455 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-01-24T14:27:47.456 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-01-24T14:27:47.456 DEBUG:teuthology.orchestra.run.smithi027:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-01-24T14:27:47.459 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-01-24T14:27:47.459 DEBUG:teuthology.orchestra.run.smithi035:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-01-24T14:27:47.489 INFO:teuthology.orchestra.run.smithi027.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-01-24T14:27:47.503 INFO:teuthology.orchestra.run.smithi027.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-01-24T14:27:47.515 INFO:teuthology.orchestra.run.smithi035.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-01-24T14:27:47.531 INFO:teuthology.orchestra.run.smithi035.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-01-24T14:27:47.570 INFO:teuthology.orchestra.run.smithi027.stderr:sudo: ntpd: command not found 2023-01-24T14:27:47.582 INFO:teuthology.orchestra.run.smithi027.stdout:506 Cannot talk to daemon 2023-01-24T14:27:47.598 INFO:teuthology.orchestra.run.smithi027.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-01-24T14:27:47.601 INFO:teuthology.orchestra.run.smithi035.stderr:sudo: ntpd: command not found 2023-01-24T14:27:47.611 INFO:teuthology.orchestra.run.smithi027.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-01-24T14:27:47.614 INFO:teuthology.orchestra.run.smithi035.stdout:506 Cannot talk to daemon 2023-01-24T14:27:47.630 INFO:teuthology.orchestra.run.smithi035.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-01-24T14:27:47.644 INFO:teuthology.orchestra.run.smithi035.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-01-24T14:27:47.674 INFO:teuthology.orchestra.run.smithi027.stderr:bash: ntpq: command not found 2023-01-24T14:27:47.677 INFO:teuthology.orchestra.run.smithi027.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-01-24T14:27:47.677 INFO:teuthology.orchestra.run.smithi027.stdout:=============================================================================== 2023-01-24T14:27:47.677 INFO:teuthology.orchestra.run.smithi027.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-01-24T14:27:47.678 INFO:teuthology.orchestra.run.smithi027.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-01-24T14:27:47.678 INFO:teuthology.orchestra.run.smithi027.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-01-24T14:27:47.678 INFO:teuthology.orchestra.run.smithi027.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-01-24T14:27:47.709 INFO:teuthology.orchestra.run.smithi035.stderr:bash: ntpq: command not found 2023-01-24T14:27:47.713 INFO:teuthology.orchestra.run.smithi035.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-01-24T14:27:47.713 INFO:teuthology.orchestra.run.smithi035.stdout:=============================================================================== 2023-01-24T14:27:47.713 INFO:teuthology.orchestra.run.smithi035.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-01-24T14:27:47.713 INFO:teuthology.orchestra.run.smithi035.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-01-24T14:27:47.714 INFO:teuthology.orchestra.run.smithi035.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-01-24T14:27:47.714 INFO:teuthology.orchestra.run.smithi035.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-01-24T14:27:47.715 INFO:teuthology.run_tasks:Running task install... 2023-01-24T14:27:47.723 DEBUG:teuthology.task.install:project ceph 2023-01-24T14:27:47.723 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'crimson', 'sha1': '1a671c198559f54389cd5afa22085a0c7b29a1ef'}} 2023-01-24T14:27:47.724 DEBUG:teuthology.task.install:config {'flavor': 'crimson', 'sha1': '1a671c198559f54389cd5afa22085a0c7b29a1ef'} 2023-01-24T14:27:47.724 INFO:teuthology.task.install:Using flavor: crimson 2023-01-24T14:27:47.730 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-01-24T14:27:47.731 INFO:teuthology.task.install:extra packages: [] 2023-01-24T14:27:47.731 DEBUG:teuthology.orchestra.run.smithi027:> sudo lsb_release -is 2023-01-24T14:27:47.770 INFO:teuthology.orchestra.run.smithi027.stdout:CentOSStream 2023-01-24T14:27:47.771 DEBUG:teuthology.misc:System to be installed: CentOSStream 2023-01-24T14:27:47.772 DEBUG:teuthology.orchestra.run.smithi035:> sudo lsb_release -is 2023-01-24T14:27:47.772 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'flavor': 'crimson', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '1a671c198559f54389cd5afa22085a0c7b29a1ef', 'tag': None, 'wait_for_package': False} 2023-01-24T14:27:47.773 DEBUG:teuthology.orchestra.run.smithi027:> sudo lsb_release -is 2023-01-24T14:27:47.811 INFO:teuthology.orchestra.run.smithi035.stdout:CentOSStream 2023-01-24T14:27:47.812 DEBUG:teuthology.misc:System to be installed: CentOSStream 2023-01-24T14:27:47.813 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'flavor': 'crimson', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '1a671c198559f54389cd5afa22085a0c7b29a1ef', 'tag': None, 'wait_for_package': False} 2023-01-24T14:27:47.813 DEBUG:teuthology.orchestra.run.smithi035:> sudo lsb_release -is 2023-01-24T14:27:47.850 INFO:teuthology.orchestra.run.smithi027.stdout:CentOSStream 2023-01-24T14:27:47.851 DEBUG:teuthology.misc:System to be installed: CentOSStream 2023-01-24T14:27:47.852 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F8%2Fx86_64&sha1=1a671c198559f54389cd5afa22085a0c7b29a1ef 2023-01-24T14:27:47.892 INFO:teuthology.orchestra.run.smithi035.stdout:CentOSStream 2023-01-24T14:27:47.893 DEBUG:teuthology.misc:System to be installed: CentOSStream 2023-01-24T14:27:47.893 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F8%2Fx86_64&sha1=1a671c198559f54389cd5afa22085a0c7b29a1ef 2023-01-24T14:27:48.038 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/main/1a671c198559f54389cd5afa22085a0c7b29a1ef/centos/8/flavors/crimson/ 2023-01-24T14:27:48.038 INFO:teuthology.task.install.rpm:Package version is 18.0.0-1954.g1a671c19 2023-01-24T14:27:48.097 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/main/1a671c198559f54389cd5afa22085a0c7b29a1ef/centos/8/flavors/crimson/ 2023-01-24T14:27:48.098 INFO:teuthology.task.install.rpm:Package version is 18.0.0-1954.g1a671c19 2023-01-24T14:27:48.311 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/main/1a671c198559f54389cd5afa22085a0c7b29a1ef/centos/8/flavors/crimson/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/main/1a671c198559f54389cd5afa22085a0c7b29a1ef/centos/8/flavors/crimson/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/main/1a671c198559f54389cd5afa22085a0c7b29a1ef/centos/8/flavors/crimson/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-01-24T14:27:48.312 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2023-01-24T14:27:48.312 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-01-24T14:27:48.343 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2023-01-24T14:27:48.344 DEBUG:teuthology.orchestra.run.smithi027:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/1a671c198559f54389cd5afa22085a0c7b29a1ef/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-01-24T14:27:48.370 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/main/1a671c198559f54389cd5afa22085a0c7b29a1ef/centos/8/flavors/crimson/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/main/1a671c198559f54389cd5afa22085a0c7b29a1ef/centos/8/flavors/crimson/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/main/1a671c198559f54389cd5afa22085a0c7b29a1ef/centos/8/flavors/crimson/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-01-24T14:27:48.371 DEBUG:teuthology.orchestra.run.smithi035:> set -ex 2023-01-24T14:27:48.371 DEBUG:teuthology.orchestra.run.smithi035:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-01-24T14:27:48.401 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2023-01-24T14:27:48.401 DEBUG:teuthology.orchestra.run.smithi035:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/1a671c198559f54389cd5afa22085a0c7b29a1ef/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-01-24T14:27:48.414 DEBUG:teuthology.orchestra.run.smithi027:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-01-24T14:27:48.473 DEBUG:teuthology.orchestra.run.smithi035:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-01-24T14:27:48.495 DEBUG:teuthology.orchestra.run.smithi027:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-01-24T14:27:48.552 DEBUG:teuthology.orchestra.run.smithi035:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-01-24T14:27:48.564 INFO:teuthology.orchestra.run.smithi027.stdout:check_obsoletes = 1 2023-01-24T14:27:48.565 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum clean all 2023-01-24T14:27:48.621 INFO:teuthology.orchestra.run.smithi035.stdout:check_obsoletes = 1 2023-01-24T14:27:48.622 DEBUG:teuthology.orchestra.run.smithi035:> sudo yum clean all 2023-01-24T14:27:48.825 INFO:teuthology.orchestra.run.smithi027.stdout:61 files removed 2023-01-24T14:27:48.855 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install ceph-radosgw 2023-01-24T14:27:48.882 INFO:teuthology.orchestra.run.smithi035.stdout:61 files removed 2023-01-24T14:27:48.911 DEBUG:teuthology.orchestra.run.smithi035:> sudo yum -y install ceph-radosgw 2023-01-24T14:27:50.006 INFO:teuthology.orchestra.run.smithi027.stdout:ceph packages for x86_64 115 kB/s | 83 kB 00:00 2023-01-24T14:27:50.164 INFO:teuthology.orchestra.run.smithi035.stdout:ceph packages for x86_64 101 kB/s | 83 kB 00:00 2023-01-24T14:27:50.507 INFO:teuthology.orchestra.run.smithi027.stdout:ceph noarch packages 20 kB/s | 9.3 kB 00:00 2023-01-24T14:27:50.681 INFO:teuthology.orchestra.run.smithi035.stdout:ceph noarch packages 20 kB/s | 9.3 kB 00:00 2023-01-24T14:27:51.091 INFO:teuthology.orchestra.run.smithi027.stdout:ceph source packages 2.9 kB/s | 1.6 kB 00:00 2023-01-24T14:27:51.257 INFO:teuthology.orchestra.run.smithi035.stdout:ceph source packages 2.9 kB/s | 1.6 kB 00:00 2023-01-24T14:27:51.885 INFO:teuthology.orchestra.run.smithi027.stdout:CentOS-8 - AppStream 36 MB/s | 27 MB 00:00 2023-01-24T14:27:52.034 INFO:teuthology.orchestra.run.smithi035.stdout:CentOS-8 - AppStream 37 MB/s | 27 MB 00:00 2023-01-24T14:27:57.844 INFO:teuthology.orchestra.run.smithi027.stdout:CentOS-8 - BaseOS 43 MB/s | 27 MB 00:00 2023-01-24T14:27:58.076 INFO:teuthology.orchestra.run.smithi035.stdout:CentOS-8 - BaseOS 39 MB/s | 27 MB 00:00 2023-01-24T14:28:02.309 INFO:teuthology.orchestra.run.smithi027.stdout:CentOS-8 - Extras 156 kB/s | 18 kB 00:00 2023-01-24T14:28:02.617 INFO:teuthology.orchestra.run.smithi035.stdout:CentOS-8 - Extras 161 kB/s | 18 kB 00:00 2023-01-24T14:28:02.711 INFO:teuthology.orchestra.run.smithi027.stdout:CentOS-8 - PowerTools 22 MB/s | 5.6 MB 00:00 2023-01-24T14:28:03.027 INFO:teuthology.orchestra.run.smithi035.stdout:CentOS-8 - PowerTools 21 MB/s | 5.6 MB 00:00 2023-01-24T14:28:04.019 INFO:teuthology.orchestra.run.smithi027.stdout:Copr repo for python3-asyncssh owned by ceph 23 kB/s | 3.5 kB 00:00 2023-01-24T14:28:04.377 INFO:teuthology.orchestra.run.smithi035.stdout:Copr repo for python3-asyncssh owned by ceph 28 kB/s | 3.5 kB 00:00 2023-01-24T14:28:05.436 INFO:teuthology.orchestra.run.smithi027.stdout:Extra Packages for Enterprise Linux 10 MB/s | 13 MB 00:01 2023-01-24T14:28:06.094 INFO:teuthology.orchestra.run.smithi035.stdout:Extra Packages for Enterprise Linux 8.5 MB/s | 13 MB 00:01 2023-01-24T14:28:08.945 INFO:teuthology.orchestra.run.smithi027.stdout:lab-extras 635 kB/s | 24 kB 00:00 2023-01-24T14:28:09.753 INFO:teuthology.orchestra.run.smithi035.stdout:lab-extras 414 kB/s | 24 kB 00:00 2023-01-24T14:28:12.616 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2023-01-24T14:28:12.619 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2023-01-24T14:28:12.619 INFO:teuthology.orchestra.run.smithi027.stdout: Package Arch Version Repository Size 2023-01-24T14:28:12.620 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2023-01-24T14:28:12.620 INFO:teuthology.orchestra.run.smithi027.stdout:Installing: 2023-01-24T14:28:12.620 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-radosgw x86_64 2:18.0.0-1954.g1a671c19.el8 ceph 14 M 2023-01-24T14:28:12.621 INFO:teuthology.orchestra.run.smithi027.stdout:Upgrading: 2023-01-24T14:28:12.621 INFO:teuthology.orchestra.run.smithi027.stdout: librados2 x86_64 2:18.0.0-1954.g1a671c19.el8 ceph 3.3 M 2023-01-24T14:28:12.621 INFO:teuthology.orchestra.run.smithi027.stdout: librbd1 x86_64 2:18.0.0-1954.g1a671c19.el8 ceph 3.5 M 2023-01-24T14:28:12.622 INFO:teuthology.orchestra.run.smithi027.stdout:Installing dependencies: 2023-01-24T14:28:12.622 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-base x86_64 2:18.0.0-1954.g1a671c19.el8 ceph 6.3 M 2023-01-24T14:28:12.622 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-common x86_64 2:18.0.0-1954.g1a671c19.el8 ceph 27 M 2023-01-24T14:28:12.622 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-selinux x86_64 2:18.0.0-1954.g1a671c19.el8 ceph 24 k 2023-01-24T14:28:12.622 INFO:teuthology.orchestra.run.smithi027.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-01-24T14:28:12.623 INFO:teuthology.orchestra.run.smithi027.stdout: libcephfs2 x86_64 2:18.0.0-1954.g1a671c19.el8 ceph 741 k 2023-01-24T14:28:12.623 INFO:teuthology.orchestra.run.smithi027.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-01-24T14:28:12.623 INFO:teuthology.orchestra.run.smithi027.stdout: librabbitmq x86_64 0.9.0-3.el8 CentOS-BaseOS 47 k 2023-01-24T14:28:12.623 INFO:teuthology.orchestra.run.smithi027.stdout: libradosstriper1 x86_64 2:18.0.0-1954.g1a671c19.el8 ceph 461 k 2023-01-24T14:28:12.624 INFO:teuthology.orchestra.run.smithi027.stdout: librdkafka x86_64 0.11.4-3.el8 CentOS-AppStream 354 k 2023-01-24T14:28:12.624 INFO:teuthology.orchestra.run.smithi027.stdout: librgw2 x86_64 2:18.0.0-1954.g1a671c19.el8 ceph 7.2 M 2023-01-24T14:28:12.624 INFO:teuthology.orchestra.run.smithi027.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-01-24T14:28:12.624 INFO:teuthology.orchestra.run.smithi027.stdout: lttng-ust x86_64 2.8.1-11.el8 CentOS-AppStream 259 k 2023-01-24T14:28:12.624 INFO:teuthology.orchestra.run.smithi027.stdout: python3-ceph-argparse 2023-01-24T14:28:12.625 INFO:teuthology.orchestra.run.smithi027.stdout: x86_64 2:18.0.0-1954.g1a671c19.el8 ceph 45 k 2023-01-24T14:28:12.625 INFO:teuthology.orchestra.run.smithi027.stdout: python3-ceph-common x86_64 2:18.0.0-1954.g1a671c19.el8 ceph 115 k 2023-01-24T14:28:12.625 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cephfs x86_64 2:18.0.0-1954.g1a671c19.el8 ceph 190 k 2023-01-24T14:28:12.625 INFO:teuthology.orchestra.run.smithi027.stdout: python3-prettytable noarch 0.7.2-14.el8 CentOS-AppStream 44 k 2023-01-24T14:28:12.625 INFO:teuthology.orchestra.run.smithi027.stdout: python3-rados x86_64 2:18.0.0-1954.g1a671c19.el8 ceph 371 k 2023-01-24T14:28:12.626 INFO:teuthology.orchestra.run.smithi027.stdout: python3-rbd x86_64 2:18.0.0-1954.g1a671c19.el8 ceph 364 k 2023-01-24T14:28:12.626 INFO:teuthology.orchestra.run.smithi027.stdout: python3-rgw x86_64 2:18.0.0-1954.g1a671c19.el8 ceph 106 k 2023-01-24T14:28:12.626 INFO:teuthology.orchestra.run.smithi027.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-01-24T14:28:12.626 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-24T14:28:12.626 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2023-01-24T14:28:12.627 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2023-01-24T14:28:12.627 INFO:teuthology.orchestra.run.smithi027.stdout:Install 21 Packages 2023-01-24T14:28:12.627 INFO:teuthology.orchestra.run.smithi027.stdout:Upgrade 2 Packages 2023-01-24T14:28:12.627 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-24T14:28:12.628 INFO:teuthology.orchestra.run.smithi027.stdout:Total download size: 67 M 2023-01-24T14:28:12.628 INFO:teuthology.orchestra.run.smithi027.stdout:Downloading Packages: 2023-01-24T14:28:13.416 INFO:teuthology.orchestra.run.smithi035.stdout:Dependencies resolved. 2023-01-24T14:28:13.419 INFO:teuthology.orchestra.run.smithi035.stdout:================================================================================ 2023-01-24T14:28:13.419 INFO:teuthology.orchestra.run.smithi035.stdout: Package Arch Version Repository Size 2023-01-24T14:28:13.419 INFO:teuthology.orchestra.run.smithi035.stdout:================================================================================ 2023-01-24T14:28:13.420 INFO:teuthology.orchestra.run.smithi035.stdout:Installing: 2023-01-24T14:28:13.420 INFO:teuthology.orchestra.run.smithi035.stdout: ceph-radosgw x86_64 2:18.0.0-1954.g1a671c19.el8 ceph 14 M 2023-01-24T14:28:13.420 INFO:teuthology.orchestra.run.smithi035.stdout:Upgrading: 2023-01-24T14:28:13.420 INFO:teuthology.orchestra.run.smithi035.stdout: librados2 x86_64 2:18.0.0-1954.g1a671c19.el8 ceph 3.3 M 2023-01-24T14:28:13.421 INFO:teuthology.orchestra.run.smithi035.stdout: librbd1 x86_64 2:18.0.0-1954.g1a671c19.el8 ceph 3.5 M 2023-01-24T14:28:13.421 INFO:teuthology.orchestra.run.smithi035.stdout:Installing dependencies: 2023-01-24T14:28:13.421 INFO:teuthology.orchestra.run.smithi035.stdout: ceph-base x86_64 2:18.0.0-1954.g1a671c19.el8 ceph 6.3 M 2023-01-24T14:28:13.421 INFO:teuthology.orchestra.run.smithi035.stdout: ceph-common x86_64 2:18.0.0-1954.g1a671c19.el8 ceph 27 M 2023-01-24T14:28:13.421 INFO:teuthology.orchestra.run.smithi035.stdout: ceph-selinux x86_64 2:18.0.0-1954.g1a671c19.el8 ceph 24 k 2023-01-24T14:28:13.422 INFO:teuthology.orchestra.run.smithi035.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-01-24T14:28:13.422 INFO:teuthology.orchestra.run.smithi035.stdout: libcephfs2 x86_64 2:18.0.0-1954.g1a671c19.el8 ceph 741 k 2023-01-24T14:28:13.422 INFO:teuthology.orchestra.run.smithi035.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-01-24T14:28:13.422 INFO:teuthology.orchestra.run.smithi035.stdout: librabbitmq x86_64 0.9.0-3.el8 CentOS-BaseOS 47 k 2023-01-24T14:28:13.422 INFO:teuthology.orchestra.run.smithi035.stdout: libradosstriper1 x86_64 2:18.0.0-1954.g1a671c19.el8 ceph 461 k 2023-01-24T14:28:13.423 INFO:teuthology.orchestra.run.smithi035.stdout: librdkafka x86_64 0.11.4-3.el8 CentOS-AppStream 354 k 2023-01-24T14:28:13.423 INFO:teuthology.orchestra.run.smithi035.stdout: librgw2 x86_64 2:18.0.0-1954.g1a671c19.el8 ceph 7.2 M 2023-01-24T14:28:13.423 INFO:teuthology.orchestra.run.smithi035.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-01-24T14:28:13.423 INFO:teuthology.orchestra.run.smithi035.stdout: lttng-ust x86_64 2.8.1-11.el8 CentOS-AppStream 259 k 2023-01-24T14:28:13.423 INFO:teuthology.orchestra.run.smithi035.stdout: python3-ceph-argparse 2023-01-24T14:28:13.424 INFO:teuthology.orchestra.run.smithi035.stdout: x86_64 2:18.0.0-1954.g1a671c19.el8 ceph 45 k 2023-01-24T14:28:13.424 INFO:teuthology.orchestra.run.smithi035.stdout: python3-ceph-common x86_64 2:18.0.0-1954.g1a671c19.el8 ceph 115 k 2023-01-24T14:28:13.424 INFO:teuthology.orchestra.run.smithi035.stdout: python3-cephfs x86_64 2:18.0.0-1954.g1a671c19.el8 ceph 190 k 2023-01-24T14:28:13.424 INFO:teuthology.orchestra.run.smithi035.stdout: python3-prettytable noarch 0.7.2-14.el8 CentOS-AppStream 44 k 2023-01-24T14:28:13.425 INFO:teuthology.orchestra.run.smithi035.stdout: python3-rados x86_64 2:18.0.0-1954.g1a671c19.el8 ceph 371 k 2023-01-24T14:28:13.425 INFO:teuthology.orchestra.run.smithi035.stdout: python3-rbd x86_64 2:18.0.0-1954.g1a671c19.el8 ceph 364 k 2023-01-24T14:28:13.425 INFO:teuthology.orchestra.run.smithi035.stdout: python3-rgw x86_64 2:18.0.0-1954.g1a671c19.el8 ceph 106 k 2023-01-24T14:28:13.426 INFO:teuthology.orchestra.run.smithi035.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-01-24T14:28:13.426 INFO:teuthology.orchestra.run.smithi035.stdout: 2023-01-24T14:28:13.426 INFO:teuthology.orchestra.run.smithi035.stdout:Transaction Summary 2023-01-24T14:28:13.427 INFO:teuthology.orchestra.run.smithi035.stdout:================================================================================ 2023-01-24T14:28:13.427 INFO:teuthology.orchestra.run.smithi035.stdout:Install 21 Packages 2023-01-24T14:28:13.428 INFO:teuthology.orchestra.run.smithi035.stdout:Upgrade 2 Packages 2023-01-24T14:28:13.428 INFO:teuthology.orchestra.run.smithi035.stdout: 2023-01-24T14:28:13.429 INFO:teuthology.orchestra.run.smithi035.stdout:Total download size: 67 M 2023-01-24T14:28:13.429 INFO:teuthology.orchestra.run.smithi035.stdout:Downloading Packages: 2023-01-24T14:28:14.414 INFO:teuthology.orchestra.run.smithi027.stdout:(1/23): ceph-base-18.0.0-1954.g1a671c19.el8.x86 3.5 MB/s | 6.3 MB 00:01 2023-01-24T14:28:14.513 INFO:teuthology.orchestra.run.smithi027.stdout:(2/23): ceph-selinux-18.0.0-1954.g1a671c19.el8. 243 kB/s | 24 kB 00:00 2023-01-24T14:28:14.796 INFO:teuthology.orchestra.run.smithi027.stdout:(3/23): libcephfs2-18.0.0-1954.g1a671c19.el8.x8 2.6 MB/s | 741 kB 00:00 2023-01-24T14:28:15.039 INFO:teuthology.orchestra.run.smithi027.stdout:(4/23): libradosstriper1-18.0.0-1954.g1a671c19. 1.9 MB/s | 461 kB 00:00 2023-01-24T14:28:16.372 INFO:teuthology.orchestra.run.smithi027.stdout:(5/23): ceph-radosgw-18.0.0-1954.g1a671c19.el8. 3.8 MB/s | 14 MB 00:03 2023-01-24T14:28:16.447 INFO:teuthology.orchestra.run.smithi027.stdout:(6/23): python3-ceph-argparse-18.0.0-1954.g1a67 598 kB/s | 45 kB 00:00 2023-01-24T14:28:16.514 INFO:teuthology.orchestra.run.smithi027.stdout:(7/23): python3-ceph-common-18.0.0-1954.g1a671c 1.7 MB/s | 115 kB 00:00 2023-01-24T14:28:16.562 INFO:teuthology.orchestra.run.smithi035.stdout:(1/23): ceph-base-18.0.0-1954.g1a671c19.el8.x86 2.0 MB/s | 6.3 MB 00:03 2023-01-24T14:28:16.614 INFO:teuthology.orchestra.run.smithi027.stdout:(8/23): python3-cephfs-18.0.0-1954.g1a671c19.el 1.9 MB/s | 190 kB 00:00 2023-01-24T14:28:16.629 INFO:teuthology.orchestra.run.smithi035.stdout:(2/23): ceph-selinux-18.0.0-1954.g1a671c19.el8. 366 kB/s | 24 kB 00:00 2023-01-24T14:28:16.756 INFO:teuthology.orchestra.run.smithi027.stdout:(9/23): python3-rados-18.0.0-1954.g1a671c19.el8 2.5 MB/s | 371 kB 00:00 2023-01-24T14:28:16.829 INFO:teuthology.orchestra.run.smithi035.stdout:(3/23): libcephfs2-18.0.0-1954.g1a671c19.el8.x8 3.6 MB/s | 741 kB 00:00 2023-01-24T14:28:16.882 INFO:teuthology.orchestra.run.smithi027.stdout:(10/23): python3-rbd-18.0.0-1954.g1a671c19.el8. 2.8 MB/s | 364 kB 00:00 2023-01-24T14:28:16.974 INFO:teuthology.orchestra.run.smithi027.stdout:(11/23): python3-rgw-18.0.0-1954.g1a671c19.el8. 1.1 MB/s | 106 kB 00:00 2023-01-24T14:28:17.004 INFO:teuthology.orchestra.run.smithi035.stdout:(4/23): libradosstriper1-18.0.0-1954.g1a671c19. 2.6 MB/s | 461 kB 00:00 2023-01-24T14:28:17.032 INFO:teuthology.orchestra.run.smithi027.stdout:(12/23): librdkafka-0.11.4-3.el8.x86_64.rpm 5.9 MB/s | 354 kB 00:00 2023-01-24T14:28:17.058 INFO:teuthology.orchestra.run.smithi027.stdout:(13/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 10 MB/s | 259 kB 00:00 2023-01-24T14:28:17.083 INFO:teuthology.orchestra.run.smithi027.stdout:(14/23): python3-prettytable-0.7.2-14.el8.noarc 1.7 MB/s | 44 kB 00:00 2023-01-24T14:28:17.133 INFO:teuthology.orchestra.run.smithi027.stdout:(15/23): librabbitmq-0.9.0-3.el8.x86_64.rpm 937 kB/s | 47 kB 00:00 2023-01-24T14:28:17.434 INFO:teuthology.orchestra.run.smithi027.stdout:(16/23): gperftools-libs-2.7-9.el8.x86_64.rpm 1.0 MB/s | 306 kB 00:00 2023-01-24T14:28:17.509 INFO:teuthology.orchestra.run.smithi027.stdout:(17/23): liboath-2.6.2-3.el8.x86_64.rpm 786 kB/s | 59 kB 00:00 2023-01-24T14:28:17.584 INFO:teuthology.orchestra.run.smithi027.stdout:(18/23): libunwind-1.3.1-3.el8.x86_64.rpm 1.0 MB/s | 75 kB 00:00 2023-01-24T14:28:17.868 INFO:teuthology.orchestra.run.smithi027.stdout:(19/23): librgw2-18.0.0-1954.g1a671c19.el8.x86_ 2.5 MB/s | 7.2 MB 00:02 2023-01-24T14:28:17.921 INFO:teuthology.orchestra.run.smithi035.stdout:(5/23): librgw2-18.0.0-1954.g1a671c19.el8.x86_6 7.8 MB/s | 7.2 MB 00:00 2023-01-24T14:28:17.989 INFO:teuthology.orchestra.run.smithi035.stdout:(6/23): python3-ceph-argparse-18.0.0-1954.g1a67 672 kB/s | 45 kB 00:00 2023-01-24T14:28:18.080 INFO:teuthology.orchestra.run.smithi035.stdout:(7/23): python3-ceph-common-18.0.0-1954.g1a671c 1.2 MB/s | 115 kB 00:00 2023-01-24T14:28:18.164 INFO:teuthology.orchestra.run.smithi035.stdout:(8/23): python3-cephfs-18.0.0-1954.g1a671c19.el 2.2 MB/s | 190 kB 00:00 2023-01-24T14:28:18.186 INFO:teuthology.orchestra.run.smithi027.stdout:(20/23): thrift-0.13.0-2.el8.x86_64.rpm 2.9 MB/s | 1.7 MB 00:00 2023-01-24T14:28:18.256 INFO:teuthology.orchestra.run.smithi035.stdout:(9/23): python3-rados-18.0.0-1954.g1a671c19.el8 4.0 MB/s | 371 kB 00:00 2023-01-24T14:28:18.348 INFO:teuthology.orchestra.run.smithi035.stdout:(10/23): python3-rbd-18.0.0-1954.g1a671c19.el8. 3.9 MB/s | 364 kB 00:00 2023-01-24T14:28:18.431 INFO:teuthology.orchestra.run.smithi035.stdout:(11/23): python3-rgw-18.0.0-1954.g1a671c19.el8. 1.3 MB/s | 106 kB 00:00 2023-01-24T14:28:18.490 INFO:teuthology.orchestra.run.smithi035.stdout:(12/23): librdkafka-0.11.4-3.el8.x86_64.rpm 5.9 MB/s | 354 kB 00:00 2023-01-24T14:28:18.515 INFO:teuthology.orchestra.run.smithi035.stdout:(13/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 10 MB/s | 259 kB 00:00 2023-01-24T14:28:18.541 INFO:teuthology.orchestra.run.smithi035.stdout:(14/23): python3-prettytable-0.7.2-14.el8.noarc 1.7 MB/s | 44 kB 00:00 2023-01-24T14:28:18.566 INFO:teuthology.orchestra.run.smithi035.stdout:(15/23): librabbitmq-0.9.0-3.el8.x86_64.rpm 1.8 MB/s | 47 kB 00:00 2023-01-24T14:28:18.883 INFO:teuthology.orchestra.run.smithi035.stdout:(16/23): gperftools-libs-2.7-9.el8.x86_64.rpm 967 kB/s | 306 kB 00:00 2023-01-24T14:28:18.933 INFO:teuthology.orchestra.run.smithi035.stdout:(17/23): liboath-2.6.2-3.el8.x86_64.rpm 1.2 MB/s | 59 kB 00:00 2023-01-24T14:28:18.992 INFO:teuthology.orchestra.run.smithi035.stdout:(18/23): libunwind-1.3.1-3.el8.x86_64.rpm 1.3 MB/s | 75 kB 00:00 2023-01-24T14:28:19.044 INFO:teuthology.orchestra.run.smithi027.stdout:(21/23): ceph-common-18.0.0-1954.g1a671c19.el8. 4.2 MB/s | 27 MB 00:06 2023-01-24T14:28:19.102 INFO:teuthology.orchestra.run.smithi027.stdout:(22/23): librados2-18.0.0-1954.g1a671c19.el8.x8 2.7 MB/s | 3.3 MB 00:01 2023-01-24T14:28:19.236 INFO:teuthology.orchestra.run.smithi027.stdout:(23/23): librbd1-18.0.0-1954.g1a671c19.el8.x86_ 3.4 MB/s | 3.5 MB 00:01 2023-01-24T14:28:19.239 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2023-01-24T14:28:19.239 INFO:teuthology.orchestra.run.smithi027.stdout:Total 10 MB/s | 67 MB 00:06 2023-01-24T14:28:19.283 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction check 2023-01-24T14:28:19.333 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction check succeeded. 2023-01-24T14:28:19.334 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction test 2023-01-24T14:28:19.409 INFO:teuthology.orchestra.run.smithi035.stdout:(19/23): ceph-radosgw-18.0.0-1954.g1a671c19.el8 2.4 MB/s | 14 MB 00:05 2023-01-24T14:28:19.543 INFO:teuthology.orchestra.run.smithi035.stdout:(20/23): thrift-0.13.0-2.el8.x86_64.rpm 3.2 MB/s | 1.7 MB 00:00 2023-01-24T14:28:19.826 INFO:teuthology.orchestra.run.smithi035.stdout:(21/23): librados2-18.0.0-1954.g1a671c19.el8.x8 7.9 MB/s | 3.3 MB 00:00 2023-01-24T14:28:19.998 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction test succeeded. 2023-01-24T14:28:20.001 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction 2023-01-24T14:28:20.360 INFO:teuthology.orchestra.run.smithi035.stdout:(22/23): librbd1-18.0.0-1954.g1a671c19.el8.x86_ 4.3 MB/s | 3.5 MB 00:00 2023-01-24T14:28:21.171 INFO:teuthology.orchestra.run.smithi027.stdout: Preparing : 1/1 2023-01-24T14:28:21.423 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : liboath-2.6.2-3.el8.x86_64 1/25 2023-01-24T14:28:21.591 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 2/25 2023-01-24T14:28:21.785 INFO:teuthology.orchestra.run.smithi035.stdout:(23/23): ceph-common-18.0.0-1954.g1a671c19.el8. 3.2 MB/s | 27 MB 00:08 2023-01-24T14:28:21.788 INFO:teuthology.orchestra.run.smithi035.stdout:-------------------------------------------------------------------------------- 2023-01-24T14:28:21.788 INFO:teuthology.orchestra.run.smithi035.stdout:Total 8.0 MB/s | 67 MB 00:08 2023-01-24T14:28:21.833 INFO:teuthology.orchestra.run.smithi035.stdout:Running transaction check 2023-01-24T14:28:21.882 INFO:teuthology.orchestra.run.smithi035.stdout:Transaction check succeeded. 2023-01-24T14:28:21.883 INFO:teuthology.orchestra.run.smithi035.stdout:Running transaction test 2023-01-24T14:28:22.036 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 2/25 2023-01-24T14:28:22.257 INFO:teuthology.orchestra.run.smithi027.stdout: Upgrading : librados2-2:18.0.0-1954.g1a671c19.el8.x86_64 3/25 2023-01-24T14:28:22.354 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: librados2-2:18.0.0-1954.g1a671c19.el8.x86_64 3/25 2023-01-24T14:28:22.531 INFO:teuthology.orchestra.run.smithi035.stdout:Transaction test succeeded. 2023-01-24T14:28:22.535 INFO:teuthology.orchestra.run.smithi035.stdout:Running transaction 2023-01-24T14:28:22.593 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-rados-2:18.0.0-1954.g1a671c19.el8.x86_64 4/25 2023-01-24T14:28:23.032 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libcephfs2-2:18.0.0-1954.g1a671c19.el8.x86_64 5/25 2023-01-24T14:28:23.488 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: libcephfs2-2:18.0.0-1954.g1a671c19.el8.x86_64 5/25 2023-01-24T14:28:23.657 INFO:teuthology.orchestra.run.smithi027.stdout: Upgrading : librbd1-2:18.0.0-1954.g1a671c19.el8.x86_64 6/25 2023-01-24T14:28:23.687 INFO:teuthology.orchestra.run.smithi035.stdout: Preparing : 1/1 2023-01-24T14:28:23.903 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: librbd1-2:18.0.0-1954.g1a671c19.el8.x86_64 6/25 2023-01-24T14:28:23.923 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : liboath-2.6.2-3.el8.x86_64 1/25 2023-01-24T14:28:24.117 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 2/25 2023-01-24T14:28:24.208 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : thrift-0.13.0-2.el8.x86_64 7/25 2023-01-24T14:28:24.399 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 8/25 2023-01-24T14:28:24.524 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 9/25 2023-01-24T14:28:24.529 INFO:teuthology.orchestra.run.smithi035.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 2/25 2023-01-24T14:28:24.587 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 9/25 2023-01-24T14:28:24.711 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-ceph-argparse-2:18.0.0-1954.g1a671c19.el8. 10/25 2023-01-24T14:28:24.717 INFO:teuthology.orchestra.run.smithi035.stdout: Upgrading : librados2-2:18.0.0-1954.g1a671c19.el8.x86_64 3/25 2023-01-24T14:28:24.819 INFO:teuthology.orchestra.run.smithi035.stdout: Running scriptlet: librados2-2:18.0.0-1954.g1a671c19.el8.x86_64 3/25 2023-01-24T14:28:24.928 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-cephfs-2:18.0.0-1954.g1a671c19.el8.x86_64 11/25 2023-01-24T14:28:25.121 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : python3-rados-2:18.0.0-1954.g1a671c19.el8.x86_64 4/25 2023-01-24T14:28:25.145 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-rbd-2:18.0.0-1954.g1a671c19.el8.x86_64 12/25 2023-01-24T14:28:25.299 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libradosstriper1-2:18.0.0-1954.g1a671c19.el8.x86_6 13/25 2023-01-24T14:28:25.325 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : libcephfs2-2:18.0.0-1954.g1a671c19.el8.x86_64 5/25 2023-01-24T14:28:25.367 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: libradosstriper1-2:18.0.0-1954.g1a671c19.el8.x86_6 13/25 2023-01-24T14:28:25.544 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 14/25 2023-01-24T14:28:25.747 INFO:teuthology.orchestra.run.smithi035.stdout: Running scriptlet: libcephfs2-2:18.0.0-1954.g1a671c19.el8.x86_64 5/25 2023-01-24T14:28:25.908 INFO:teuthology.orchestra.run.smithi035.stdout: Upgrading : librbd1-2:18.0.0-1954.g1a671c19.el8.x86_64 6/25 2023-01-24T14:28:26.138 INFO:teuthology.orchestra.run.smithi035.stdout: Running scriptlet: librbd1-2:18.0.0-1954.g1a671c19.el8.x86_64 6/25 2023-01-24T14:28:26.314 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : thrift-0.13.0-2.el8.x86_64 7/25 2023-01-24T14:28:26.483 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 8/25 2023-01-24T14:28:26.544 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 15/25 2023-01-24T14:28:26.625 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 9/25 2023-01-24T14:28:26.694 INFO:teuthology.orchestra.run.smithi035.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 9/25 2023-01-24T14:28:26.724 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : librgw2-2:18.0.0-1954.g1a671c19.el8.x86_64 16/25 2023-01-24T14:28:26.832 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: librgw2-2:18.0.0-1954.g1a671c19.el8.x86_64 16/25 2023-01-24T14:28:26.845 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : python3-ceph-argparse-2:18.0.0-1954.g1a671c19.el8. 10/25 2023-01-24T14:28:27.032 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-rgw-2:18.0.0-1954.g1a671c19.el8.x86_64 17/25 2023-01-24T14:28:27.088 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : python3-cephfs-2:18.0.0-1954.g1a671c19.el8.x86_64 11/25 2023-01-24T14:28:27.209 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 18/25 2023-01-24T14:28:27.323 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : python3-rbd-2:18.0.0-1954.g1a671c19.el8.x86_64 12/25 2023-01-24T14:28:27.399 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-ceph-common-2:18.0.0-1954.g1a671c19.el8.x8 19/25 2023-01-24T14:28:27.483 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : libradosstriper1-2:18.0.0-1954.g1a671c19.el8.x86_6 13/25 2023-01-24T14:28:27.558 INFO:teuthology.orchestra.run.smithi035.stdout: Running scriptlet: libradosstriper1-2:18.0.0-1954.g1a671c19.el8.x86_6 13/25 2023-01-24T14:28:27.766 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 14/25 2023-01-24T14:28:28.778 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 15/25 2023-01-24T14:28:28.975 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : librgw2-2:18.0.0-1954.g1a671c19.el8.x86_64 16/25 2023-01-24T14:28:29.029 INFO:teuthology.orchestra.run.smithi035.stdout: Running scriptlet: librgw2-2:18.0.0-1954.g1a671c19.el8.x86_64 16/25 2023-01-24T14:28:29.241 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : python3-rgw-2:18.0.0-1954.g1a671c19.el8.x86_64 17/25 2023-01-24T14:28:29.437 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 18/25 2023-01-24T14:28:29.659 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : python3-ceph-common-2:18.0.0-1954.g1a671c19.el8.x8 19/25 2023-01-24T14:28:32.011 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-common-2:18.0.0-1954.g1a671c19.el8.x86_64 20/25 2023-01-24T14:28:32.215 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-common-2:18.0.0-1954.g1a671c19.el8.x86_64 20/25 2023-01-24T14:28:33.006 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-common-2:18.0.0-1954.g1a671c19.el8.x86_64 20/25 2023-01-24T14:28:33.302 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-base-2:18.0.0-1954.g1a671c19.el8.x86_64 21/25 2023-01-24T14:28:33.421 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-base-2:18.0.0-1954.g1a671c19.el8.x86_64 21/25 2023-01-24T14:28:33.790 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-selinux-2:18.0.0-1954.g1a671c19.el8.x86_64 22/25 2023-01-24T14:28:34.171 INFO:teuthology.orchestra.run.smithi035.stdout: Running scriptlet: ceph-common-2:18.0.0-1954.g1a671c19.el8.x86_64 20/25 2023-01-24T14:28:34.424 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : ceph-common-2:18.0.0-1954.g1a671c19.el8.x86_64 20/25 2023-01-24T14:28:35.205 INFO:teuthology.orchestra.run.smithi035.stdout: Running scriptlet: ceph-common-2:18.0.0-1954.g1a671c19.el8.x86_64 20/25 2023-01-24T14:28:35.566 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : ceph-base-2:18.0.0-1954.g1a671c19.el8.x86_64 21/25 2023-01-24T14:28:35.828 INFO:teuthology.orchestra.run.smithi035.stdout: Running scriptlet: ceph-base-2:18.0.0-1954.g1a671c19.el8.x86_64 21/25 2023-01-24T14:28:35.974 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : ceph-selinux-2:18.0.0-1954.g1a671c19.el8.x86_64 22/25 2023-01-24T14:28:54.043 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-selinux-2:18.0.0-1954.g1a671c19.el8.x86_64 22/25 2023-01-24T14:28:54.043 INFO:teuthology.orchestra.run.smithi027.stdout:skipping the directory /sys 2023-01-24T14:28:54.044 INFO:teuthology.orchestra.run.smithi027.stdout:skipping the directory /proc 2023-01-24T14:28:54.044 INFO:teuthology.orchestra.run.smithi027.stdout:skipping the directory /dev 2023-01-24T14:28:54.044 INFO:teuthology.orchestra.run.smithi027.stdout:skipping the directory /run 2023-01-24T14:28:54.044 INFO:teuthology.orchestra.run.smithi027.stdout:skipping the directory /mnt 2023-01-24T14:28:54.044 INFO:teuthology.orchestra.run.smithi027.stdout:skipping the directory /var/tmp 2023-01-24T14:28:54.045 INFO:teuthology.orchestra.run.smithi027.stdout:skipping the directory /home 2023-01-24T14:28:54.045 INFO:teuthology.orchestra.run.smithi027.stdout:skipping the directory /tmp 2023-01-24T14:28:54.045 INFO:teuthology.orchestra.run.smithi027.stdout:skipping the directory /dev 2023-01-24T14:28:54.046 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-24T14:28:55.758 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-radosgw-2:18.0.0-1954.g1a671c19.el8.x86_64 23/25 2023-01-24T14:28:55.779 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-1954.g1a671c19.el8.x86_64 23/25 2023-01-24T14:28:55.780 INFO:teuthology.orchestra.run.smithi027.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2023-01-24T14:28:55.989 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2023-01-24T14:28:55.990 INFO:teuthology.orchestra.run.smithi027.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2023-01-24T14:28:56.459 INFO:teuthology.orchestra.run.smithi035.stdout: Running scriptlet: ceph-selinux-2:18.0.0-1954.g1a671c19.el8.x86_64 22/25 2023-01-24T14:28:56.459 INFO:teuthology.orchestra.run.smithi035.stdout:skipping the directory /sys 2023-01-24T14:28:56.460 INFO:teuthology.orchestra.run.smithi035.stdout:skipping the directory /proc 2023-01-24T14:28:56.460 INFO:teuthology.orchestra.run.smithi035.stdout:skipping the directory /dev 2023-01-24T14:28:56.460 INFO:teuthology.orchestra.run.smithi035.stdout:skipping the directory /run 2023-01-24T14:28:56.460 INFO:teuthology.orchestra.run.smithi035.stdout:skipping the directory /mnt 2023-01-24T14:28:56.460 INFO:teuthology.orchestra.run.smithi035.stdout:skipping the directory /var/tmp 2023-01-24T14:28:56.461 INFO:teuthology.orchestra.run.smithi035.stdout:skipping the directory /home 2023-01-24T14:28:56.461 INFO:teuthology.orchestra.run.smithi035.stdout:skipping the directory /tmp 2023-01-24T14:28:56.461 INFO:teuthology.orchestra.run.smithi035.stdout:skipping the directory /dev 2023-01-24T14:28:56.461 INFO:teuthology.orchestra.run.smithi035.stdout: 2023-01-24T14:28:56.714 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2023-01-24T14:28:56.715 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-base-2:18.0.0-1954.g1a671c19.el8.x86_64 1/25 2023-01-24T14:28:56.715 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-common-2:18.0.0-1954.g1a671c19.el8.x86_64 2/25 2023-01-24T14:28:56.715 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-radosgw-2:18.0.0-1954.g1a671c19.el8.x86_64 3/25 2023-01-24T14:28:56.715 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-selinux-2:18.0.0-1954.g1a671c19.el8.x86_64 4/25 2023-01-24T14:28:56.715 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libcephfs2-2:18.0.0-1954.g1a671c19.el8.x86_64 5/25 2023-01-24T14:28:56.716 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libradosstriper1-2:18.0.0-1954.g1a671c19.el8.x86_6 6/25 2023-01-24T14:28:56.716 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : librgw2-2:18.0.0-1954.g1a671c19.el8.x86_64 7/25 2023-01-24T14:28:56.716 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-ceph-argparse-2:18.0.0-1954.g1a671c19.el8. 8/25 2023-01-24T14:28:56.716 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-ceph-common-2:18.0.0-1954.g1a671c19.el8.x8 9/25 2023-01-24T14:28:56.716 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-cephfs-2:18.0.0-1954.g1a671c19.el8.x86_64 10/25 2023-01-24T14:28:56.717 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-rados-2:18.0.0-1954.g1a671c19.el8.x86_64 11/25 2023-01-24T14:28:56.717 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-rbd-2:18.0.0-1954.g1a671c19.el8.x86_64 12/25 2023-01-24T14:28:56.717 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-rgw-2:18.0.0-1954.g1a671c19.el8.x86_64 13/25 2023-01-24T14:28:56.717 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 14/25 2023-01-24T14:28:56.717 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 15/25 2023-01-24T14:28:56.718 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 16/25 2023-01-24T14:28:56.718 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 17/25 2023-01-24T14:28:56.718 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 18/25 2023-01-24T14:28:56.718 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 19/25 2023-01-24T14:28:56.718 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 20/25 2023-01-24T14:28:56.719 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 21/25 2023-01-24T14:28:56.719 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : librados2-2:18.0.0-1954.g1a671c19.el8.x86_64 22/25 2023-01-24T14:28:56.719 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2023-01-24T14:28:56.719 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : librbd1-2:18.0.0-1954.g1a671c19.el8.x86_64 24/25 2023-01-24T14:28:57.665 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2023-01-24T14:28:57.665 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-24T14:28:57.665 INFO:teuthology.orchestra.run.smithi027.stdout:Upgraded: 2023-01-24T14:28:57.666 INFO:teuthology.orchestra.run.smithi027.stdout: librados2-2:18.0.0-1954.g1a671c19.el8.x86_64 2023-01-24T14:28:57.666 INFO:teuthology.orchestra.run.smithi027.stdout: librbd1-2:18.0.0-1954.g1a671c19.el8.x86_64 2023-01-24T14:28:57.666 INFO:teuthology.orchestra.run.smithi027.stdout:Installed: 2023-01-24T14:28:57.666 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-base-2:18.0.0-1954.g1a671c19.el8.x86_64 2023-01-24T14:28:57.666 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-common-2:18.0.0-1954.g1a671c19.el8.x86_64 2023-01-24T14:28:57.667 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-radosgw-2:18.0.0-1954.g1a671c19.el8.x86_64 2023-01-24T14:28:57.667 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-selinux-2:18.0.0-1954.g1a671c19.el8.x86_64 2023-01-24T14:28:57.667 INFO:teuthology.orchestra.run.smithi027.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-01-24T14:28:57.667 INFO:teuthology.orchestra.run.smithi027.stdout: libcephfs2-2:18.0.0-1954.g1a671c19.el8.x86_64 2023-01-24T14:28:57.667 INFO:teuthology.orchestra.run.smithi027.stdout: liboath-2.6.2-3.el8.x86_64 2023-01-24T14:28:57.668 INFO:teuthology.orchestra.run.smithi027.stdout: librabbitmq-0.9.0-3.el8.x86_64 2023-01-24T14:28:57.668 INFO:teuthology.orchestra.run.smithi027.stdout: libradosstriper1-2:18.0.0-1954.g1a671c19.el8.x86_64 2023-01-24T14:28:57.668 INFO:teuthology.orchestra.run.smithi027.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-01-24T14:28:57.668 INFO:teuthology.orchestra.run.smithi027.stdout: librgw2-2:18.0.0-1954.g1a671c19.el8.x86_64 2023-01-24T14:28:57.668 INFO:teuthology.orchestra.run.smithi027.stdout: libunwind-1.3.1-3.el8.x86_64 2023-01-24T14:28:57.669 INFO:teuthology.orchestra.run.smithi027.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-01-24T14:28:57.669 INFO:teuthology.orchestra.run.smithi027.stdout: python3-ceph-argparse-2:18.0.0-1954.g1a671c19.el8.x86_64 2023-01-24T14:28:57.669 INFO:teuthology.orchestra.run.smithi027.stdout: python3-ceph-common-2:18.0.0-1954.g1a671c19.el8.x86_64 2023-01-24T14:28:57.669 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cephfs-2:18.0.0-1954.g1a671c19.el8.x86_64 2023-01-24T14:28:57.669 INFO:teuthology.orchestra.run.smithi027.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-01-24T14:28:57.670 INFO:teuthology.orchestra.run.smithi027.stdout: python3-rados-2:18.0.0-1954.g1a671c19.el8.x86_64 2023-01-24T14:28:57.670 INFO:teuthology.orchestra.run.smithi027.stdout: python3-rbd-2:18.0.0-1954.g1a671c19.el8.x86_64 2023-01-24T14:28:57.670 INFO:teuthology.orchestra.run.smithi027.stdout: python3-rgw-2:18.0.0-1954.g1a671c19.el8.x86_64 2023-01-24T14:28:57.670 INFO:teuthology.orchestra.run.smithi027.stdout: thrift-0.13.0-2.el8.x86_64 2023-01-24T14:28:57.670 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-24T14:28:57.671 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2023-01-24T14:28:57.951 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install ceph-test 2023-01-24T14:28:58.197 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : ceph-radosgw-2:18.0.0-1954.g1a671c19.el8.x86_64 23/25 2023-01-24T14:28:58.218 INFO:teuthology.orchestra.run.smithi035.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-1954.g1a671c19.el8.x86_64 23/25 2023-01-24T14:28:58.219 INFO:teuthology.orchestra.run.smithi035.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2023-01-24T14:28:58.423 INFO:teuthology.orchestra.run.smithi035.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2023-01-24T14:28:58.424 INFO:teuthology.orchestra.run.smithi035.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2023-01-24T14:28:58.711 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:00:50 ago on Tue 24 Jan 2023 02:28:08 PM UTC. 2023-01-24T14:28:59.160 INFO:teuthology.orchestra.run.smithi035.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2023-01-24T14:28:59.160 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : ceph-base-2:18.0.0-1954.g1a671c19.el8.x86_64 1/25 2023-01-24T14:28:59.160 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : ceph-common-2:18.0.0-1954.g1a671c19.el8.x86_64 2/25 2023-01-24T14:28:59.160 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : ceph-radosgw-2:18.0.0-1954.g1a671c19.el8.x86_64 3/25 2023-01-24T14:28:59.161 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : ceph-selinux-2:18.0.0-1954.g1a671c19.el8.x86_64 4/25 2023-01-24T14:28:59.161 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : libcephfs2-2:18.0.0-1954.g1a671c19.el8.x86_64 5/25 2023-01-24T14:28:59.161 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : libradosstriper1-2:18.0.0-1954.g1a671c19.el8.x86_6 6/25 2023-01-24T14:28:59.161 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : librgw2-2:18.0.0-1954.g1a671c19.el8.x86_64 7/25 2023-01-24T14:28:59.161 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : python3-ceph-argparse-2:18.0.0-1954.g1a671c19.el8. 8/25 2023-01-24T14:28:59.162 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : python3-ceph-common-2:18.0.0-1954.g1a671c19.el8.x8 9/25 2023-01-24T14:28:59.162 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : python3-cephfs-2:18.0.0-1954.g1a671c19.el8.x86_64 10/25 2023-01-24T14:28:59.162 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : python3-rados-2:18.0.0-1954.g1a671c19.el8.x86_64 11/25 2023-01-24T14:28:59.162 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : python3-rbd-2:18.0.0-1954.g1a671c19.el8.x86_64 12/25 2023-01-24T14:28:59.162 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : python3-rgw-2:18.0.0-1954.g1a671c19.el8.x86_64 13/25 2023-01-24T14:28:59.163 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 14/25 2023-01-24T14:28:59.163 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 15/25 2023-01-24T14:28:59.163 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 16/25 2023-01-24T14:28:59.163 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 17/25 2023-01-24T14:28:59.163 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 18/25 2023-01-24T14:28:59.164 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 19/25 2023-01-24T14:28:59.164 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 20/25 2023-01-24T14:28:59.164 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 21/25 2023-01-24T14:28:59.164 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : librados2-2:18.0.0-1954.g1a671c19.el8.x86_64 22/25 2023-01-24T14:28:59.164 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2023-01-24T14:28:59.165 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : librbd1-2:18.0.0-1954.g1a671c19.el8.x86_64 24/25 2023-01-24T14:28:59.936 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2023-01-24T14:28:59.937 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2023-01-24T14:28:59.937 INFO:teuthology.orchestra.run.smithi027.stdout: Package Arch Version Repository Size 2023-01-24T14:28:59.938 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2023-01-24T14:28:59.939 INFO:teuthology.orchestra.run.smithi027.stdout:Installing: 2023-01-24T14:28:59.939 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-test x86_64 2:18.0.0-1954.g1a671c19.el8 ceph 149 M 2023-01-24T14:28:59.939 INFO:teuthology.orchestra.run.smithi027.stdout:Installing dependencies: 2023-01-24T14:28:59.940 INFO:teuthology.orchestra.run.smithi027.stdout: cryptopp x86_64 8.6.0-1.el8 epel 1.3 M 2023-01-24T14:28:59.940 INFO:teuthology.orchestra.run.smithi027.stdout: jq x86_64 1.6-6.el8 CentOS-AppStream 202 k 2023-01-24T14:28:59.940 INFO:teuthology.orchestra.run.smithi027.stdout: libasan6 x86_64 11.2.1-9.1.el8 CentOS-AppStream 402 k 2023-01-24T14:28:59.941 INFO:teuthology.orchestra.run.smithi027.stdout: libatomic x86_64 8.5.0-18.el8 CentOS-BaseOS 25 k 2023-01-24T14:28:59.942 INFO:teuthology.orchestra.run.smithi027.stdout: libcephsqlite x86_64 2:18.0.0-1954.g1a671c19.el8 ceph 134 k 2023-01-24T14:28:59.942 INFO:teuthology.orchestra.run.smithi027.stdout: libubsan x86_64 8.5.0-18.el8 CentOS-BaseOS 152 k 2023-01-24T14:28:59.943 INFO:teuthology.orchestra.run.smithi027.stdout: oniguruma x86_64 6.8.2-2.el8 CentOS-AppStream 187 k 2023-01-24T14:28:59.943 INFO:teuthology.orchestra.run.smithi027.stdout: socat x86_64 1.7.4.1-1.el8 CentOS-AppStream 323 k 2023-01-24T14:28:59.943 INFO:teuthology.orchestra.run.smithi027.stdout: xmlstarlet x86_64 1.6.1-20.el8 CentOS-AppStream 71 k 2023-01-24T14:28:59.944 INFO:teuthology.orchestra.run.smithi027.stdout: yaml-cpp x86_64 0.6.2-3.el8 epel 171 k 2023-01-24T14:28:59.944 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-24T14:28:59.944 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2023-01-24T14:28:59.945 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2023-01-24T14:28:59.945 INFO:teuthology.orchestra.run.smithi027.stdout:Install 11 Packages 2023-01-24T14:28:59.946 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-24T14:28:59.947 INFO:teuthology.orchestra.run.smithi027.stdout:Total download size: 152 M 2023-01-24T14:28:59.947 INFO:teuthology.orchestra.run.smithi027.stdout:Installed size: 1.7 G 2023-01-24T14:28:59.948 INFO:teuthology.orchestra.run.smithi027.stdout:Downloading Packages: 2023-01-24T14:29:00.006 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2023-01-24T14:29:00.006 INFO:teuthology.orchestra.run.smithi035.stdout: 2023-01-24T14:29:00.006 INFO:teuthology.orchestra.run.smithi035.stdout:Upgraded: 2023-01-24T14:29:00.006 INFO:teuthology.orchestra.run.smithi035.stdout: librados2-2:18.0.0-1954.g1a671c19.el8.x86_64 2023-01-24T14:29:00.007 INFO:teuthology.orchestra.run.smithi035.stdout: librbd1-2:18.0.0-1954.g1a671c19.el8.x86_64 2023-01-24T14:29:00.007 INFO:teuthology.orchestra.run.smithi035.stdout:Installed: 2023-01-24T14:29:00.007 INFO:teuthology.orchestra.run.smithi035.stdout: ceph-base-2:18.0.0-1954.g1a671c19.el8.x86_64 2023-01-24T14:29:00.008 INFO:teuthology.orchestra.run.smithi035.stdout: ceph-common-2:18.0.0-1954.g1a671c19.el8.x86_64 2023-01-24T14:29:00.008 INFO:teuthology.orchestra.run.smithi035.stdout: ceph-radosgw-2:18.0.0-1954.g1a671c19.el8.x86_64 2023-01-24T14:29:00.008 INFO:teuthology.orchestra.run.smithi035.stdout: ceph-selinux-2:18.0.0-1954.g1a671c19.el8.x86_64 2023-01-24T14:29:00.009 INFO:teuthology.orchestra.run.smithi035.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-01-24T14:29:00.009 INFO:teuthology.orchestra.run.smithi035.stdout: libcephfs2-2:18.0.0-1954.g1a671c19.el8.x86_64 2023-01-24T14:29:00.009 INFO:teuthology.orchestra.run.smithi035.stdout: liboath-2.6.2-3.el8.x86_64 2023-01-24T14:29:00.009 INFO:teuthology.orchestra.run.smithi035.stdout: librabbitmq-0.9.0-3.el8.x86_64 2023-01-24T14:29:00.009 INFO:teuthology.orchestra.run.smithi035.stdout: libradosstriper1-2:18.0.0-1954.g1a671c19.el8.x86_64 2023-01-24T14:29:00.010 INFO:teuthology.orchestra.run.smithi035.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-01-24T14:29:00.010 INFO:teuthology.orchestra.run.smithi035.stdout: librgw2-2:18.0.0-1954.g1a671c19.el8.x86_64 2023-01-24T14:29:00.010 INFO:teuthology.orchestra.run.smithi035.stdout: libunwind-1.3.1-3.el8.x86_64 2023-01-24T14:29:00.010 INFO:teuthology.orchestra.run.smithi035.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-01-24T14:29:00.011 INFO:teuthology.orchestra.run.smithi035.stdout: python3-ceph-argparse-2:18.0.0-1954.g1a671c19.el8.x86_64 2023-01-24T14:29:00.011 INFO:teuthology.orchestra.run.smithi035.stdout: python3-ceph-common-2:18.0.0-1954.g1a671c19.el8.x86_64 2023-01-24T14:29:00.011 INFO:teuthology.orchestra.run.smithi035.stdout: python3-cephfs-2:18.0.0-1954.g1a671c19.el8.x86_64 2023-01-24T14:29:00.011 INFO:teuthology.orchestra.run.smithi035.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-01-24T14:29:00.012 INFO:teuthology.orchestra.run.smithi035.stdout: python3-rados-2:18.0.0-1954.g1a671c19.el8.x86_64 2023-01-24T14:29:00.012 INFO:teuthology.orchestra.run.smithi035.stdout: python3-rbd-2:18.0.0-1954.g1a671c19.el8.x86_64 2023-01-24T14:29:00.012 INFO:teuthology.orchestra.run.smithi035.stdout: python3-rgw-2:18.0.0-1954.g1a671c19.el8.x86_64 2023-01-24T14:29:00.012 INFO:teuthology.orchestra.run.smithi035.stdout: thrift-0.13.0-2.el8.x86_64 2023-01-24T14:29:00.013 INFO:teuthology.orchestra.run.smithi035.stdout: 2023-01-24T14:29:00.013 INFO:teuthology.orchestra.run.smithi035.stdout:Complete! 2023-01-24T14:29:00.014 INFO:teuthology.orchestra.run.smithi027.stdout:(1/11): jq-1.6-6.el8.x86_64.rpm 2.9 MB/s | 202 kB 00:00 2023-01-24T14:29:00.051 INFO:teuthology.orchestra.run.smithi027.stdout:(2/11): libasan6-11.2.1-9.1.el8.x86_64.rpm 12 MB/s | 402 kB 00:00 2023-01-24T14:29:00.076 INFO:teuthology.orchestra.run.smithi027.stdout:(3/11): oniguruma-6.8.2-2.el8.x86_64.rpm 7.4 MB/s | 187 kB 00:00 2023-01-24T14:29:00.127 INFO:teuthology.orchestra.run.smithi027.stdout:(4/11): socat-1.7.4.1-1.el8.x86_64.rpm 6.4 MB/s | 323 kB 00:00 2023-01-24T14:29:00.152 INFO:teuthology.orchestra.run.smithi027.stdout:(5/11): xmlstarlet-1.6.1-20.el8.x86_64.rpm 2.8 MB/s | 71 kB 00:00 2023-01-24T14:29:00.178 INFO:teuthology.orchestra.run.smithi027.stdout:(6/11): libatomic-8.5.0-18.el8.x86_64.rpm 1.0 MB/s | 25 kB 00:00 2023-01-24T14:29:00.203 INFO:teuthology.orchestra.run.smithi027.stdout:(7/11): libubsan-8.5.0-18.el8.x86_64.rpm 5.9 MB/s | 152 kB 00:00 2023-01-24T14:29:00.312 DEBUG:teuthology.orchestra.run.smithi035:> sudo yum -y install ceph-test 2023-01-24T14:29:00.461 INFO:teuthology.orchestra.run.smithi027.stdout:(8/11): libcephsqlite-18.0.0-1954.g1a671c19.el8 258 kB/s | 134 kB 00:00 2023-01-24T14:29:00.746 INFO:teuthology.orchestra.run.smithi027.stdout:(9/11): yaml-cpp-0.6.2-3.el8.x86_64.rpm 602 kB/s | 171 kB 00:00 2023-01-24T14:29:01.045 INFO:teuthology.orchestra.run.smithi027.stdout:(10/11): cryptopp-8.6.0-1.el8.x86_64.rpm 1.5 MB/s | 1.3 MB 00:00 2023-01-24T14:29:01.068 INFO:teuthology.orchestra.run.smithi035.stdout:Last metadata expiration check: 0:00:52 ago on Tue 24 Jan 2023 02:28:09 PM UTC. 2023-01-24T14:29:02.299 INFO:teuthology.orchestra.run.smithi035.stdout:Dependencies resolved. 2023-01-24T14:29:02.300 INFO:teuthology.orchestra.run.smithi035.stdout:================================================================================ 2023-01-24T14:29:02.300 INFO:teuthology.orchestra.run.smithi035.stdout: Package Arch Version Repository Size 2023-01-24T14:29:02.300 INFO:teuthology.orchestra.run.smithi035.stdout:================================================================================ 2023-01-24T14:29:02.301 INFO:teuthology.orchestra.run.smithi035.stdout:Installing: 2023-01-24T14:29:02.301 INFO:teuthology.orchestra.run.smithi035.stdout: ceph-test x86_64 2:18.0.0-1954.g1a671c19.el8 ceph 149 M 2023-01-24T14:29:02.301 INFO:teuthology.orchestra.run.smithi035.stdout:Installing dependencies: 2023-01-24T14:29:02.301 INFO:teuthology.orchestra.run.smithi035.stdout: cryptopp x86_64 8.6.0-1.el8 epel 1.3 M 2023-01-24T14:29:02.301 INFO:teuthology.orchestra.run.smithi035.stdout: jq x86_64 1.6-6.el8 CentOS-AppStream 202 k 2023-01-24T14:29:02.302 INFO:teuthology.orchestra.run.smithi035.stdout: libasan6 x86_64 11.2.1-9.1.el8 CentOS-AppStream 402 k 2023-01-24T14:29:02.302 INFO:teuthology.orchestra.run.smithi035.stdout: libatomic x86_64 8.5.0-18.el8 CentOS-BaseOS 25 k 2023-01-24T14:29:02.302 INFO:teuthology.orchestra.run.smithi035.stdout: libcephsqlite x86_64 2:18.0.0-1954.g1a671c19.el8 ceph 134 k 2023-01-24T14:29:02.302 INFO:teuthology.orchestra.run.smithi035.stdout: libubsan x86_64 8.5.0-18.el8 CentOS-BaseOS 152 k 2023-01-24T14:29:02.303 INFO:teuthology.orchestra.run.smithi035.stdout: oniguruma x86_64 6.8.2-2.el8 CentOS-AppStream 187 k 2023-01-24T14:29:02.303 INFO:teuthology.orchestra.run.smithi035.stdout: socat x86_64 1.7.4.1-1.el8 CentOS-AppStream 323 k 2023-01-24T14:29:02.303 INFO:teuthology.orchestra.run.smithi035.stdout: xmlstarlet x86_64 1.6.1-20.el8 CentOS-AppStream 71 k 2023-01-24T14:29:02.304 INFO:teuthology.orchestra.run.smithi035.stdout: yaml-cpp x86_64 0.6.2-3.el8 epel 171 k 2023-01-24T14:29:02.304 INFO:teuthology.orchestra.run.smithi035.stdout: 2023-01-24T14:29:02.304 INFO:teuthology.orchestra.run.smithi035.stdout:Transaction Summary 2023-01-24T14:29:02.305 INFO:teuthology.orchestra.run.smithi035.stdout:================================================================================ 2023-01-24T14:29:02.305 INFO:teuthology.orchestra.run.smithi035.stdout:Install 11 Packages 2023-01-24T14:29:02.305 INFO:teuthology.orchestra.run.smithi035.stdout: 2023-01-24T14:29:02.306 INFO:teuthology.orchestra.run.smithi035.stdout:Total download size: 152 M 2023-01-24T14:29:02.307 INFO:teuthology.orchestra.run.smithi035.stdout:Installed size: 1.7 G 2023-01-24T14:29:02.307 INFO:teuthology.orchestra.run.smithi035.stdout:Downloading Packages: 2023-01-24T14:29:02.374 INFO:teuthology.orchestra.run.smithi035.stdout:(1/11): jq-1.6-6.el8.x86_64.rpm 2.9 MB/s | 202 kB 00:00 2023-01-24T14:29:02.416 INFO:teuthology.orchestra.run.smithi035.stdout:(2/11): libasan6-11.2.1-9.1.el8.x86_64.rpm 12 MB/s | 402 kB 00:00 2023-01-24T14:29:02.441 INFO:teuthology.orchestra.run.smithi035.stdout:(3/11): oniguruma-6.8.2-2.el8.x86_64.rpm 7.3 MB/s | 187 kB 00:00 2023-01-24T14:29:02.474 INFO:teuthology.orchestra.run.smithi035.stdout:(4/11): socat-1.7.4.1-1.el8.x86_64.rpm 9.8 MB/s | 323 kB 00:00 2023-01-24T14:29:02.500 INFO:teuthology.orchestra.run.smithi035.stdout:(5/11): xmlstarlet-1.6.1-20.el8.x86_64.rpm 2.7 MB/s | 71 kB 00:00 2023-01-24T14:29:02.525 INFO:teuthology.orchestra.run.smithi035.stdout:(6/11): libatomic-8.5.0-18.el8.x86_64.rpm 1.0 MB/s | 25 kB 00:00 2023-01-24T14:29:02.550 INFO:teuthology.orchestra.run.smithi035.stdout:(7/11): libubsan-8.5.0-18.el8.x86_64.rpm 5.9 MB/s | 152 kB 00:00 2023-01-24T14:29:02.767 INFO:teuthology.orchestra.run.smithi035.stdout:(8/11): libcephsqlite-18.0.0-1954.g1a671c19.el8 290 kB/s | 134 kB 00:00 2023-01-24T14:29:03.084 INFO:teuthology.orchestra.run.smithi035.stdout:(9/11): yaml-cpp-0.6.2-3.el8.x86_64.rpm 539 kB/s | 171 kB 00:00 2023-01-24T14:29:03.126 INFO:teuthology.orchestra.run.smithi035.stdout:(10/11): cryptopp-8.6.0-1.el8.x86_64.rpm 2.2 MB/s | 1.3 MB 00:00 2023-01-24T14:29:16.894 INFO:teuthology.orchestra.run.smithi027.stdout:(11/11): ceph-test-18.0.0-1954.g1a671c19.el8.x8 8.8 MB/s | 149 MB 00:16 2023-01-24T14:29:16.896 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2023-01-24T14:29:16.896 INFO:teuthology.orchestra.run.smithi027.stdout:Total 8.9 MB/s | 152 MB 00:16 2023-01-24T14:29:16.966 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction check 2023-01-24T14:29:16.991 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction check succeeded. 2023-01-24T14:29:16.992 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction test 2023-01-24T14:29:17.719 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction test succeeded. 2023-01-24T14:29:17.723 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction 2023-01-24T14:29:18.821 INFO:teuthology.orchestra.run.smithi027.stdout: Preparing : 1/1 2023-01-24T14:29:19.136 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : yaml-cpp-0.6.2-3.el8.x86_64 1/11 2023-01-24T14:29:19.321 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : cryptopp-8.6.0-1.el8.x86_64 2/11 2023-01-24T14:29:19.567 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libubsan-8.5.0-18.el8.x86_64 3/11 2023-01-24T14:29:19.617 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: libubsan-8.5.0-18.el8.x86_64 3/11 2023-01-24T14:29:19.708 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libatomic-8.5.0-18.el8.x86_64 4/11 2023-01-24T14:29:19.769 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: libatomic-8.5.0-18.el8.x86_64 4/11 2023-01-24T14:29:19.998 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 5/11 2023-01-24T14:29:20.188 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 6/11 2023-01-24T14:29:20.366 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 7/11 2023-01-24T14:29:20.461 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 7/11 2023-01-24T14:29:20.654 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : jq-1.6-6.el8.x86_64 8/11 2023-01-24T14:29:20.825 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libasan6-11.2.1-9.1.el8.x86_64 9/11 2023-01-24T14:29:20.882 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: libasan6-11.2.1-9.1.el8.x86_64 9/11 2023-01-24T14:29:21.067 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libcephsqlite-2:18.0.0-1954.g1a671c19.el8.x86_64 10/11 2023-01-24T14:29:45.645 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: libcephsqlite-2:18.0.0-1954.g1a671c19.el8.x86_64 10/11 2023-01-24T14:29:45.903 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-test-2:18.0.0-1954.g1a671c19.el8.x86_64 11/11 2023-01-24T14:29:46.209 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-test-2:18.0.0-1954.g1a671c19.el8.x86_64 11/11 2023-01-24T14:29:46.209 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-test-2:18.0.0-1954.g1a671c19.el8.x86_64 1/11 2023-01-24T14:29:46.209 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libcephsqlite-2:18.0.0-1954.g1a671c19.el8.x86_64 2/11 2023-01-24T14:29:46.209 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : jq-1.6-6.el8.x86_64 3/11 2023-01-24T14:29:46.210 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libasan6-11.2.1-9.1.el8.x86_64 4/11 2023-01-24T14:29:46.210 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 5/11 2023-01-24T14:29:46.210 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 6/11 2023-01-24T14:29:46.210 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 7/11 2023-01-24T14:29:46.210 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libatomic-8.5.0-18.el8.x86_64 8/11 2023-01-24T14:29:46.211 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libubsan-8.5.0-18.el8.x86_64 9/11 2023-01-24T14:29:46.211 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : cryptopp-8.6.0-1.el8.x86_64 10/11 2023-01-24T14:29:46.934 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : yaml-cpp-0.6.2-3.el8.x86_64 11/11 2023-01-24T14:29:46.935 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-24T14:29:46.935 INFO:teuthology.orchestra.run.smithi027.stdout:Installed: 2023-01-24T14:29:46.935 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-test-2:18.0.0-1954.g1a671c19.el8.x86_64 2023-01-24T14:29:46.935 INFO:teuthology.orchestra.run.smithi027.stdout: cryptopp-8.6.0-1.el8.x86_64 2023-01-24T14:29:46.936 INFO:teuthology.orchestra.run.smithi027.stdout: jq-1.6-6.el8.x86_64 2023-01-24T14:29:46.936 INFO:teuthology.orchestra.run.smithi027.stdout: libasan6-11.2.1-9.1.el8.x86_64 2023-01-24T14:29:46.936 INFO:teuthology.orchestra.run.smithi027.stdout: libatomic-8.5.0-18.el8.x86_64 2023-01-24T14:29:46.936 INFO:teuthology.orchestra.run.smithi027.stdout: libcephsqlite-2:18.0.0-1954.g1a671c19.el8.x86_64 2023-01-24T14:29:46.936 INFO:teuthology.orchestra.run.smithi027.stdout: libubsan-8.5.0-18.el8.x86_64 2023-01-24T14:29:46.937 INFO:teuthology.orchestra.run.smithi027.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-01-24T14:29:46.937 INFO:teuthology.orchestra.run.smithi027.stdout: socat-1.7.4.1-1.el8.x86_64 2023-01-24T14:29:46.937 INFO:teuthology.orchestra.run.smithi027.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-01-24T14:29:46.937 INFO:teuthology.orchestra.run.smithi027.stdout: yaml-cpp-0.6.2-3.el8.x86_64 2023-01-24T14:29:46.937 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-24T14:29:46.937 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2023-01-24T14:29:47.163 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install ceph 2023-01-24T14:29:47.921 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:01:39 ago on Tue 24 Jan 2023 02:28:08 PM UTC. 2023-01-24T14:29:49.147 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2023-01-24T14:29:49.151 INFO:teuthology.orchestra.run.smithi027.stdout:======================================================================================= 2023-01-24T14:29:49.151 INFO:teuthology.orchestra.run.smithi027.stdout: Package Arch Version Repository Size 2023-01-24T14:29:49.151 INFO:teuthology.orchestra.run.smithi027.stdout:======================================================================================= 2023-01-24T14:29:49.151 INFO:teuthology.orchestra.run.smithi027.stdout:Installing: 2023-01-24T14:29:49.151 INFO:teuthology.orchestra.run.smithi027.stdout: ceph x86_64 2:18.0.0-1954.g1a671c19.el8 ceph 6.4 k 2023-01-24T14:29:49.152 INFO:teuthology.orchestra.run.smithi027.stdout:Installing dependencies: 2023-01-24T14:29:49.152 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mds x86_64 2:18.0.0-1954.g1a671c19.el8 ceph 2.2 M 2023-01-24T14:29:49.152 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr x86_64 2:18.0.0-1954.g1a671c19.el8 ceph 1.5 M 2023-01-24T14:29:49.152 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-modules-core noarch 2:18.0.0-1954.g1a671c19.el8 ceph-noarch 236 k 2023-01-24T14:29:49.152 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mon x86_64 2:18.0.0-1954.g1a671c19.el8 ceph 7.4 M 2023-01-24T14:29:49.153 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-osd x86_64 2:18.0.0-1954.g1a671c19.el8 ceph 151 M 2023-01-24T14:29:49.153 INFO:teuthology.orchestra.run.smithi027.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-01-24T14:29:49.153 INFO:teuthology.orchestra.run.smithi027.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-01-24T14:29:49.153 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cffi x86_64 1.11.5-5.el8 CentOS-BaseOS 237 k 2023-01-24T14:29:49.153 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-01-24T14:29:49.154 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-01-24T14:29:49.154 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cryptography x86_64 3.2.1-5.el8 CentOS-BaseOS 559 k 2023-01-24T14:29:49.154 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-01-24T14:29:49.154 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-01-24T14:29:49.154 INFO:teuthology.orchestra.run.smithi027.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-01-24T14:29:49.155 INFO:teuthology.orchestra.run.smithi027.stdout: python3-mako noarch 1.0.6-13.el8 CentOS-AppStream 157 k 2023-01-24T14:29:49.155 INFO:teuthology.orchestra.run.smithi027.stdout: python3-markupsafe x86_64 0.23-19.el8 CentOS-AppStream 39 k 2023-01-24T14:29:49.155 INFO:teuthology.orchestra.run.smithi027.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-01-24T14:29:49.155 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-01-24T14:29:49.155 INFO:teuthology.orchestra.run.smithi027.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-01-24T14:29:49.156 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 CentOS-AppStream 103 k 2023-01-24T14:29:49.156 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pycparser noarch 2.14-14.el8 CentOS-BaseOS 109 k 2023-01-24T14:29:49.156 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pytz noarch 2017.2-9.el8 CentOS-AppStream 54 k 2023-01-24T14:29:49.156 INFO:teuthology.orchestra.run.smithi027.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-01-24T14:29:49.156 INFO:teuthology.orchestra.run.smithi027.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-01-24T14:29:49.156 INFO:teuthology.orchestra.run.smithi027.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-01-24T14:29:49.157 INFO:teuthology.orchestra.run.smithi027.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-01-24T14:29:49.157 INFO:teuthology.orchestra.run.smithi027.stdout: python3-waitress noarch 1.2.1-2.el8.1 epel 241 k 2023-01-24T14:29:49.157 INFO:teuthology.orchestra.run.smithi027.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-01-24T14:29:49.157 INFO:teuthology.orchestra.run.smithi027.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-01-24T14:29:49.157 INFO:teuthology.orchestra.run.smithi027.stdout: python3-werkzeug noarch 0.12.2-4.el8 CentOS-AppStream 457 k 2023-01-24T14:29:49.158 INFO:teuthology.orchestra.run.smithi027.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-01-24T14:29:49.158 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-24T14:29:49.158 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2023-01-24T14:29:49.158 INFO:teuthology.orchestra.run.smithi027.stdout:======================================================================================= 2023-01-24T14:29:49.159 INFO:teuthology.orchestra.run.smithi027.stdout:Install 32 Packages 2023-01-24T14:29:49.159 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-24T14:29:49.160 INFO:teuthology.orchestra.run.smithi027.stdout:Total download size: 165 M 2023-01-24T14:29:49.160 INFO:teuthology.orchestra.run.smithi027.stdout:Installed size: 2.2 G 2023-01-24T14:29:49.160 INFO:teuthology.orchestra.run.smithi027.stdout:Downloading Packages: 2023-01-24T14:29:49.466 INFO:teuthology.orchestra.run.smithi027.stdout:(1/32): ceph-18.0.0-1954.g1a671c19.el8.x86_64.r 21 kB/s | 6.4 kB 00:00 2023-01-24T14:29:50.650 INFO:teuthology.orchestra.run.smithi027.stdout:(2/32): ceph-mgr-18.0.0-1954.g1a671c19.el8.x86_ 1.0 MB/s | 1.5 MB 00:01 2023-01-24T14:29:51.100 INFO:teuthology.orchestra.run.smithi027.stdout:(3/32): ceph-mds-18.0.0-1954.g1a671c19.el8.x86_ 1.1 MB/s | 2.2 MB 00:01 2023-01-24T14:29:51.259 INFO:teuthology.orchestra.run.smithi027.stdout:(4/32): ceph-mgr-modules-core-18.0.0-1954.g1a67 1.5 MB/s | 236 kB 00:00 2023-01-24T14:29:51.318 INFO:teuthology.orchestra.run.smithi027.stdout:(5/32): python3-mako-1.0.6-13.el8.noarch.rpm 2.6 MB/s | 157 kB 00:00 2023-01-24T14:29:51.343 INFO:teuthology.orchestra.run.smithi027.stdout:(6/32): python3-markupsafe-0.23-19.el8.x86_64.r 1.5 MB/s | 39 kB 00:00 2023-01-24T14:29:51.368 INFO:teuthology.orchestra.run.smithi027.stdout:(7/32): python3-pyOpenSSL-19.0.0-1.el8.noarch.r 4.1 MB/s | 103 kB 00:00 2023-01-24T14:29:51.394 INFO:teuthology.orchestra.run.smithi027.stdout:(8/32): python3-pytz-2017.2-9.el8.noarch.rpm 2.1 MB/s | 54 kB 00:00 2023-01-24T14:29:51.456 INFO:teuthology.orchestra.run.smithi027.stdout:(9/32): python3-werkzeug-0.12.2-4.el8.noarch.rp 7.2 MB/s | 457 kB 00:00 2023-01-24T14:29:51.486 INFO:teuthology.orchestra.run.smithi027.stdout:(10/32): python3-cffi-1.11.5-5.el8.x86_64.rpm 7.9 MB/s | 237 kB 00:00 2023-01-24T14:29:51.528 INFO:teuthology.orchestra.run.smithi027.stdout:(11/32): python3-cryptography-3.2.1-5.el8.x86_6 13 MB/s | 559 kB 00:00 2023-01-24T14:29:51.553 INFO:teuthology.orchestra.run.smithi027.stdout:(12/32): python3-pycparser-2.14-14.el8.noarch.r 4.3 MB/s | 109 kB 00:00 2023-01-24T14:29:51.720 INFO:teuthology.orchestra.run.smithi027.stdout:(13/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 262 kB/s | 44 kB 00:00 2023-01-24T14:29:51.904 INFO:teuthology.orchestra.run.smithi027.stdout:(14/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 1.0 MB/s | 185 kB 00:00 2023-01-24T14:29:52.038 INFO:teuthology.orchestra.run.smithi027.stdout:(15/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 1.3 MB/s | 173 kB 00:00 2023-01-24T14:29:52.288 INFO:teuthology.orchestra.run.smithi027.stdout:(16/32): python3-cherrypy-18.4.0-1.el8.noarch.r 1.5 MB/s | 384 kB 00:00 2023-01-24T14:29:52.338 INFO:teuthology.orchestra.run.smithi027.stdout:(17/32): python3-jaraco-6.2-6.el8.noarch.rpm 222 kB/s | 11 kB 00:00 2023-01-24T14:29:52.389 INFO:teuthology.orchestra.run.smithi027.stdout:(18/32): python3-jaraco-functools-2.0-4.el8.noa 352 kB/s | 18 kB 00:00 2023-01-24T14:29:52.439 INFO:teuthology.orchestra.run.smithi027.stdout:(19/32): python3-logutils-0.3.5-11.el8.noarch.r 977 kB/s | 49 kB 00:00 2023-01-24T14:29:52.506 INFO:teuthology.orchestra.run.smithi027.stdout:(20/32): python3-more-itertools-7.2.0-3.el8.noa 879 kB/s | 59 kB 00:00 2023-01-24T14:29:52.698 INFO:teuthology.orchestra.run.smithi027.stdout:(21/32): python3-pecan-1.3.2-9.el8.noarch.rpm 1.4 MB/s | 283 kB 00:00 2023-01-24T14:29:52.748 INFO:teuthology.orchestra.run.smithi027.stdout:(22/32): python3-portend-2.6-1.el8.noarch.rpm 312 kB/s | 16 kB 00:00 2023-01-24T14:29:52.799 INFO:teuthology.orchestra.run.smithi027.stdout:(23/32): python3-simplegeneric-0.8.1-17.el8.noa 374 kB/s | 19 kB 00:00 2023-01-24T14:29:52.840 INFO:teuthology.orchestra.run.smithi027.stdout:(24/32): python3-singledispatch-3.4.0.3-18.el8. 576 kB/s | 24 kB 00:00 2023-01-24T14:29:52.882 INFO:teuthology.orchestra.run.smithi027.stdout:(25/32): python3-tempora-1.14.1-5.el8.noarch.rp 703 kB/s | 29 kB 00:00 2023-01-24T14:29:52.928 INFO:teuthology.orchestra.run.smithi027.stdout:(26/32): python3-trustme-0.6.0-4.el8.noarch.rpm 661 kB/s | 27 kB 00:00 2023-01-24T14:29:53.100 INFO:teuthology.orchestra.run.smithi027.stdout:(27/32): python3-waitress-1.2.1-2.el8.1.noarch. 1.3 MB/s | 241 kB 00:00 2023-01-24T14:29:53.225 INFO:teuthology.orchestra.run.smithi027.stdout:(28/32): ceph-mon-18.0.0-1954.g1a671c19.el8.x86 2.0 MB/s | 7.4 MB 00:03 2023-01-24T14:29:53.276 INFO:teuthology.orchestra.run.smithi027.stdout:(29/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 1.4 MB/s | 251 kB 00:00 2023-01-24T14:29:53.326 INFO:teuthology.orchestra.run.smithi027.stdout:(30/32): python3-zc-lockfile-2.0-2.el8.noarch.r 460 kB/s | 23 kB 00:00 2023-01-24T14:29:53.401 INFO:teuthology.orchestra.run.smithi027.stdout:(31/32): python3-webtest-2.0.33-1.el8.noarch.rp 487 kB/s | 85 kB 00:00 2023-01-24T14:34:09.312 INFO:teuthology.orchestra.run.smithi035.stdout:[MIRROR] ceph-test-18.0.0-1954.g1a671c19.el8.x86_64.rpm: Curl error (28): Timeout was reached for https://1.chacra.ceph.com/r/ceph/main/1a671c198559f54389cd5afa22085a0c7b29a1ef/centos/8/flavors/crimson/x86_64/ceph-test-18.0.0-1954.g1a671c19.el8.x86_64.rpm [Operation too slow. Less than 1000 bytes/sec transferred the last 300 seconds] 2023-01-24T14:35:11.668 INFO:teuthology.orchestra.run.smithi027.stdout:[MIRROR] ceph-osd-18.0.0-1954.g1a671c19.el8.x86_64.rpm: Curl error (28): Timeout was reached for https://1.chacra.ceph.com/r/ceph/main/1a671c198559f54389cd5afa22085a0c7b29a1ef/centos/8/flavors/crimson/x86_64/ceph-osd-18.0.0-1954.g1a671c19.el8.x86_64.rpm [Operation too slow. Less than 1000 bytes/sec transferred the last 300 seconds] 2023-01-24T14:39:21.007 INFO:teuthology.orchestra.run.smithi027.stdout:(32/32): ceph-osd-18.0.0-1954.g1a671c19.el8.x86 270 kB/s | 151 MB 09:30 2023-01-24T14:39:21.008 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2023-01-24T14:39:21.009 INFO:teuthology.orchestra.run.smithi027.stdout:Total 296 kB/s | 165 MB 09:31 2023-01-24T14:39:21.085 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction check 2023-01-24T14:39:21.127 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction check succeeded. 2023-01-24T14:39:21.128 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction test 2023-01-24T14:39:22.049 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction test succeeded. 2023-01-24T14:39:22.053 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction 2023-01-24T14:39:23.878 INFO:teuthology.orchestra.run.smithi027.stdout: Preparing : 1/1 2023-01-24T14:39:24.174 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2023-01-24T14:39:24.389 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2023-01-24T14:39:24.657 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2023-01-24T14:39:24.888 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-waitress-1.2.1-2.el8.1.noarch 4/32 2023-01-24T14:39:25.102 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2023-01-24T14:39:25.292 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2023-01-24T14:39:25.501 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2023-01-24T14:39:25.549 INFO:teuthology.orchestra.run.smithi035.stdout:(11/11): ceph-test-18.0.0-1954.g1a671c19.el8.x8 244 kB/s | 149 MB 10:23 2023-01-24T14:39:25.549 INFO:teuthology.orchestra.run.smithi035.stdout:-------------------------------------------------------------------------------- 2023-01-24T14:39:25.550 INFO:teuthology.orchestra.run.smithi035.stdout:Total 249 kB/s | 152 MB 10:23 2023-01-24T14:39:25.612 INFO:teuthology.orchestra.run.smithi035.stdout:Running transaction check 2023-01-24T14:39:25.637 INFO:teuthology.orchestra.run.smithi035.stdout:Transaction check succeeded. 2023-01-24T14:39:25.638 INFO:teuthology.orchestra.run.smithi035.stdout:Running transaction test 2023-01-24T14:39:25.829 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2023-01-24T14:39:26.042 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2023-01-24T14:39:26.362 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2023-01-24T14:39:26.370 INFO:teuthology.orchestra.run.smithi035.stdout:Transaction test succeeded. 2023-01-24T14:39:26.374 INFO:teuthology.orchestra.run.smithi035.stdout:Running transaction 2023-01-24T14:39:26.622 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2023-01-24T14:39:26.828 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2023-01-24T14:39:27.228 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 13/32 2023-01-24T14:39:27.435 INFO:teuthology.orchestra.run.smithi035.stdout: Preparing : 1/1 2023-01-24T14:39:27.502 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-cryptography-3.2.1-5.el8.x86_64 14/32 2023-01-24T14:39:27.739 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2023-01-24T14:39:27.758 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : yaml-cpp-0.6.2-3.el8.x86_64 1/11 2023-01-24T14:39:27.970 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : cryptopp-8.6.0-1.el8.x86_64 2/11 2023-01-24T14:39:27.976 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2023-01-24T14:39:28.107 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : libubsan-8.5.0-18.el8.x86_64 3/11 2023-01-24T14:39:28.180 INFO:teuthology.orchestra.run.smithi035.stdout: Running scriptlet: libubsan-8.5.0-18.el8.x86_64 3/11 2023-01-24T14:39:28.232 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2023-01-24T14:39:28.299 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : libatomic-8.5.0-18.el8.x86_64 4/11 2023-01-24T14:39:28.380 INFO:teuthology.orchestra.run.smithi035.stdout: Running scriptlet: libatomic-8.5.0-18.el8.x86_64 4/11 2023-01-24T14:39:28.531 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2023-01-24T14:39:28.605 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 5/11 2023-01-24T14:39:28.754 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2023-01-24T14:39:28.820 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 6/11 2023-01-24T14:39:28.950 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 20/32 2023-01-24T14:39:29.140 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 7/11 2023-01-24T14:39:29.152 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2023-01-24T14:39:29.216 INFO:teuthology.orchestra.run.smithi035.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 7/11 2023-01-24T14:39:29.444 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : jq-1.6-6.el8.x86_64 8/11 2023-01-24T14:39:29.452 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2023-01-24T14:39:29.740 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : libasan6-11.2.1-9.1.el8.x86_64 9/11 2023-01-24T14:39:29.744 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2023-01-24T14:39:29.812 INFO:teuthology.orchestra.run.smithi035.stdout: Running scriptlet: libasan6-11.2.1-9.1.el8.x86_64 9/11 2023-01-24T14:39:29.980 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2023-01-24T14:39:29.982 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : libcephsqlite-2:18.0.0-1954.g1a671c19.el8.x86_64 10/11 2023-01-24T14:39:30.347 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 25/32 2023-01-24T14:39:30.742 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2023-01-24T14:39:31.126 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-1954.g1a671c19.el8. 27/32 2023-01-24T14:39:31.381 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-mgr-2:18.0.0-1954.g1a671c19.el8.x86_64 28/32 2023-01-24T14:39:54.416 INFO:teuthology.orchestra.run.smithi035.stdout: Running scriptlet: libcephsqlite-2:18.0.0-1954.g1a671c19.el8.x86_64 10/11 2023-01-24T14:39:54.702 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : ceph-test-2:18.0.0-1954.g1a671c19.el8.x86_64 11/11 2023-01-24T14:39:55.011 INFO:teuthology.orchestra.run.smithi035.stdout: Running scriptlet: ceph-test-2:18.0.0-1954.g1a671c19.el8.x86_64 11/11 2023-01-24T14:39:55.011 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : ceph-test-2:18.0.0-1954.g1a671c19.el8.x86_64 1/11 2023-01-24T14:39:55.011 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : libcephsqlite-2:18.0.0-1954.g1a671c19.el8.x86_64 2/11 2023-01-24T14:39:55.011 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : jq-1.6-6.el8.x86_64 3/11 2023-01-24T14:39:55.012 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : libasan6-11.2.1-9.1.el8.x86_64 4/11 2023-01-24T14:39:55.012 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 5/11 2023-01-24T14:39:55.012 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 6/11 2023-01-24T14:39:55.012 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 7/11 2023-01-24T14:39:55.012 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : libatomic-8.5.0-18.el8.x86_64 8/11 2023-01-24T14:39:55.013 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : libubsan-8.5.0-18.el8.x86_64 9/11 2023-01-24T14:39:55.013 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : cryptopp-8.6.0-1.el8.x86_64 10/11 2023-01-24T14:39:55.682 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : yaml-cpp-0.6.2-3.el8.x86_64 11/11 2023-01-24T14:39:55.683 INFO:teuthology.orchestra.run.smithi035.stdout: 2023-01-24T14:39:55.683 INFO:teuthology.orchestra.run.smithi035.stdout:Installed: 2023-01-24T14:39:55.683 INFO:teuthology.orchestra.run.smithi035.stdout: ceph-test-2:18.0.0-1954.g1a671c19.el8.x86_64 2023-01-24T14:39:55.683 INFO:teuthology.orchestra.run.smithi035.stdout: cryptopp-8.6.0-1.el8.x86_64 2023-01-24T14:39:55.684 INFO:teuthology.orchestra.run.smithi035.stdout: jq-1.6-6.el8.x86_64 2023-01-24T14:39:55.684 INFO:teuthology.orchestra.run.smithi035.stdout: libasan6-11.2.1-9.1.el8.x86_64 2023-01-24T14:39:55.684 INFO:teuthology.orchestra.run.smithi035.stdout: libatomic-8.5.0-18.el8.x86_64 2023-01-24T14:39:55.684 INFO:teuthology.orchestra.run.smithi035.stdout: libcephsqlite-2:18.0.0-1954.g1a671c19.el8.x86_64 2023-01-24T14:39:55.684 INFO:teuthology.orchestra.run.smithi035.stdout: libubsan-8.5.0-18.el8.x86_64 2023-01-24T14:39:55.685 INFO:teuthology.orchestra.run.smithi035.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-01-24T14:39:55.685 INFO:teuthology.orchestra.run.smithi035.stdout: socat-1.7.4.1-1.el8.x86_64 2023-01-24T14:39:55.685 INFO:teuthology.orchestra.run.smithi035.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-01-24T14:39:55.685 INFO:teuthology.orchestra.run.smithi035.stdout: yaml-cpp-0.6.2-3.el8.x86_64 2023-01-24T14:39:55.685 INFO:teuthology.orchestra.run.smithi035.stdout: 2023-01-24T14:39:55.686 INFO:teuthology.orchestra.run.smithi035.stdout:Complete! 2023-01-24T14:39:55.925 DEBUG:teuthology.orchestra.run.smithi035:> sudo yum -y install ceph 2023-01-24T14:39:56.692 INFO:teuthology.orchestra.run.smithi035.stdout:Last metadata expiration check: 0:11:47 ago on Tue 24 Jan 2023 02:28:09 PM UTC. 2023-01-24T14:39:57.923 INFO:teuthology.orchestra.run.smithi035.stdout:Dependencies resolved. 2023-01-24T14:39:57.926 INFO:teuthology.orchestra.run.smithi035.stdout:======================================================================================= 2023-01-24T14:39:57.926 INFO:teuthology.orchestra.run.smithi035.stdout: Package Arch Version Repository Size 2023-01-24T14:39:57.927 INFO:teuthology.orchestra.run.smithi035.stdout:======================================================================================= 2023-01-24T14:39:57.927 INFO:teuthology.orchestra.run.smithi035.stdout:Installing: 2023-01-24T14:39:57.927 INFO:teuthology.orchestra.run.smithi035.stdout: ceph x86_64 2:18.0.0-1954.g1a671c19.el8 ceph 6.4 k 2023-01-24T14:39:57.927 INFO:teuthology.orchestra.run.smithi035.stdout:Installing dependencies: 2023-01-24T14:39:57.927 INFO:teuthology.orchestra.run.smithi035.stdout: ceph-mds x86_64 2:18.0.0-1954.g1a671c19.el8 ceph 2.2 M 2023-01-24T14:39:57.928 INFO:teuthology.orchestra.run.smithi035.stdout: ceph-mgr x86_64 2:18.0.0-1954.g1a671c19.el8 ceph 1.5 M 2023-01-24T14:39:57.928 INFO:teuthology.orchestra.run.smithi035.stdout: ceph-mgr-modules-core noarch 2:18.0.0-1954.g1a671c19.el8 ceph-noarch 236 k 2023-01-24T14:39:57.928 INFO:teuthology.orchestra.run.smithi035.stdout: ceph-mon x86_64 2:18.0.0-1954.g1a671c19.el8 ceph 7.4 M 2023-01-24T14:39:57.928 INFO:teuthology.orchestra.run.smithi035.stdout: ceph-osd x86_64 2:18.0.0-1954.g1a671c19.el8 ceph 151 M 2023-01-24T14:39:57.928 INFO:teuthology.orchestra.run.smithi035.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-01-24T14:39:57.929 INFO:teuthology.orchestra.run.smithi035.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-01-24T14:39:57.929 INFO:teuthology.orchestra.run.smithi035.stdout: python3-cffi x86_64 1.11.5-5.el8 CentOS-BaseOS 237 k 2023-01-24T14:39:57.929 INFO:teuthology.orchestra.run.smithi035.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-01-24T14:39:57.929 INFO:teuthology.orchestra.run.smithi035.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-01-24T14:39:57.929 INFO:teuthology.orchestra.run.smithi035.stdout: python3-cryptography x86_64 3.2.1-5.el8 CentOS-BaseOS 559 k 2023-01-24T14:39:57.929 INFO:teuthology.orchestra.run.smithi035.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-01-24T14:39:57.930 INFO:teuthology.orchestra.run.smithi035.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-01-24T14:39:57.930 INFO:teuthology.orchestra.run.smithi035.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-01-24T14:39:57.930 INFO:teuthology.orchestra.run.smithi035.stdout: python3-mako noarch 1.0.6-13.el8 CentOS-AppStream 157 k 2023-01-24T14:39:57.930 INFO:teuthology.orchestra.run.smithi035.stdout: python3-markupsafe x86_64 0.23-19.el8 CentOS-AppStream 39 k 2023-01-24T14:39:57.930 INFO:teuthology.orchestra.run.smithi035.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-01-24T14:39:57.931 INFO:teuthology.orchestra.run.smithi035.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-01-24T14:39:57.931 INFO:teuthology.orchestra.run.smithi035.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-01-24T14:39:57.931 INFO:teuthology.orchestra.run.smithi035.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 CentOS-AppStream 103 k 2023-01-24T14:39:57.931 INFO:teuthology.orchestra.run.smithi035.stdout: python3-pycparser noarch 2.14-14.el8 CentOS-BaseOS 109 k 2023-01-24T14:39:57.931 INFO:teuthology.orchestra.run.smithi035.stdout: python3-pytz noarch 2017.2-9.el8 CentOS-AppStream 54 k 2023-01-24T14:39:57.932 INFO:teuthology.orchestra.run.smithi035.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-01-24T14:39:57.932 INFO:teuthology.orchestra.run.smithi035.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-01-24T14:39:57.932 INFO:teuthology.orchestra.run.smithi035.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-01-24T14:39:57.932 INFO:teuthology.orchestra.run.smithi035.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-01-24T14:39:57.932 INFO:teuthology.orchestra.run.smithi035.stdout: python3-waitress noarch 1.2.1-2.el8.1 epel 241 k 2023-01-24T14:39:57.933 INFO:teuthology.orchestra.run.smithi035.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-01-24T14:39:57.933 INFO:teuthology.orchestra.run.smithi035.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-01-24T14:39:57.933 INFO:teuthology.orchestra.run.smithi035.stdout: python3-werkzeug noarch 0.12.2-4.el8 CentOS-AppStream 457 k 2023-01-24T14:39:57.933 INFO:teuthology.orchestra.run.smithi035.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-01-24T14:39:57.933 INFO:teuthology.orchestra.run.smithi035.stdout: 2023-01-24T14:39:57.933 INFO:teuthology.orchestra.run.smithi035.stdout:Transaction Summary 2023-01-24T14:39:57.934 INFO:teuthology.orchestra.run.smithi035.stdout:======================================================================================= 2023-01-24T14:39:57.934 INFO:teuthology.orchestra.run.smithi035.stdout:Install 32 Packages 2023-01-24T14:39:57.934 INFO:teuthology.orchestra.run.smithi035.stdout: 2023-01-24T14:39:57.935 INFO:teuthology.orchestra.run.smithi035.stdout:Total download size: 165 M 2023-01-24T14:39:57.935 INFO:teuthology.orchestra.run.smithi035.stdout:Installed size: 2.2 G 2023-01-24T14:39:57.935 INFO:teuthology.orchestra.run.smithi035.stdout:Downloading Packages: 2023-01-24T14:39:58.254 INFO:teuthology.orchestra.run.smithi035.stdout:(1/32): ceph-18.0.0-1954.g1a671c19.el8.x86_64.r 20 kB/s | 6.4 kB 00:00 2023-01-24T14:39:59.804 INFO:teuthology.orchestra.run.smithi035.stdout:(2/32): ceph-mgr-18.0.0-1954.g1a671c19.el8.x86_ 807 kB/s | 1.5 MB 00:01 2023-01-24T14:40:00.410 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-mgr-2:18.0.0-1954.g1a671c19.el8.x86_64 28/32 2023-01-24T14:40:00.627 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-osd-2:18.0.0-1954.g1a671c19.el8.x86_64 29/32 2023-01-24T14:40:00.780 INFO:teuthology.orchestra.run.smithi035.stdout:(3/32): ceph-mds-18.0.0-1954.g1a671c19.el8.x86_ 785 kB/s | 2.2 MB 00:02 2023-01-24T14:40:01.196 INFO:teuthology.orchestra.run.smithi035.stdout:(4/32): ceph-mgr-modules-core-18.0.0-1954.g1a67 566 kB/s | 236 kB 00:00 2023-01-24T14:40:01.427 INFO:teuthology.orchestra.run.smithi035.stdout:(5/32): python3-mako-1.0.6-13.el8.noarch.rpm 682 kB/s | 157 kB 00:00 2023-01-24T14:40:01.429 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-osd-2:18.0.0-1954.g1a671c19.el8.x86_64 29/32 2023-01-24T14:40:01.660 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-mon-2:18.0.0-1954.g1a671c19.el8.x86_64 30/32 2023-01-24T14:40:01.795 INFO:teuthology.orchestra.run.smithi035.stdout:(6/32): python3-markupsafe-0.23-19.el8.x86_64.r 105 kB/s | 39 kB 00:00 2023-01-24T14:40:01.890 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-mon-2:18.0.0-1954.g1a671c19.el8.x86_64 30/32 2023-01-24T14:40:02.076 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-mds-2:18.0.0-1954.g1a671c19.el8.x86_64 31/32 2023-01-24T14:40:02.094 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-mds-2:18.0.0-1954.g1a671c19.el8.x86_64 31/32 2023-01-24T14:40:02.221 INFO:teuthology.orchestra.run.smithi035.stdout:(7/32): python3-pyOpenSSL-19.0.0-1.el8.noarch.r 243 kB/s | 103 kB 00:00 2023-01-24T14:40:02.242 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-2:18.0.0-1954.g1a671c19.el8.x86_64 32/32 2023-01-24T14:40:02.490 INFO:teuthology.orchestra.run.smithi035.stdout:(8/32): python3-pytz-2017.2-9.el8.noarch.rpm 201 kB/s | 54 kB 00:00 2023-01-24T14:40:02.524 INFO:teuthology.orchestra.run.smithi035.stdout:(9/32): python3-werkzeug-0.12.2-4.el8.noarch.rp 13 MB/s | 457 kB 00:00 2023-01-24T14:40:02.541 INFO:teuthology.orchestra.run.smithi035.stdout:(10/32): python3-cffi-1.11.5-5.el8.x86_64.rpm 14 MB/s | 237 kB 00:00 2023-01-24T14:40:02.574 INFO:teuthology.orchestra.run.smithi035.stdout:(11/32): python3-cryptography-3.2.1-5.el8.x86_6 17 MB/s | 559 kB 00:00 2023-01-24T14:40:02.599 INFO:teuthology.orchestra.run.smithi035.stdout:(12/32): python3-pycparser-2.14-14.el8.noarch.r 4.3 MB/s | 109 kB 00:00 2023-01-24T14:40:02.677 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-2:18.0.0-1954.g1a671c19.el8.x86_64 32/32 2023-01-24T14:40:02.677 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-2:18.0.0-1954.g1a671c19.el8.x86_64 1/32 2023-01-24T14:40:02.677 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-mds-2:18.0.0-1954.g1a671c19.el8.x86_64 2/32 2023-01-24T14:40:02.677 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-mgr-2:18.0.0-1954.g1a671c19.el8.x86_64 3/32 2023-01-24T14:40:02.678 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-mon-2:18.0.0-1954.g1a671c19.el8.x86_64 4/32 2023-01-24T14:40:02.678 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-osd-2:18.0.0-1954.g1a671c19.el8.x86_64 5/32 2023-01-24T14:40:02.678 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-1954.g1a671c19.el8. 6/32 2023-01-24T14:40:02.678 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 7/32 2023-01-24T14:40:02.678 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2023-01-24T14:40:02.679 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2023-01-24T14:40:02.680 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 10/32 2023-01-24T14:40:02.680 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2023-01-24T14:40:02.680 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 12/32 2023-01-24T14:40:02.680 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-cryptography-3.2.1-5.el8.x86_64 13/32 2023-01-24T14:40:02.681 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2023-01-24T14:40:02.681 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2023-01-24T14:40:02.681 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2023-01-24T14:40:02.681 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2023-01-24T14:40:02.681 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2023-01-24T14:40:02.682 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2023-01-24T14:40:02.682 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2023-01-24T14:40:02.682 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2023-01-24T14:40:02.682 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2023-01-24T14:40:02.682 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2023-01-24T14:40:02.683 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2023-01-24T14:40:02.683 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2023-01-24T14:40:02.683 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2023-01-24T14:40:02.683 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2023-01-24T14:40:02.683 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2023-01-24T14:40:02.684 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-waitress-1.2.1-2.el8.1.noarch 29/32 2023-01-24T14:40:02.684 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2023-01-24T14:40:02.684 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2023-01-24T14:40:02.733 INFO:teuthology.orchestra.run.smithi035.stdout:(13/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 327 kB/s | 44 kB 00:00 2023-01-24T14:40:02.845 INFO:teuthology.orchestra.run.smithi035.stdout:(14/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 1.7 MB/s | 185 kB 00:00 2023-01-24T14:40:02.892 INFO:teuthology.orchestra.run.smithi035.stdout:(15/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 3.4 MB/s | 173 kB 00:00 2023-01-24T14:40:03.001 INFO:teuthology.orchestra.run.smithi035.stdout:(16/32): python3-cherrypy-18.4.0-1.el8.noarch.r 3.5 MB/s | 384 kB 00:00 2023-01-24T14:40:03.043 INFO:teuthology.orchestra.run.smithi035.stdout:(17/32): python3-jaraco-6.2-6.el8.noarch.rpm 267 kB/s | 11 kB 00:00 2023-01-24T14:40:03.084 INFO:teuthology.orchestra.run.smithi035.stdout:(18/32): python3-jaraco-functools-2.0-4.el8.noa 423 kB/s | 18 kB 00:00 2023-01-24T14:40:03.135 INFO:teuthology.orchestra.run.smithi035.stdout:(19/32): python3-logutils-0.3.5-11.el8.noarch.r 975 kB/s | 49 kB 00:00 2023-01-24T14:40:03.185 INFO:teuthology.orchestra.run.smithi035.stdout:(20/32): python3-more-itertools-7.2.0-3.el8.noa 1.2 MB/s | 59 kB 00:00 2023-01-24T14:40:03.269 INFO:teuthology.orchestra.run.smithi035.stdout:(21/32): python3-pecan-1.3.2-9.el8.noarch.rpm 3.3 MB/s | 283 kB 00:00 2023-01-24T14:40:03.319 INFO:teuthology.orchestra.run.smithi035.stdout:(22/32): python3-portend-2.6-1.el8.noarch.rpm 311 kB/s | 16 kB 00:00 2023-01-24T14:40:03.369 INFO:teuthology.orchestra.run.smithi035.stdout:(23/32): python3-simplegeneric-0.8.1-17.el8.noa 376 kB/s | 19 kB 00:00 2023-01-24T14:40:03.428 INFO:teuthology.orchestra.run.smithi035.stdout:(24/32): python3-singledispatch-3.4.0.3-18.el8. 409 kB/s | 24 kB 00:00 2023-01-24T14:40:03.478 INFO:teuthology.orchestra.run.smithi035.stdout:(25/32): python3-tempora-1.14.1-5.el8.noarch.rp 587 kB/s | 29 kB 00:00 2023-01-24T14:40:03.579 INFO:teuthology.orchestra.run.smithi035.stdout:(26/32): ceph-mon-18.0.0-1954.g1a671c19.el8.x86 1.4 MB/s | 7.4 MB 00:05 2023-01-24T14:40:03.604 INFO:teuthology.orchestra.run.smithi035.stdout:(27/32): python3-trustme-0.6.0-4.el8.noarch.rpm 220 kB/s | 27 kB 00:00 2023-01-24T14:40:03.715 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2023-01-24T14:40:03.716 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-24T14:40:03.716 INFO:teuthology.orchestra.run.smithi027.stdout:Installed: 2023-01-24T14:40:03.716 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-2:18.0.0-1954.g1a671c19.el8.x86_64 2023-01-24T14:40:03.716 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mds-2:18.0.0-1954.g1a671c19.el8.x86_64 2023-01-24T14:40:03.717 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-2:18.0.0-1954.g1a671c19.el8.x86_64 2023-01-24T14:40:03.717 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-modules-core-2:18.0.0-1954.g1a671c19.el8.noarch 2023-01-24T14:40:03.717 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mon-2:18.0.0-1954.g1a671c19.el8.x86_64 2023-01-24T14:40:03.717 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-osd-2:18.0.0-1954.g1a671c19.el8.x86_64 2023-01-24T14:40:03.718 INFO:teuthology.orchestra.run.smithi027.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-01-24T14:40:03.718 INFO:teuthology.orchestra.run.smithi027.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-01-24T14:40:03.718 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cffi-1.11.5-5.el8.x86_64 2023-01-24T14:40:03.718 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-01-24T14:40:03.718 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-01-24T14:40:03.719 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cryptography-3.2.1-5.el8.x86_64 2023-01-24T14:40:03.719 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jaraco-6.2-6.el8.noarch 2023-01-24T14:40:03.719 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-01-24T14:40:03.719 INFO:teuthology.orchestra.run.smithi027.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-01-24T14:40:03.719 INFO:teuthology.orchestra.run.smithi027.stdout: python3-mako-1.0.6-13.el8.noarch 2023-01-24T14:40:03.720 INFO:teuthology.orchestra.run.smithi027.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-01-24T14:40:03.720 INFO:teuthology.orchestra.run.smithi027.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-01-24T14:40:03.720 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-01-24T14:40:03.720 INFO:teuthology.orchestra.run.smithi027.stdout: python3-portend-2.6-1.el8.noarch 2023-01-24T14:40:03.721 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-01-24T14:40:03.721 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pycparser-2.14-14.el8.noarch 2023-01-24T14:40:03.721 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pytz-2017.2-9.el8.noarch 2023-01-24T14:40:03.721 INFO:teuthology.orchestra.run.smithi027.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-01-24T14:40:03.721 INFO:teuthology.orchestra.run.smithi027.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-01-24T14:40:03.722 INFO:teuthology.orchestra.run.smithi027.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-01-24T14:40:03.722 INFO:teuthology.orchestra.run.smithi027.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-01-24T14:40:03.722 INFO:teuthology.orchestra.run.smithi027.stdout: python3-waitress-1.2.1-2.el8.1.noarch 2023-01-24T14:40:03.722 INFO:teuthology.orchestra.run.smithi027.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-01-24T14:40:03.722 INFO:teuthology.orchestra.run.smithi027.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-01-24T14:40:03.723 INFO:teuthology.orchestra.run.smithi027.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-01-24T14:40:03.723 INFO:teuthology.orchestra.run.smithi027.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-01-24T14:40:03.723 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-24T14:40:03.723 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2023-01-24T14:40:03.835 INFO:teuthology.orchestra.run.smithi035.stdout:(28/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 1.1 MB/s | 251 kB 00:00 2023-01-24T14:40:03.913 INFO:teuthology.orchestra.run.smithi035.stdout:(29/32): python3-webtest-2.0.33-1.el8.noarch.rp 1.1 MB/s | 85 kB 00:00 2023-01-24T14:40:03.930 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install ceph-base 2023-01-24T14:40:03.947 INFO:teuthology.orchestra.run.smithi035.stdout:(30/32): python3-waitress-1.2.1-2.el8.1.noarch. 655 kB/s | 241 kB 00:00 2023-01-24T14:40:04.005 INFO:teuthology.orchestra.run.smithi035.stdout:(31/32): python3-zc-lockfile-2.0-2.el8.noarch.r 249 kB/s | 23 kB 00:00 2023-01-24T14:40:04.696 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:11:56 ago on Tue 24 Jan 2023 02:28:08 PM UTC. 2023-01-24T14:40:05.822 INFO:teuthology.orchestra.run.smithi027.stdout:Package ceph-base-2:18.0.0-1954.g1a671c19.el8.x86_64 is already installed. 2023-01-24T14:40:05.933 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2023-01-24T14:40:05.934 INFO:teuthology.orchestra.run.smithi027.stdout:Nothing to do. 2023-01-24T14:40:05.935 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2023-01-24T14:40:06.387 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install cephadm 2023-01-24T14:40:07.145 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:11:59 ago on Tue 24 Jan 2023 02:28:08 PM UTC. 2023-01-24T14:40:08.371 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2023-01-24T14:40:08.372 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2023-01-24T14:40:08.372 INFO:teuthology.orchestra.run.smithi027.stdout: Package Arch Version Repository Size 2023-01-24T14:40:08.372 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2023-01-24T14:40:08.373 INFO:teuthology.orchestra.run.smithi027.stdout:Installing: 2023-01-24T14:40:08.373 INFO:teuthology.orchestra.run.smithi027.stdout: cephadm noarch 2:18.0.0-1954.g1a671c19.el8 ceph-noarch 82 k 2023-01-24T14:40:08.373 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-24T14:40:08.373 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2023-01-24T14:40:08.373 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2023-01-24T14:40:08.374 INFO:teuthology.orchestra.run.smithi027.stdout:Install 1 Package 2023-01-24T14:40:08.374 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-24T14:40:08.374 INFO:teuthology.orchestra.run.smithi027.stdout:Total download size: 82 k 2023-01-24T14:40:08.374 INFO:teuthology.orchestra.run.smithi027.stdout:Installed size: 374 k 2023-01-24T14:40:08.374 INFO:teuthology.orchestra.run.smithi027.stdout:Downloading Packages: 2023-01-24T14:40:09.177 INFO:teuthology.orchestra.run.smithi027.stdout:cephadm-18.0.0-1954.g1a671c19.el8.noarch.rpm 102 kB/s | 82 kB 00:00 2023-01-24T14:40:09.177 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2023-01-24T14:40:09.178 INFO:teuthology.orchestra.run.smithi027.stdout:Total 102 kB/s | 82 kB 00:00 2023-01-24T14:40:09.178 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction check 2023-01-24T14:40:09.188 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction check succeeded. 2023-01-24T14:40:09.188 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction test 2023-01-24T14:40:09.191 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction test succeeded. 2023-01-24T14:40:09.195 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction 2023-01-24T14:40:09.760 INFO:teuthology.orchestra.run.smithi027.stdout: Preparing : 1/1 2023-01-24T14:40:19.200 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: cephadm-2:18.0.0-1954.g1a671c19.el8.noarch 1/1 2023-01-24T14:40:19.375 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : cephadm-2:18.0.0-1954.g1a671c19.el8.noarch 1/1 2023-01-24T14:40:19.636 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: cephadm-2:18.0.0-1954.g1a671c19.el8.noarch 1/1 2023-01-24T14:40:20.110 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : cephadm-2:18.0.0-1954.g1a671c19.el8.noarch 1/1 2023-01-24T14:40:20.110 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-24T14:40:20.110 INFO:teuthology.orchestra.run.smithi027.stdout:Installed: 2023-01-24T14:40:20.111 INFO:teuthology.orchestra.run.smithi027.stdout: cephadm-2:18.0.0-1954.g1a671c19.el8.noarch 2023-01-24T14:40:20.111 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-24T14:40:20.111 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2023-01-24T14:40:20.284 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install ceph-immutable-object-cache 2023-01-24T14:40:21.041 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:12:13 ago on Tue 24 Jan 2023 02:28:08 PM UTC. 2023-01-24T14:40:22.269 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2023-01-24T14:40:22.270 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2023-01-24T14:40:22.270 INFO:teuthology.orchestra.run.smithi027.stdout: Package Arch Version Repo Size 2023-01-24T14:40:22.270 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2023-01-24T14:40:22.270 INFO:teuthology.orchestra.run.smithi027.stdout:Installing: 2023-01-24T14:40:22.271 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-1954.g1a671c19.el8 ceph 157 k 2023-01-24T14:40:22.271 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-24T14:40:22.271 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2023-01-24T14:40:22.271 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2023-01-24T14:40:22.271 INFO:teuthology.orchestra.run.smithi027.stdout:Install 1 Package 2023-01-24T14:40:22.272 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-24T14:40:22.272 INFO:teuthology.orchestra.run.smithi027.stdout:Total download size: 157 k 2023-01-24T14:40:22.272 INFO:teuthology.orchestra.run.smithi027.stdout:Installed size: 540 k 2023-01-24T14:40:22.272 INFO:teuthology.orchestra.run.smithi027.stdout:Downloading Packages: 2023-01-24T14:40:22.598 INFO:teuthology.orchestra.run.smithi027.stdout:ceph-immutable-object-cache-18.0.0-1954.g1a671c 480 kB/s | 157 kB 00:00 2023-01-24T14:40:22.598 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2023-01-24T14:40:22.599 INFO:teuthology.orchestra.run.smithi027.stdout:Total 478 kB/s | 157 kB 00:00 2023-01-24T14:40:22.599 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction check 2023-01-24T14:40:22.610 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction check succeeded. 2023-01-24T14:40:22.610 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction test 2023-01-24T14:40:22.665 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction test succeeded. 2023-01-24T14:40:22.668 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction 2023-01-24T14:40:22.797 INFO:teuthology.orchestra.run.smithi027.stdout: Preparing : 1/1 2023-01-24T14:40:23.105 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-1954.g1a671c19. 1/1 2023-01-24T14:40:23.573 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-1954.g1a671c19. 1/1 2023-01-24T14:40:24.090 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-1954.g1a671c19. 1/1 2023-01-24T14:40:24.090 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-24T14:40:24.090 INFO:teuthology.orchestra.run.smithi027.stdout:Installed: 2023-01-24T14:40:24.090 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-immutable-object-cache-2:18.0.0-1954.g1a671c19.el8.x86_64 2023-01-24T14:40:24.091 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-24T14:40:24.091 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2023-01-24T14:40:24.299 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install ceph-mgr 2023-01-24T14:40:25.059 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:12:17 ago on Tue 24 Jan 2023 02:28:08 PM UTC. 2023-01-24T14:40:26.196 INFO:teuthology.orchestra.run.smithi027.stdout:Package ceph-mgr-2:18.0.0-1954.g1a671c19.el8.x86_64 is already installed. 2023-01-24T14:40:26.307 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2023-01-24T14:40:26.309 INFO:teuthology.orchestra.run.smithi027.stdout:Nothing to do. 2023-01-24T14:40:26.309 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2023-01-24T14:40:26.391 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install ceph-mgr-dashboard 2023-01-24T14:40:27.141 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:12:19 ago on Tue 24 Jan 2023 02:28:08 PM UTC. 2023-01-24T14:40:28.383 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2023-01-24T14:40:28.384 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2023-01-24T14:40:28.384 INFO:teuthology.orchestra.run.smithi027.stdout: Package Arch Version Repository Size 2023-01-24T14:40:28.385 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2023-01-24T14:40:28.385 INFO:teuthology.orchestra.run.smithi027.stdout:Installing: 2023-01-24T14:40:28.385 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-dashboard noarch 2:18.0.0-1954.g1a671c19.el8 ceph-noarch 1.7 M 2023-01-24T14:40:28.385 INFO:teuthology.orchestra.run.smithi027.stdout:Installing dependencies: 2023-01-24T14:40:28.385 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-grafana-dashboards noarch 2:18.0.0-1954.g1a671c19.el8 ceph-noarch 23 k 2023-01-24T14:40:28.386 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-prometheus-alerts noarch 2:18.0.0-1954.g1a671c19.el8 ceph-noarch 15 k 2023-01-24T14:40:28.386 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jwt noarch 1.6.1-2.el8 CentOS-BaseOS 43 k 2023-01-24T14:40:28.386 INFO:teuthology.orchestra.run.smithi027.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-01-24T14:40:28.386 INFO:teuthology.orchestra.run.smithi027.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-01-24T14:40:28.386 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-24T14:40:28.387 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2023-01-24T14:40:28.387 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2023-01-24T14:40:28.387 INFO:teuthology.orchestra.run.smithi027.stdout:Install 6 Packages 2023-01-24T14:40:28.387 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-24T14:40:28.388 INFO:teuthology.orchestra.run.smithi027.stdout:Total download size: 2.0 M 2023-01-24T14:40:28.388 INFO:teuthology.orchestra.run.smithi027.stdout:Installed size: 8.2 M 2023-01-24T14:40:28.388 INFO:teuthology.orchestra.run.smithi027.stdout:Downloading Packages: 2023-01-24T14:40:28.567 INFO:teuthology.orchestra.run.smithi027.stdout:(1/6): ceph-grafana-dashboards-18.0.0-1954.g1a6 128 kB/s | 23 kB 00:00 2023-01-24T14:40:28.592 INFO:teuthology.orchestra.run.smithi027.stdout:(2/6): ceph-prometheus-alerts-18.0.0-1954.g1a67 71 kB/s | 15 kB 00:00 2023-01-24T14:40:28.651 INFO:teuthology.orchestra.run.smithi027.stdout:(3/6): python3-jwt-1.6.1-2.el8.noarch.rpm 509 kB/s | 43 kB 00:00 2023-01-24T14:40:28.702 INFO:teuthology.orchestra.run.smithi027.stdout:(4/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 309 kB/s | 34 kB 00:00 2023-01-24T14:40:28.843 INFO:teuthology.orchestra.run.smithi027.stdout:(5/6): python3-routes-2.4.1-12.el8.noarch.rpm 1.0 MB/s | 196 kB 00:00 2023-01-24T14:40:28.986 INFO:teuthology.orchestra.run.smithi027.stdout:(6/6): ceph-mgr-dashboard-18.0.0-1954.g1a671c19 2.9 MB/s | 1.7 MB 00:00 2023-01-24T14:40:28.987 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2023-01-24T14:40:28.988 INFO:teuthology.orchestra.run.smithi027.stdout:Total 3.4 MB/s | 2.0 MB 00:00 2023-01-24T14:40:28.998 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction check 2023-01-24T14:40:29.008 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction check succeeded. 2023-01-24T14:40:29.009 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction test 2023-01-24T14:40:29.096 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction test succeeded. 2023-01-24T14:40:29.099 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction 2023-01-24T14:40:29.400 INFO:teuthology.orchestra.run.smithi027.stdout: Preparing : 1/1 2023-01-24T14:40:29.631 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2023-01-24T14:40:29.875 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2023-01-24T14:40:30.067 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2023-01-24T14:40:30.196 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-1954.g1a671c19.el8.n 4/6 2023-01-24T14:40:30.635 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-1954.g1a671c19.el8. 5/6 2023-01-24T14:40:30.898 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-1954.g1a671c19.el8.noarc 6/6 2023-01-24T14:40:31.217 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-1954.g1a671c19.el8.noarc 6/6 2023-01-24T14:40:31.217 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-1954.g1a671c19.el8. 1/6 2023-01-24T14:40:31.217 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-1954.g1a671c19.el8.noarc 2/6 2023-01-24T14:40:31.217 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-1954.g1a671c19.el8.n 3/6 2023-01-24T14:40:31.218 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2023-01-24T14:40:31.218 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2023-01-24T14:40:31.729 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2023-01-24T14:40:31.729 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-24T14:40:31.729 INFO:teuthology.orchestra.run.smithi027.stdout:Installed: 2023-01-24T14:40:31.729 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-grafana-dashboards-2:18.0.0-1954.g1a671c19.el8.noarch 2023-01-24T14:40:31.730 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-dashboard-2:18.0.0-1954.g1a671c19.el8.noarch 2023-01-24T14:40:31.730 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-prometheus-alerts-2:18.0.0-1954.g1a671c19.el8.noarch 2023-01-24T14:40:31.730 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-01-24T14:40:31.730 INFO:teuthology.orchestra.run.smithi027.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-01-24T14:40:31.730 INFO:teuthology.orchestra.run.smithi027.stdout: python3-routes-2.4.1-12.el8.noarch 2023-01-24T14:40:31.731 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-24T14:40:31.731 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2023-01-24T14:40:31.839 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install ceph-mgr-diskprediction-local 2023-01-24T14:40:32.592 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:12:24 ago on Tue 24 Jan 2023 02:28:08 PM UTC. 2023-01-24T14:40:33.815 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2023-01-24T14:40:33.816 INFO:teuthology.orchestra.run.smithi027.stdout:===================================================================================================== 2023-01-24T14:40:33.816 INFO:teuthology.orchestra.run.smithi027.stdout: Package Arch Version Repository Size 2023-01-24T14:40:33.817 INFO:teuthology.orchestra.run.smithi027.stdout:===================================================================================================== 2023-01-24T14:40:33.817 INFO:teuthology.orchestra.run.smithi027.stdout:Installing: 2023-01-24T14:40:33.817 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-1954.g1a671c19.el8 ceph-noarch 7.4 M 2023-01-24T14:40:33.817 INFO:teuthology.orchestra.run.smithi027.stdout:Installing dependencies: 2023-01-24T14:40:33.817 INFO:teuthology.orchestra.run.smithi027.stdout: libgfortran x86_64 8.5.0-18.el8 CentOS-BaseOS 645 k 2023-01-24T14:40:33.818 INFO:teuthology.orchestra.run.smithi027.stdout: libquadmath x86_64 8.5.0-18.el8 CentOS-BaseOS 172 k 2023-01-24T14:40:33.818 INFO:teuthology.orchestra.run.smithi027.stdout: openblas x86_64 0.3.15-4.el8 CentOS-AppStream 4.7 M 2023-01-24T14:40:33.818 INFO:teuthology.orchestra.run.smithi027.stdout: openblas-threads x86_64 0.3.15-4.el8 CentOS-AppStream 4.9 M 2023-01-24T14:40:33.818 INFO:teuthology.orchestra.run.smithi027.stdout: python3-numpy x86_64 1:1.14.3-10.el8 CentOS-AppStream 3.7 M 2023-01-24T14:40:33.818 INFO:teuthology.orchestra.run.smithi027.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 CentOS-AppStream 225 k 2023-01-24T14:40:33.819 INFO:teuthology.orchestra.run.smithi027.stdout: python3-scipy x86_64 1.0.0-21.module_el8.5.0+771+e5d9a225 CentOS-AppStream 14 M 2023-01-24T14:40:33.819 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-24T14:40:33.819 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2023-01-24T14:40:33.819 INFO:teuthology.orchestra.run.smithi027.stdout:===================================================================================================== 2023-01-24T14:40:33.819 INFO:teuthology.orchestra.run.smithi027.stdout:Install 8 Packages 2023-01-24T14:40:33.819 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-24T14:40:33.820 INFO:teuthology.orchestra.run.smithi027.stdout:Total download size: 35 M 2023-01-24T14:40:33.820 INFO:teuthology.orchestra.run.smithi027.stdout:Installed size: 210 M 2023-01-24T14:40:33.821 INFO:teuthology.orchestra.run.smithi027.stdout:Downloading Packages: 2023-01-24T14:40:34.008 INFO:teuthology.orchestra.run.smithi027.stdout:(1/8): openblas-0.3.15-4.el8.x86_64.rpm 25 MB/s | 4.7 MB 00:00 2023-01-24T14:40:34.100 INFO:teuthology.orchestra.run.smithi027.stdout:(2/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 17 MB/s | 4.9 MB 00:00 2023-01-24T14:40:34.134 INFO:teuthology.orchestra.run.smithi027.stdout:(3/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 6.6 MB/s | 225 kB 00:00 2023-01-24T14:40:34.201 INFO:teuthology.orchestra.run.smithi027.stdout:(4/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 19 MB/s | 3.7 MB 00:00 2023-01-24T14:40:34.243 INFO:teuthology.orchestra.run.smithi027.stdout:(5/8): libgfortran-8.5.0-18.el8.x86_64.rpm 15 MB/s | 645 kB 00:00 2023-01-24T14:40:34.276 INFO:teuthology.orchestra.run.smithi027.stdout:(6/8): libquadmath-8.5.0-18.el8.x86_64.rpm 5.1 MB/s | 172 kB 00:00 2023-01-24T14:40:34.735 INFO:teuthology.orchestra.run.smithi027.stdout:(7/8): python3-scipy-1.0.0-21.module_el8.5.0+77 23 MB/s | 14 MB 00:00 2023-01-24T14:40:36.511 INFO:teuthology.orchestra.run.smithi027.stdout:(8/8): ceph-mgr-diskprediction-local-18.0.0-195 2.7 MB/s | 7.4 MB 00:02 2023-01-24T14:40:36.513 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2023-01-24T14:40:36.514 INFO:teuthology.orchestra.run.smithi027.stdout:Total 13 MB/s | 35 MB 00:02 2023-01-24T14:40:36.756 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction check 2023-01-24T14:40:36.775 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction check succeeded. 2023-01-24T14:40:36.776 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction test 2023-01-24T14:40:37.199 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction test succeeded. 2023-01-24T14:40:37.203 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction 2023-01-24T14:40:37.808 INFO:teuthology.orchestra.run.smithi027.stdout: Preparing : 1/1 2023-01-24T14:40:37.988 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libquadmath-8.5.0-18.el8.x86_64 1/8 2023-01-24T14:40:38.112 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: libquadmath-8.5.0-18.el8.x86_64 1/8 2023-01-24T14:40:38.246 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libgfortran-8.5.0-18.el8.x86_64 2/8 2023-01-24T14:40:38.868 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: libgfortran-8.5.0-18.el8.x86_64 2/8 2023-01-24T14:40:39.046 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2023-01-24T14:40:39.700 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2023-01-24T14:40:39.838 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2023-01-24T14:40:40.973 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2023-01-24T14:40:41.371 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2023-01-24T14:40:41.638 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-01-24T14:40:42.120 INFO:teuthology.orchestra.run.smithi035.stdout:(32/32): ceph-osd-18.0.0-1954.g1a671c19.el8.x86 3.6 MB/s | 151 MB 00:42 2023-01-24T14:40:42.123 INFO:teuthology.orchestra.run.smithi035.stdout:-------------------------------------------------------------------------------- 2023-01-24T14:40:42.123 INFO:teuthology.orchestra.run.smithi035.stdout:Total 3.7 MB/s | 165 MB 00:44 2023-01-24T14:40:42.201 INFO:teuthology.orchestra.run.smithi035.stdout:Running transaction check 2023-01-24T14:40:42.243 INFO:teuthology.orchestra.run.smithi035.stdout:Transaction check succeeded. 2023-01-24T14:40:42.244 INFO:teuthology.orchestra.run.smithi035.stdout:Running transaction test 2023-01-24T14:40:43.140 INFO:teuthology.orchestra.run.smithi035.stdout:Transaction test succeeded. 2023-01-24T14:40:43.144 INFO:teuthology.orchestra.run.smithi035.stdout:Running transaction 2023-01-24T14:40:43.532 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-01-24T14:40:44.842 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 7/8 2023-01-24T14:40:44.936 INFO:teuthology.orchestra.run.smithi035.stdout: Preparing : 1/1 2023-01-24T14:40:45.079 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-1954.g1a671c1 8/8 2023-01-24T14:40:45.199 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2023-01-24T14:40:45.433 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-1954.g1a671c1 8/8 2023-01-24T14:40:45.434 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-1954.g1a671c1 1/8 2023-01-24T14:40:45.434 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2023-01-24T14:40:45.434 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2023-01-24T14:40:45.434 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2023-01-24T14:40:45.435 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2023-01-24T14:40:45.435 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 6/8 2023-01-24T14:40:45.435 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libgfortran-8.5.0-18.el8.x86_64 7/8 2023-01-24T14:40:45.464 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2023-01-24T14:40:45.708 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2023-01-24T14:40:45.954 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : python3-waitress-1.2.1-2.el8.1.noarch 4/32 2023-01-24T14:40:45.970 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libquadmath-8.5.0-18.el8.x86_64 8/8 2023-01-24T14:40:45.971 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-24T14:40:45.971 INFO:teuthology.orchestra.run.smithi027.stdout:Installed: 2023-01-24T14:40:45.971 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-diskprediction-local-2:18.0.0-1954.g1a671c19.el8.noarch 2023-01-24T14:40:45.971 INFO:teuthology.orchestra.run.smithi027.stdout: libgfortran-8.5.0-18.el8.x86_64 2023-01-24T14:40:45.972 INFO:teuthology.orchestra.run.smithi027.stdout: libquadmath-8.5.0-18.el8.x86_64 2023-01-24T14:40:45.972 INFO:teuthology.orchestra.run.smithi027.stdout: openblas-0.3.15-4.el8.x86_64 2023-01-24T14:40:45.972 INFO:teuthology.orchestra.run.smithi027.stdout: openblas-threads-0.3.15-4.el8.x86_64 2023-01-24T14:40:45.972 INFO:teuthology.orchestra.run.smithi027.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-01-24T14:40:45.972 INFO:teuthology.orchestra.run.smithi027.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-01-24T14:40:45.973 INFO:teuthology.orchestra.run.smithi027.stdout: python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x86_64 2023-01-24T14:40:45.973 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-24T14:40:45.973 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2023-01-24T14:40:46.093 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install ceph-mgr-rook 2023-01-24T14:40:46.118 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2023-01-24T14:40:46.326 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2023-01-24T14:40:46.517 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2023-01-24T14:40:46.704 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2023-01-24T14:40:46.853 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:12:38 ago on Tue 24 Jan 2023 02:28:08 PM UTC. 2023-01-24T14:40:46.916 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2023-01-24T14:40:47.227 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2023-01-24T14:40:47.457 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2023-01-24T14:40:47.692 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2023-01-24T14:40:48.083 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2023-01-24T14:40:48.084 INFO:teuthology.orchestra.run.smithi027.stdout:======================================================================================== 2023-01-24T14:40:48.085 INFO:teuthology.orchestra.run.smithi027.stdout: Package Arch Version Repository Size 2023-01-24T14:40:48.085 INFO:teuthology.orchestra.run.smithi027.stdout:======================================================================================== 2023-01-24T14:40:48.085 INFO:teuthology.orchestra.run.smithi027.stdout:Installing: 2023-01-24T14:40:48.085 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-rook noarch 2:18.0.0-1954.g1a671c19.el8 ceph-noarch 47 k 2023-01-24T14:40:48.086 INFO:teuthology.orchestra.run.smithi027.stdout:Installing dependencies: 2023-01-24T14:40:48.086 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-01-24T14:40:48.086 INFO:teuthology.orchestra.run.smithi027.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-01-24T14:40:48.086 INFO:teuthology.orchestra.run.smithi027.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-01-24T14:40:48.086 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jsonpatch noarch 1.21-2.el8 CentOS-AppStream 27 k 2023-01-24T14:40:48.086 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jsonpointer noarch 1.10-11.el8 CentOS-AppStream 20 k 2023-01-24T14:40:48.087 INFO:teuthology.orchestra.run.smithi027.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-01-24T14:40:48.087 INFO:teuthology.orchestra.run.smithi027.stdout: python3-oauthlib noarch 2.1.0-1.el8 CentOS-BaseOS 155 k 2023-01-24T14:40:48.087 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pyasn1 noarch 0.3.7-6.el8 CentOS-AppStream 126 k 2023-01-24T14:40:48.087 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 CentOS-AppStream 110 k 2023-01-24T14:40:48.087 INFO:teuthology.orchestra.run.smithi027.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 CentOS-BaseOS 43 k 2023-01-24T14:40:48.088 INFO:teuthology.orchestra.run.smithi027.stdout: python3-rsa noarch 4.8-1.el8 epel 61 k 2023-01-24T14:40:48.088 INFO:teuthology.orchestra.run.smithi027.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-01-24T14:40:48.088 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-24T14:40:48.088 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2023-01-24T14:40:48.088 INFO:teuthology.orchestra.run.smithi027.stdout:======================================================================================== 2023-01-24T14:40:48.089 INFO:teuthology.orchestra.run.smithi027.stdout:Install 13 Packages 2023-01-24T14:40:48.089 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-24T14:40:48.089 INFO:teuthology.orchestra.run.smithi027.stdout:Total download size: 1.9 M 2023-01-24T14:40:48.090 INFO:teuthology.orchestra.run.smithi027.stdout:Installed size: 25 M 2023-01-24T14:40:48.090 INFO:teuthology.orchestra.run.smithi027.stdout:Downloading Packages: 2023-01-24T14:40:48.133 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 13/32 2023-01-24T14:40:48.166 INFO:teuthology.orchestra.run.smithi027.stdout:(1/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 343 kB/s | 27 kB 00:00 2023-01-24T14:40:48.191 INFO:teuthology.orchestra.run.smithi027.stdout:(2/13): python3-jsonpointer-1.10-11.el8.noarch. 195 kB/s | 20 kB 00:00 2023-01-24T14:40:48.217 INFO:teuthology.orchestra.run.smithi027.stdout:(3/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 2.5 MB/s | 126 kB 00:00 2023-01-24T14:40:48.250 INFO:teuthology.orchestra.run.smithi027.stdout:(4/13): python3-pyasn1-modules-0.3.7-6.el8.noar 1.8 MB/s | 110 kB 00:00 2023-01-24T14:40:48.275 INFO:teuthology.orchestra.run.smithi027.stdout:(5/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 2.6 MB/s | 155 kB 00:00 2023-01-24T14:40:48.292 INFO:teuthology.orchestra.run.smithi027.stdout:(6/13): ceph-mgr-rook-18.0.0-1954.g1a671c19.el8 232 kB/s | 47 kB 00:00 2023-01-24T14:40:48.309 INFO:teuthology.orchestra.run.smithi027.stdout:(7/13): python3-requests-oauthlib-1.0.0-1.el8.n 721 kB/s | 43 kB 00:00 2023-01-24T14:40:48.401 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : python3-cryptography-3.2.1-5.el8.x86_64 14/32 2023-01-24T14:40:48.410 INFO:teuthology.orchestra.run.smithi027.stdout:(8/13): python3-cachetools-3.1.1-4.el8.noarch.r 246 kB/s | 33 kB 00:00 2023-01-24T14:40:48.427 INFO:teuthology.orchestra.run.smithi027.stdout:(9/13): python3-certifi-2018.10.15-7.el8.noarch 122 kB/s | 16 kB 00:00 2023-01-24T14:40:48.494 INFO:teuthology.orchestra.run.smithi027.stdout:(10/13): python3-rsa-4.8-1.el8.noarch.rpm 921 kB/s | 61 kB 00:00 2023-01-24T14:40:48.536 INFO:teuthology.orchestra.run.smithi027.stdout:(11/13): python3-google-auth-1.1.1-10.el8.noarc 360 kB/s | 81 kB 00:00 2023-01-24T14:40:48.597 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2023-01-24T14:40:48.611 INFO:teuthology.orchestra.run.smithi027.stdout:(12/13): python3-websocket-client-0.56.0-5.el8. 523 kB/s | 61 kB 00:00 2023-01-24T14:40:48.879 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2023-01-24T14:40:48.936 INFO:teuthology.orchestra.run.smithi027.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 2.2 MB/s | 1.2 MB 00:00 2023-01-24T14:40:48.938 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2023-01-24T14:40:48.938 INFO:teuthology.orchestra.run.smithi027.stdout:Total 2.3 MB/s | 1.9 MB 00:00 2023-01-24T14:40:48.995 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction check 2023-01-24T14:40:49.012 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction check succeeded. 2023-01-24T14:40:49.012 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction test 2023-01-24T14:40:49.099 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2023-01-24T14:40:49.116 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction test succeeded. 2023-01-24T14:40:49.119 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction 2023-01-24T14:40:49.405 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2023-01-24T14:40:49.637 INFO:teuthology.orchestra.run.smithi027.stdout: Preparing : 1/1 2023-01-24T14:40:49.654 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2023-01-24T14:40:49.858 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 20/32 2023-01-24T14:40:49.884 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2023-01-24T14:40:50.073 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2023-01-24T14:40:50.077 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2023-01-24T14:40:50.357 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-rsa-4.8-1.el8.noarch 3/13 2023-01-24T14:40:50.362 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2023-01-24T14:40:50.586 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2023-01-24T14:40:50.643 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2023-01-24T14:40:50.880 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2023-01-24T14:40:51.062 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2023-01-24T14:40:51.224 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 25/32 2023-01-24T14:40:51.339 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2023-01-24T14:40:51.616 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2023-01-24T14:40:51.752 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2023-01-24T14:40:51.994 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2023-01-24T14:40:52.031 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-1954.g1a671c19.el8. 27/32 2023-01-24T14:40:52.303 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : ceph-mgr-2:18.0.0-1954.g1a671c19.el8.x86_64 28/32 2023-01-24T14:40:53.132 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2023-01-24T14:40:53.468 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2023-01-24T14:40:53.744 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2023-01-24T14:40:53.980 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2023-01-24T14:40:54.136 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-mgr-rook-2:18.0.0-1954.g1a671c19.el8.noarch 13/13 2023-01-24T14:40:54.467 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-1954.g1a671c19.el8.noarch 13/13 2023-01-24T14:40:54.468 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-mgr-rook-2:18.0.0-1954.g1a671c19.el8.noarch 1/13 2023-01-24T14:40:54.468 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2023-01-24T14:40:54.468 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2023-01-24T14:40:54.468 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2023-01-24T14:40:54.468 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2023-01-24T14:40:54.469 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2023-01-24T14:40:54.469 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2023-01-24T14:40:54.469 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2023-01-24T14:40:54.469 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2023-01-24T14:40:54.469 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2023-01-24T14:40:54.469 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2023-01-24T14:40:54.470 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-rsa-4.8-1.el8.noarch 12/13 2023-01-24T14:40:55.024 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2023-01-24T14:40:55.024 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-24T14:40:55.025 INFO:teuthology.orchestra.run.smithi027.stdout:Installed: 2023-01-24T14:40:55.025 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-rook-2:18.0.0-1954.g1a671c19.el8.noarch 2023-01-24T14:40:55.025 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-01-24T14:40:55.025 INFO:teuthology.orchestra.run.smithi027.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-01-24T14:40:55.025 INFO:teuthology.orchestra.run.smithi027.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-01-24T14:40:55.026 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-01-24T14:40:55.026 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-01-24T14:40:55.026 INFO:teuthology.orchestra.run.smithi027.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-01-24T14:40:55.026 INFO:teuthology.orchestra.run.smithi027.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-01-24T14:40:55.026 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-01-24T14:40:55.026 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-01-24T14:40:55.027 INFO:teuthology.orchestra.run.smithi027.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-01-24T14:40:55.027 INFO:teuthology.orchestra.run.smithi027.stdout: python3-rsa-4.8-1.el8.noarch 2023-01-24T14:40:55.027 INFO:teuthology.orchestra.run.smithi027.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-01-24T14:40:55.027 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-24T14:40:55.027 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2023-01-24T14:40:55.153 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install ceph-mgr-cephadm 2023-01-24T14:40:55.928 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:12:47 ago on Tue 24 Jan 2023 02:28:08 PM UTC. 2023-01-24T14:40:57.160 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2023-01-24T14:40:57.161 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2023-01-24T14:40:57.161 INFO:teuthology.orchestra.run.smithi027.stdout: Package Arch Version Repository Size 2023-01-24T14:40:57.161 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2023-01-24T14:40:57.162 INFO:teuthology.orchestra.run.smithi027.stdout:Installing: 2023-01-24T14:40:57.162 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-cephadm noarch 2:18.0.0-1954.g1a671c19.el8 ceph-noarch 120 k 2023-01-24T14:40:57.162 INFO:teuthology.orchestra.run.smithi027.stdout:Installing dependencies: 2023-01-24T14:40:57.162 INFO:teuthology.orchestra.run.smithi027.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-01-24T14:40:57.162 INFO:teuthology.orchestra.run.smithi027.stdout: python3-babel noarch 2.5.1-7.el8 CentOS-AppStream 4.8 M 2023-01-24T14:40:57.163 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jinja2 noarch 2.10.1-3.el8 CentOS-AppStream 538 k 2023-01-24T14:40:57.163 INFO:teuthology.orchestra.run.smithi027.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-01-24T14:40:57.163 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-24T14:40:57.163 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2023-01-24T14:40:57.163 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2023-01-24T14:40:57.164 INFO:teuthology.orchestra.run.smithi027.stdout:Install 5 Packages 2023-01-24T14:40:57.164 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-24T14:40:57.164 INFO:teuthology.orchestra.run.smithi027.stdout:Total download size: 6.0 M 2023-01-24T14:40:57.165 INFO:teuthology.orchestra.run.smithi027.stdout:Installed size: 26 M 2023-01-24T14:40:57.165 INFO:teuthology.orchestra.run.smithi027.stdout:Downloading Packages: 2023-01-24T14:40:57.252 INFO:teuthology.orchestra.run.smithi027.stdout:(1/5): python3-jinja2-2.10.1-3.el8.noarch.rpm 5.9 MB/s | 538 kB 00:00 2023-01-24T14:40:57.353 INFO:teuthology.orchestra.run.smithi027.stdout:(2/5): python3-babel-2.5.1-7.el8.noarch.rpm 25 MB/s | 4.8 MB 00:00 2023-01-24T14:40:57.395 INFO:teuthology.orchestra.run.smithi027.stdout:(3/5): ceph-mgr-cephadm-18.0.0-1954.g1a671c19.e 519 kB/s | 120 kB 00:00 2023-01-24T14:40:57.503 INFO:teuthology.orchestra.run.smithi027.stdout:(4/5): python3-natsort-7.1.1-2.el8.noarch.rpm 401 kB/s | 60 kB 00:00 2023-01-24T14:40:57.587 INFO:teuthology.orchestra.run.smithi027.stdout:(5/5): python3-asyncssh-2.7.0-2.el8.noarch.rpm 1.4 MB/s | 476 kB 00:00 2023-01-24T14:40:57.589 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2023-01-24T14:40:57.589 INFO:teuthology.orchestra.run.smithi027.stdout:Total 14 MB/s | 6.0 MB 00:00 2023-01-24T14:40:57.641 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction check 2023-01-24T14:40:57.653 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction check succeeded. 2023-01-24T14:40:57.661 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction test 2023-01-24T14:40:57.803 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction test succeeded. 2023-01-24T14:40:57.807 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction 2023-01-24T14:40:58.098 INFO:teuthology.orchestra.run.smithi027.stdout: Preparing : 1/1 2023-01-24T14:40:58.493 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 1/5 2023-01-24T14:40:59.583 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 2/5 2023-01-24T14:41:00.023 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 3/5 2023-01-24T14:41:00.320 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 4/5 2023-01-24T14:41:00.544 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-1954.g1a671c19.el8.noarch 5/5 2023-01-24T14:41:00.851 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-1954.g1a671c19.el8.noarch 5/5 2023-01-24T14:41:00.851 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-1954.g1a671c19.el8.noarch 1/5 2023-01-24T14:41:00.852 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/5 2023-01-24T14:41:00.852 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 3/5 2023-01-24T14:41:00.852 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 4/5 2023-01-24T14:41:01.376 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 5/5 2023-01-24T14:41:01.377 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-24T14:41:01.377 INFO:teuthology.orchestra.run.smithi027.stdout:Installed: 2023-01-24T14:41:01.377 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-cephadm-2:18.0.0-1954.g1a671c19.el8.noarch 2023-01-24T14:41:01.378 INFO:teuthology.orchestra.run.smithi027.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-01-24T14:41:01.378 INFO:teuthology.orchestra.run.smithi027.stdout: python3-babel-2.5.1-7.el8.noarch 2023-01-24T14:41:01.378 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jinja2-2.10.1-3.el8.noarch 2023-01-24T14:41:01.378 INFO:teuthology.orchestra.run.smithi027.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-01-24T14:41:01.378 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-24T14:41:01.379 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2023-01-24T14:41:01.479 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install ceph-fuse 2023-01-24T14:41:02.241 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:12:54 ago on Tue 24 Jan 2023 02:28:08 PM UTC. 2023-01-24T14:41:03.469 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2023-01-24T14:41:03.470 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2023-01-24T14:41:03.470 INFO:teuthology.orchestra.run.smithi027.stdout: Package Arch Version Repository Size 2023-01-24T14:41:03.470 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2023-01-24T14:41:03.470 INFO:teuthology.orchestra.run.smithi027.stdout:Installing: 2023-01-24T14:41:03.470 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-fuse x86_64 2:18.0.0-1954.g1a671c19.el8 ceph 814 k 2023-01-24T14:41:03.471 INFO:teuthology.orchestra.run.smithi027.stdout:Installing dependencies: 2023-01-24T14:41:03.471 INFO:teuthology.orchestra.run.smithi027.stdout: fuse x86_64 2.9.7-16.el8 CentOS-BaseOS 83 k 2023-01-24T14:41:03.471 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-24T14:41:03.471 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2023-01-24T14:41:03.471 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2023-01-24T14:41:03.472 INFO:teuthology.orchestra.run.smithi027.stdout:Install 2 Packages 2023-01-24T14:41:03.472 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-24T14:41:03.472 INFO:teuthology.orchestra.run.smithi027.stdout:Total download size: 897 k 2023-01-24T14:41:03.473 INFO:teuthology.orchestra.run.smithi027.stdout:Installed size: 3.3 M 2023-01-24T14:41:03.473 INFO:teuthology.orchestra.run.smithi027.stdout:Downloading Packages: 2023-01-24T14:41:03.545 INFO:teuthology.orchestra.run.smithi027.stdout:(1/2): fuse-2.9.7-16.el8.x86_64.rpm 1.1 MB/s | 83 kB 00:00 2023-01-24T14:41:03.978 INFO:teuthology.orchestra.run.smithi027.stdout:(2/2): ceph-fuse-18.0.0-1954.g1a671c19.el8.x86_ 1.6 MB/s | 814 kB 00:00 2023-01-24T14:41:03.980 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2023-01-24T14:41:03.980 INFO:teuthology.orchestra.run.smithi027.stdout:Total 1.7 MB/s | 897 kB 00:00 2023-01-24T14:41:03.990 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction check 2023-01-24T14:41:04.013 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction check succeeded. 2023-01-24T14:41:04.013 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction test 2023-01-24T14:41:04.113 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction test succeeded. 2023-01-24T14:41:04.116 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction 2023-01-24T14:41:04.308 INFO:teuthology.orchestra.run.smithi027.stdout: Preparing : 1/1 2023-01-24T14:41:04.608 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : fuse-2.9.7-16.el8.x86_64 1/2 2023-01-24T14:41:04.776 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-fuse-2:18.0.0-1954.g1a671c19.el8.x86_64 2/2 2023-01-24T14:41:05.240 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-fuse-2:18.0.0-1954.g1a671c19.el8.x86_64 2/2 2023-01-24T14:41:05.240 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-fuse-2:18.0.0-1954.g1a671c19.el8.x86_64 1/2 2023-01-24T14:41:05.785 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : fuse-2.9.7-16.el8.x86_64 2/2 2023-01-24T14:41:05.786 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-24T14:41:05.786 INFO:teuthology.orchestra.run.smithi027.stdout:Installed: 2023-01-24T14:41:05.786 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-fuse-2:18.0.0-1954.g1a671c19.el8.x86_64 fuse-2.9.7-16.el8.x86_64 2023-01-24T14:41:05.786 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-24T14:41:05.787 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2023-01-24T14:41:06.017 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install ceph-volume 2023-01-24T14:41:06.789 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:12:58 ago on Tue 24 Jan 2023 02:28:08 PM UTC. 2023-01-24T14:41:08.018 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2023-01-24T14:41:08.019 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2023-01-24T14:41:08.019 INFO:teuthology.orchestra.run.smithi027.stdout: Package Arch Version Repository Size 2023-01-24T14:41:08.019 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2023-01-24T14:41:08.020 INFO:teuthology.orchestra.run.smithi027.stdout:Installing: 2023-01-24T14:41:08.020 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-volume noarch 2:18.0.0-1954.g1a671c19.el8 ceph-noarch 263 k 2023-01-24T14:41:08.020 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-24T14:41:08.020 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2023-01-24T14:41:08.021 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2023-01-24T14:41:08.021 INFO:teuthology.orchestra.run.smithi027.stdout:Install 1 Package 2023-01-24T14:41:08.021 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-24T14:41:08.021 INFO:teuthology.orchestra.run.smithi027.stdout:Total download size: 263 k 2023-01-24T14:41:08.021 INFO:teuthology.orchestra.run.smithi027.stdout:Installed size: 1.3 M 2023-01-24T14:41:08.021 INFO:teuthology.orchestra.run.smithi027.stdout:Downloading Packages: 2023-01-24T14:41:08.249 INFO:teuthology.orchestra.run.smithi027.stdout:ceph-volume-18.0.0-1954.g1a671c19.el8.noarch.rp 1.1 MB/s | 263 kB 00:00 2023-01-24T14:41:08.250 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2023-01-24T14:41:08.250 INFO:teuthology.orchestra.run.smithi027.stdout:Total 1.1 MB/s | 263 kB 00:00 2023-01-24T14:41:08.250 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction check 2023-01-24T14:41:08.261 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction check succeeded. 2023-01-24T14:41:08.261 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction test 2023-01-24T14:41:08.362 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction test succeeded. 2023-01-24T14:41:08.365 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction 2023-01-24T14:41:08.595 INFO:teuthology.orchestra.run.smithi027.stdout: Preparing : 1/1 2023-01-24T14:41:08.868 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-volume-2:18.0.0-1954.g1a671c19.el8.noarch 1/1 2023-01-24T14:41:09.312 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-volume-2:18.0.0-1954.g1a671c19.el8.noarch 1/1 2023-01-24T14:41:09.931 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-volume-2:18.0.0-1954.g1a671c19.el8.noarch 1/1 2023-01-24T14:41:09.931 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-24T14:41:09.931 INFO:teuthology.orchestra.run.smithi027.stdout:Installed: 2023-01-24T14:41:09.931 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-volume-2:18.0.0-1954.g1a671c19.el8.noarch 2023-01-24T14:41:09.932 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-24T14:41:09.932 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2023-01-24T14:41:10.155 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install librados-devel 2023-01-24T14:41:10.930 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:13:02 ago on Tue 24 Jan 2023 02:28:08 PM UTC. 2023-01-24T14:41:12.148 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2023-01-24T14:41:12.149 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2023-01-24T14:41:12.149 INFO:teuthology.orchestra.run.smithi027.stdout: Package Arch Version Repo Size 2023-01-24T14:41:12.149 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2023-01-24T14:41:12.149 INFO:teuthology.orchestra.run.smithi027.stdout:Installing: 2023-01-24T14:41:12.150 INFO:teuthology.orchestra.run.smithi027.stdout: librados-devel x86_64 2:18.0.0-1954.g1a671c19.el8 ceph 139 k 2023-01-24T14:41:12.150 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-24T14:41:12.150 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2023-01-24T14:41:12.150 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2023-01-24T14:41:12.151 INFO:teuthology.orchestra.run.smithi027.stdout:Install 1 Package 2023-01-24T14:41:12.151 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-24T14:41:12.152 INFO:teuthology.orchestra.run.smithi027.stdout:Total download size: 139 k 2023-01-24T14:41:12.152 INFO:teuthology.orchestra.run.smithi027.stdout:Installed size: 512 k 2023-01-24T14:41:12.152 INFO:teuthology.orchestra.run.smithi027.stdout:Downloading Packages: 2023-01-24T14:41:13.231 INFO:teuthology.orchestra.run.smithi027.stdout:librados-devel-18.0.0-1954.g1a671c19.el8.x86_64 128 kB/s | 139 kB 00:01 2023-01-24T14:41:13.232 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2023-01-24T14:41:13.232 INFO:teuthology.orchestra.run.smithi027.stdout:Total 128 kB/s | 139 kB 00:01 2023-01-24T14:41:13.233 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction check 2023-01-24T14:41:13.239 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction check succeeded. 2023-01-24T14:41:13.239 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction test 2023-01-24T14:41:13.301 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction test succeeded. 2023-01-24T14:41:13.304 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction 2023-01-24T14:41:13.429 INFO:teuthology.orchestra.run.smithi027.stdout: Preparing : 1/1 2023-01-24T14:41:13.615 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : librados-devel-2:18.0.0-1954.g1a671c19.el8.x86_64 1/1 2023-01-24T14:41:13.921 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: librados-devel-2:18.0.0-1954.g1a671c19.el8.x86_64 1/1 2023-01-24T14:41:14.500 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : librados-devel-2:18.0.0-1954.g1a671c19.el8.x86_64 1/1 2023-01-24T14:41:14.500 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-24T14:41:14.500 INFO:teuthology.orchestra.run.smithi027.stdout:Installed: 2023-01-24T14:41:14.500 INFO:teuthology.orchestra.run.smithi027.stdout: librados-devel-2:18.0.0-1954.g1a671c19.el8.x86_64 2023-01-24T14:41:14.501 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-24T14:41:14.501 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2023-01-24T14:41:14.704 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install libcephfs2 2023-01-24T14:41:15.462 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:13:07 ago on Tue 24 Jan 2023 02:28:08 PM UTC. 2023-01-24T14:41:16.578 INFO:teuthology.orchestra.run.smithi027.stdout:Package libcephfs2-2:18.0.0-1954.g1a671c19.el8.x86_64 is already installed. 2023-01-24T14:41:16.691 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2023-01-24T14:41:16.692 INFO:teuthology.orchestra.run.smithi027.stdout:Nothing to do. 2023-01-24T14:41:16.692 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2023-01-24T14:41:16.790 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install libcephfs-devel 2023-01-24T14:41:17.568 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:13:09 ago on Tue 24 Jan 2023 02:28:08 PM UTC. 2023-01-24T14:41:18.805 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2023-01-24T14:41:18.806 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2023-01-24T14:41:18.806 INFO:teuthology.orchestra.run.smithi027.stdout: Package Arch Version Repo Size 2023-01-24T14:41:18.806 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2023-01-24T14:41:18.806 INFO:teuthology.orchestra.run.smithi027.stdout:Installing: 2023-01-24T14:41:18.807 INFO:teuthology.orchestra.run.smithi027.stdout: libcephfs-devel x86_64 2:18.0.0-1954.g1a671c19.el8 ceph 30 k 2023-01-24T14:41:18.807 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-24T14:41:18.807 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2023-01-24T14:41:18.807 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2023-01-24T14:41:18.807 INFO:teuthology.orchestra.run.smithi027.stdout:Install 1 Package 2023-01-24T14:41:18.808 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-24T14:41:18.808 INFO:teuthology.orchestra.run.smithi027.stdout:Total download size: 30 k 2023-01-24T14:41:18.808 INFO:teuthology.orchestra.run.smithi027.stdout:Installed size: 137 k 2023-01-24T14:41:18.808 INFO:teuthology.orchestra.run.smithi027.stdout:Downloading Packages: 2023-01-24T14:41:19.043 INFO:teuthology.orchestra.run.smithi027.stdout:libcephfs-devel-18.0.0-1954.g1a671c19.el8.x86_6 129 kB/s | 30 kB 00:00 2023-01-24T14:41:19.044 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2023-01-24T14:41:19.044 INFO:teuthology.orchestra.run.smithi027.stdout:Total 128 kB/s | 30 kB 00:00 2023-01-24T14:41:19.044 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction check 2023-01-24T14:41:19.047 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction check succeeded. 2023-01-24T14:41:19.048 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction test 2023-01-24T14:41:19.073 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction test succeeded. 2023-01-24T14:41:19.077 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction 2023-01-24T14:41:19.136 INFO:teuthology.orchestra.run.smithi027.stdout: Preparing : 1/1 2023-01-24T14:41:19.356 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libcephfs-devel-2:18.0.0-1954.g1a671c19.el8.x86_64 1/1 2023-01-24T14:41:19.662 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: libcephfs-devel-2:18.0.0-1954.g1a671c19.el8.x86_64 1/1 2023-01-24T14:41:20.172 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libcephfs-devel-2:18.0.0-1954.g1a671c19.el8.x86_64 1/1 2023-01-24T14:41:20.173 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-24T14:41:20.173 INFO:teuthology.orchestra.run.smithi027.stdout:Installed: 2023-01-24T14:41:20.173 INFO:teuthology.orchestra.run.smithi027.stdout: libcephfs-devel-2:18.0.0-1954.g1a671c19.el8.x86_64 2023-01-24T14:41:20.173 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-24T14:41:20.173 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2023-01-24T14:41:20.276 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install librados2 2023-01-24T14:41:21.030 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:13:13 ago on Tue 24 Jan 2023 02:28:08 PM UTC. 2023-01-24T14:41:21.116 INFO:teuthology.orchestra.run.smithi035.stdout: Running scriptlet: ceph-mgr-2:18.0.0-1954.g1a671c19.el8.x86_64 28/32 2023-01-24T14:41:21.375 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : ceph-osd-2:18.0.0-1954.g1a671c19.el8.x86_64 29/32 2023-01-24T14:41:22.143 INFO:teuthology.orchestra.run.smithi027.stdout:Package librados2-2:18.0.0-1954.g1a671c19.el8.x86_64 is already installed. 2023-01-24T14:41:22.167 INFO:teuthology.orchestra.run.smithi035.stdout: Running scriptlet: ceph-osd-2:18.0.0-1954.g1a671c19.el8.x86_64 29/32 2023-01-24T14:41:22.257 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2023-01-24T14:41:22.259 INFO:teuthology.orchestra.run.smithi027.stdout:Nothing to do. 2023-01-24T14:41:22.259 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2023-01-24T14:41:22.354 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install librbd1 2023-01-24T14:41:22.393 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : ceph-mon-2:18.0.0-1954.g1a671c19.el8.x86_64 30/32 2023-01-24T14:41:22.620 INFO:teuthology.orchestra.run.smithi035.stdout: Running scriptlet: ceph-mon-2:18.0.0-1954.g1a671c19.el8.x86_64 30/32 2023-01-24T14:41:22.793 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : ceph-mds-2:18.0.0-1954.g1a671c19.el8.x86_64 31/32 2023-01-24T14:41:22.811 INFO:teuthology.orchestra.run.smithi035.stdout: Running scriptlet: ceph-mds-2:18.0.0-1954.g1a671c19.el8.x86_64 31/32 2023-01-24T14:41:22.983 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : ceph-2:18.0.0-1954.g1a671c19.el8.x86_64 32/32 2023-01-24T14:41:23.120 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:13:15 ago on Tue 24 Jan 2023 02:28:08 PM UTC. 2023-01-24T14:41:23.424 INFO:teuthology.orchestra.run.smithi035.stdout: Running scriptlet: ceph-2:18.0.0-1954.g1a671c19.el8.x86_64 32/32 2023-01-24T14:41:23.424 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : ceph-2:18.0.0-1954.g1a671c19.el8.x86_64 1/32 2023-01-24T14:41:23.425 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : ceph-mds-2:18.0.0-1954.g1a671c19.el8.x86_64 2/32 2023-01-24T14:41:23.425 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : ceph-mgr-2:18.0.0-1954.g1a671c19.el8.x86_64 3/32 2023-01-24T14:41:23.425 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : ceph-mon-2:18.0.0-1954.g1a671c19.el8.x86_64 4/32 2023-01-24T14:41:23.425 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : ceph-osd-2:18.0.0-1954.g1a671c19.el8.x86_64 5/32 2023-01-24T14:41:23.425 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-1954.g1a671c19.el8. 6/32 2023-01-24T14:41:23.426 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 7/32 2023-01-24T14:41:23.426 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2023-01-24T14:41:23.426 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2023-01-24T14:41:23.426 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 10/32 2023-01-24T14:41:23.426 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2023-01-24T14:41:23.427 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 12/32 2023-01-24T14:41:23.427 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : python3-cryptography-3.2.1-5.el8.x86_64 13/32 2023-01-24T14:41:23.427 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2023-01-24T14:41:23.427 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2023-01-24T14:41:23.427 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2023-01-24T14:41:23.428 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2023-01-24T14:41:23.428 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2023-01-24T14:41:23.428 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2023-01-24T14:41:23.428 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2023-01-24T14:41:23.428 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2023-01-24T14:41:23.429 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2023-01-24T14:41:23.429 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2023-01-24T14:41:23.429 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2023-01-24T14:41:23.429 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2023-01-24T14:41:23.430 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2023-01-24T14:41:23.431 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2023-01-24T14:41:23.431 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2023-01-24T14:41:23.431 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : python3-waitress-1.2.1-2.el8.1.noarch 29/32 2023-01-24T14:41:23.431 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2023-01-24T14:41:23.431 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2023-01-24T14:41:24.250 INFO:teuthology.orchestra.run.smithi027.stdout:Package librbd1-2:18.0.0-1954.g1a671c19.el8.x86_64 is already installed. 2023-01-24T14:41:24.349 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2023-01-24T14:41:24.349 INFO:teuthology.orchestra.run.smithi035.stdout: 2023-01-24T14:41:24.349 INFO:teuthology.orchestra.run.smithi035.stdout:Installed: 2023-01-24T14:41:24.349 INFO:teuthology.orchestra.run.smithi035.stdout: ceph-2:18.0.0-1954.g1a671c19.el8.x86_64 2023-01-24T14:41:24.350 INFO:teuthology.orchestra.run.smithi035.stdout: ceph-mds-2:18.0.0-1954.g1a671c19.el8.x86_64 2023-01-24T14:41:24.350 INFO:teuthology.orchestra.run.smithi035.stdout: ceph-mgr-2:18.0.0-1954.g1a671c19.el8.x86_64 2023-01-24T14:41:24.350 INFO:teuthology.orchestra.run.smithi035.stdout: ceph-mgr-modules-core-2:18.0.0-1954.g1a671c19.el8.noarch 2023-01-24T14:41:24.350 INFO:teuthology.orchestra.run.smithi035.stdout: ceph-mon-2:18.0.0-1954.g1a671c19.el8.x86_64 2023-01-24T14:41:24.350 INFO:teuthology.orchestra.run.smithi035.stdout: ceph-osd-2:18.0.0-1954.g1a671c19.el8.x86_64 2023-01-24T14:41:24.350 INFO:teuthology.orchestra.run.smithi035.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-01-24T14:41:24.351 INFO:teuthology.orchestra.run.smithi035.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-01-24T14:41:24.351 INFO:teuthology.orchestra.run.smithi035.stdout: python3-cffi-1.11.5-5.el8.x86_64 2023-01-24T14:41:24.351 INFO:teuthology.orchestra.run.smithi035.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-01-24T14:41:24.351 INFO:teuthology.orchestra.run.smithi035.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-01-24T14:41:24.351 INFO:teuthology.orchestra.run.smithi035.stdout: python3-cryptography-3.2.1-5.el8.x86_64 2023-01-24T14:41:24.352 INFO:teuthology.orchestra.run.smithi035.stdout: python3-jaraco-6.2-6.el8.noarch 2023-01-24T14:41:24.352 INFO:teuthology.orchestra.run.smithi035.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-01-24T14:41:24.352 INFO:teuthology.orchestra.run.smithi035.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-01-24T14:41:24.352 INFO:teuthology.orchestra.run.smithi035.stdout: python3-mako-1.0.6-13.el8.noarch 2023-01-24T14:41:24.352 INFO:teuthology.orchestra.run.smithi035.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-01-24T14:41:24.353 INFO:teuthology.orchestra.run.smithi035.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-01-24T14:41:24.353 INFO:teuthology.orchestra.run.smithi035.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-01-24T14:41:24.353 INFO:teuthology.orchestra.run.smithi035.stdout: python3-portend-2.6-1.el8.noarch 2023-01-24T14:41:24.353 INFO:teuthology.orchestra.run.smithi035.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-01-24T14:41:24.353 INFO:teuthology.orchestra.run.smithi035.stdout: python3-pycparser-2.14-14.el8.noarch 2023-01-24T14:41:24.354 INFO:teuthology.orchestra.run.smithi035.stdout: python3-pytz-2017.2-9.el8.noarch 2023-01-24T14:41:24.354 INFO:teuthology.orchestra.run.smithi035.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-01-24T14:41:24.354 INFO:teuthology.orchestra.run.smithi035.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-01-24T14:41:24.354 INFO:teuthology.orchestra.run.smithi035.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-01-24T14:41:24.354 INFO:teuthology.orchestra.run.smithi035.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-01-24T14:41:24.354 INFO:teuthology.orchestra.run.smithi035.stdout: python3-waitress-1.2.1-2.el8.1.noarch 2023-01-24T14:41:24.355 INFO:teuthology.orchestra.run.smithi035.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-01-24T14:41:24.355 INFO:teuthology.orchestra.run.smithi035.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-01-24T14:41:24.355 INFO:teuthology.orchestra.run.smithi035.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-01-24T14:41:24.355 INFO:teuthology.orchestra.run.smithi035.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-01-24T14:41:24.355 INFO:teuthology.orchestra.run.smithi035.stdout: 2023-01-24T14:41:24.356 INFO:teuthology.orchestra.run.smithi035.stdout:Complete! 2023-01-24T14:41:24.363 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2023-01-24T14:41:24.365 INFO:teuthology.orchestra.run.smithi027.stdout:Nothing to do. 2023-01-24T14:41:24.365 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2023-01-24T14:41:24.455 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install python3-rados 2023-01-24T14:41:24.581 DEBUG:teuthology.orchestra.run.smithi035:> sudo yum -y install ceph-base 2023-01-24T14:41:25.222 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:13:17 ago on Tue 24 Jan 2023 02:28:08 PM UTC. 2023-01-24T14:41:25.337 INFO:teuthology.orchestra.run.smithi035.stdout:Last metadata expiration check: 0:13:16 ago on Tue 24 Jan 2023 02:28:09 PM UTC. 2023-01-24T14:41:26.332 INFO:teuthology.orchestra.run.smithi027.stdout:Package python3-rados-2:18.0.0-1954.g1a671c19.el8.x86_64 is already installed. 2023-01-24T14:41:26.445 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2023-01-24T14:41:26.446 INFO:teuthology.orchestra.run.smithi027.stdout:Nothing to do. 2023-01-24T14:41:26.447 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2023-01-24T14:41:26.465 INFO:teuthology.orchestra.run.smithi035.stdout:Package ceph-base-2:18.0.0-1954.g1a671c19.el8.x86_64 is already installed. 2023-01-24T14:41:26.534 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install python3-rgw 2023-01-24T14:41:26.575 INFO:teuthology.orchestra.run.smithi035.stdout:Dependencies resolved. 2023-01-24T14:41:26.576 INFO:teuthology.orchestra.run.smithi035.stdout:Nothing to do. 2023-01-24T14:41:26.576 INFO:teuthology.orchestra.run.smithi035.stdout:Complete! 2023-01-24T14:41:26.960 DEBUG:teuthology.orchestra.run.smithi035:> sudo yum -y install cephadm 2023-01-24T14:41:27.303 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:13:19 ago on Tue 24 Jan 2023 02:28:08 PM UTC. 2023-01-24T14:41:27.717 INFO:teuthology.orchestra.run.smithi035.stdout:Last metadata expiration check: 0:13:18 ago on Tue 24 Jan 2023 02:28:09 PM UTC. 2023-01-24T14:41:28.418 INFO:teuthology.orchestra.run.smithi027.stdout:Package python3-rgw-2:18.0.0-1954.g1a671c19.el8.x86_64 is already installed. 2023-01-24T14:41:28.532 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2023-01-24T14:41:28.533 INFO:teuthology.orchestra.run.smithi027.stdout:Nothing to do. 2023-01-24T14:41:28.533 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2023-01-24T14:41:28.613 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install python3-cephfs 2023-01-24T14:41:28.951 INFO:teuthology.orchestra.run.smithi035.stdout:Dependencies resolved. 2023-01-24T14:41:28.952 INFO:teuthology.orchestra.run.smithi035.stdout:================================================================================ 2023-01-24T14:41:28.952 INFO:teuthology.orchestra.run.smithi035.stdout: Package Arch Version Repository Size 2023-01-24T14:41:28.952 INFO:teuthology.orchestra.run.smithi035.stdout:================================================================================ 2023-01-24T14:41:28.952 INFO:teuthology.orchestra.run.smithi035.stdout:Installing: 2023-01-24T14:41:28.952 INFO:teuthology.orchestra.run.smithi035.stdout: cephadm noarch 2:18.0.0-1954.g1a671c19.el8 ceph-noarch 82 k 2023-01-24T14:41:28.953 INFO:teuthology.orchestra.run.smithi035.stdout: 2023-01-24T14:41:28.953 INFO:teuthology.orchestra.run.smithi035.stdout:Transaction Summary 2023-01-24T14:41:28.953 INFO:teuthology.orchestra.run.smithi035.stdout:================================================================================ 2023-01-24T14:41:28.953 INFO:teuthology.orchestra.run.smithi035.stdout:Install 1 Package 2023-01-24T14:41:28.953 INFO:teuthology.orchestra.run.smithi035.stdout: 2023-01-24T14:41:28.954 INFO:teuthology.orchestra.run.smithi035.stdout:Total download size: 82 k 2023-01-24T14:41:28.954 INFO:teuthology.orchestra.run.smithi035.stdout:Installed size: 374 k 2023-01-24T14:41:28.954 INFO:teuthology.orchestra.run.smithi035.stdout:Downloading Packages: 2023-01-24T14:41:29.383 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:13:21 ago on Tue 24 Jan 2023 02:28:08 PM UTC. 2023-01-24T14:41:29.617 INFO:teuthology.orchestra.run.smithi035.stdout:cephadm-18.0.0-1954.g1a671c19.el8.noarch.rpm 124 kB/s | 82 kB 00:00 2023-01-24T14:41:29.618 INFO:teuthology.orchestra.run.smithi035.stdout:-------------------------------------------------------------------------------- 2023-01-24T14:41:29.618 INFO:teuthology.orchestra.run.smithi035.stdout:Total 124 kB/s | 82 kB 00:00 2023-01-24T14:41:29.619 INFO:teuthology.orchestra.run.smithi035.stdout:Running transaction check 2023-01-24T14:41:29.628 INFO:teuthology.orchestra.run.smithi035.stdout:Transaction check succeeded. 2023-01-24T14:41:29.628 INFO:teuthology.orchestra.run.smithi035.stdout:Running transaction test 2023-01-24T14:41:29.631 INFO:teuthology.orchestra.run.smithi035.stdout:Transaction test succeeded. 2023-01-24T14:41:29.635 INFO:teuthology.orchestra.run.smithi035.stdout:Running transaction 2023-01-24T14:41:30.152 INFO:teuthology.orchestra.run.smithi035.stdout: Preparing : 1/1 2023-01-24T14:41:30.500 INFO:teuthology.orchestra.run.smithi027.stdout:Package python3-cephfs-2:18.0.0-1954.g1a671c19.el8.x86_64 is already installed. 2023-01-24T14:41:30.614 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2023-01-24T14:41:30.615 INFO:teuthology.orchestra.run.smithi027.stdout:Nothing to do. 2023-01-24T14:41:30.616 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2023-01-24T14:41:30.713 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install python3-rbd 2023-01-24T14:41:31.490 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:13:23 ago on Tue 24 Jan 2023 02:28:08 PM UTC. 2023-01-24T14:41:32.601 INFO:teuthology.orchestra.run.smithi027.stdout:Package python3-rbd-2:18.0.0-1954.g1a671c19.el8.x86_64 is already installed. 2023-01-24T14:41:32.715 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2023-01-24T14:41:32.716 INFO:teuthology.orchestra.run.smithi027.stdout:Nothing to do. 2023-01-24T14:41:32.716 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2023-01-24T14:41:32.813 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install rbd-fuse 2023-01-24T14:41:33.565 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:13:25 ago on Tue 24 Jan 2023 02:28:08 PM UTC. 2023-01-24T14:41:34.783 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2023-01-24T14:41:34.784 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2023-01-24T14:41:34.785 INFO:teuthology.orchestra.run.smithi027.stdout: Package Architecture Version Repository Size 2023-01-24T14:41:34.785 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2023-01-24T14:41:34.785 INFO:teuthology.orchestra.run.smithi027.stdout:Installing: 2023-01-24T14:41:34.785 INFO:teuthology.orchestra.run.smithi027.stdout: rbd-fuse x86_64 2:18.0.0-1954.g1a671c19.el8 ceph 91 k 2023-01-24T14:41:34.786 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-24T14:41:34.786 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2023-01-24T14:41:34.786 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2023-01-24T14:41:34.786 INFO:teuthology.orchestra.run.smithi027.stdout:Install 1 Package 2023-01-24T14:41:34.786 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-24T14:41:34.786 INFO:teuthology.orchestra.run.smithi027.stdout:Total download size: 91 k 2023-01-24T14:41:34.787 INFO:teuthology.orchestra.run.smithi027.stdout:Installed size: 303 k 2023-01-24T14:41:34.787 INFO:teuthology.orchestra.run.smithi027.stdout:Downloading Packages: 2023-01-24T14:41:34.966 INFO:teuthology.orchestra.run.smithi027.stdout:rbd-fuse-18.0.0-1954.g1a671c19.el8.x86_64.rpm 504 kB/s | 91 kB 00:00 2023-01-24T14:41:34.967 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2023-01-24T14:41:34.967 INFO:teuthology.orchestra.run.smithi027.stdout:Total 500 kB/s | 91 kB 00:00 2023-01-24T14:41:34.967 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction check 2023-01-24T14:41:34.977 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction check succeeded. 2023-01-24T14:41:34.978 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction test 2023-01-24T14:41:35.036 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction test succeeded. 2023-01-24T14:41:35.039 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction 2023-01-24T14:41:35.163 INFO:teuthology.orchestra.run.smithi027.stdout: Preparing : 1/1 2023-01-24T14:41:35.380 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : rbd-fuse-2:18.0.0-1954.g1a671c19.el8.x86_64 1/1 2023-01-24T14:41:35.689 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: rbd-fuse-2:18.0.0-1954.g1a671c19.el8.x86_64 1/1 2023-01-24T14:41:36.187 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : rbd-fuse-2:18.0.0-1954.g1a671c19.el8.x86_64 1/1 2023-01-24T14:41:36.188 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-24T14:41:36.188 INFO:teuthology.orchestra.run.smithi027.stdout:Installed: 2023-01-24T14:41:36.188 INFO:teuthology.orchestra.run.smithi027.stdout: rbd-fuse-2:18.0.0-1954.g1a671c19.el8.x86_64 2023-01-24T14:41:36.188 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-24T14:41:36.189 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2023-01-24T14:41:36.397 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install rbd-mirror 2023-01-24T14:41:37.160 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:13:29 ago on Tue 24 Jan 2023 02:28:08 PM UTC. 2023-01-24T14:41:38.385 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2023-01-24T14:41:38.386 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2023-01-24T14:41:38.386 INFO:teuthology.orchestra.run.smithi027.stdout: Package Arch Version Repo Size 2023-01-24T14:41:38.386 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2023-01-24T14:41:38.386 INFO:teuthology.orchestra.run.smithi027.stdout:Installing: 2023-01-24T14:41:38.386 INFO:teuthology.orchestra.run.smithi027.stdout: rbd-mirror x86_64 2:18.0.0-1954.g1a671c19.el8 ceph 3.3 M 2023-01-24T14:41:38.387 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-24T14:41:38.387 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2023-01-24T14:41:38.387 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2023-01-24T14:41:38.387 INFO:teuthology.orchestra.run.smithi027.stdout:Install 1 Package 2023-01-24T14:41:38.387 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-24T14:41:38.388 INFO:teuthology.orchestra.run.smithi027.stdout:Total download size: 3.3 M 2023-01-24T14:41:38.388 INFO:teuthology.orchestra.run.smithi027.stdout:Installed size: 18 M 2023-01-24T14:41:38.388 INFO:teuthology.orchestra.run.smithi027.stdout:Downloading Packages: 2023-01-24T14:41:38.843 INFO:teuthology.orchestra.run.smithi035.stdout: Running scriptlet: cephadm-2:18.0.0-1954.g1a671c19.el8.noarch 1/1 2023-01-24T14:41:38.894 INFO:teuthology.orchestra.run.smithi027.stdout:rbd-mirror-18.0.0-1954.g1a671c19.el8.x86_64.rpm 6.5 MB/s | 3.3 MB 00:00 2023-01-24T14:41:38.895 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2023-01-24T14:41:38.895 INFO:teuthology.orchestra.run.smithi027.stdout:Total 6.5 MB/s | 3.3 MB 00:00 2023-01-24T14:41:38.896 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction check 2023-01-24T14:41:38.908 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction check succeeded. 2023-01-24T14:41:38.909 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction test 2023-01-24T14:41:38.977 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction test succeeded. 2023-01-24T14:41:38.980 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction 2023-01-24T14:41:39.451 INFO:teuthology.orchestra.run.smithi027.stdout: Preparing : 1/1 2023-01-24T14:41:39.664 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : rbd-mirror-2:18.0.0-1954.g1a671c19.el8.x86_64 1/1 2023-01-24T14:41:39.998 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : cephadm-2:18.0.0-1954.g1a671c19.el8.noarch 1/1 2023-01-24T14:41:40.137 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: rbd-mirror-2:18.0.0-1954.g1a671c19.el8.x86_64 1/1 2023-01-24T14:41:40.258 INFO:teuthology.orchestra.run.smithi035.stdout: Running scriptlet: cephadm-2:18.0.0-1954.g1a671c19.el8.noarch 1/1 2023-01-24T14:41:40.744 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : cephadm-2:18.0.0-1954.g1a671c19.el8.noarch 1/1 2023-01-24T14:41:40.745 INFO:teuthology.orchestra.run.smithi035.stdout: 2023-01-24T14:41:40.745 INFO:teuthology.orchestra.run.smithi035.stdout:Installed: 2023-01-24T14:41:40.745 INFO:teuthology.orchestra.run.smithi035.stdout: cephadm-2:18.0.0-1954.g1a671c19.el8.noarch 2023-01-24T14:41:40.745 INFO:teuthology.orchestra.run.smithi035.stdout: 2023-01-24T14:41:40.746 INFO:teuthology.orchestra.run.smithi035.stdout:Complete! 2023-01-24T14:41:40.775 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : rbd-mirror-2:18.0.0-1954.g1a671c19.el8.x86_64 1/1 2023-01-24T14:41:40.776 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-24T14:41:40.776 INFO:teuthology.orchestra.run.smithi027.stdout:Installed: 2023-01-24T14:41:40.776 INFO:teuthology.orchestra.run.smithi027.stdout: rbd-mirror-2:18.0.0-1954.g1a671c19.el8.x86_64 2023-01-24T14:41:40.777 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-24T14:41:40.777 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2023-01-24T14:41:40.957 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install rbd-nbd 2023-01-24T14:41:40.961 DEBUG:teuthology.orchestra.run.smithi035:> sudo yum -y install ceph-immutable-object-cache 2023-01-24T14:41:41.723 INFO:teuthology.orchestra.run.smithi035.stdout:Last metadata expiration check: 0:13:32 ago on Tue 24 Jan 2023 02:28:09 PM UTC. 2023-01-24T14:41:41.726 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:13:33 ago on Tue 24 Jan 2023 02:28:08 PM UTC. 2023-01-24T14:41:42.945 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2023-01-24T14:41:42.946 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2023-01-24T14:41:42.946 INFO:teuthology.orchestra.run.smithi027.stdout: Package Architecture Version Repository Size 2023-01-24T14:41:42.946 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2023-01-24T14:41:42.946 INFO:teuthology.orchestra.run.smithi027.stdout:Installing: 2023-01-24T14:41:42.946 INFO:teuthology.orchestra.run.smithi027.stdout: rbd-nbd x86_64 2:18.0.0-1954.g1a671c19.el8 ceph 177 k 2023-01-24T14:41:42.947 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-24T14:41:42.947 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2023-01-24T14:41:42.947 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2023-01-24T14:41:42.947 INFO:teuthology.orchestra.run.smithi027.stdout:Install 1 Package 2023-01-24T14:41:42.947 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-24T14:41:42.948 INFO:teuthology.orchestra.run.smithi027.stdout:Total download size: 177 k 2023-01-24T14:41:42.948 INFO:teuthology.orchestra.run.smithi027.stdout:Installed size: 626 k 2023-01-24T14:41:42.948 INFO:teuthology.orchestra.run.smithi027.stdout:Downloading Packages: 2023-01-24T14:41:42.958 INFO:teuthology.orchestra.run.smithi035.stdout:Dependencies resolved. 2023-01-24T14:41:42.959 INFO:teuthology.orchestra.run.smithi035.stdout:================================================================================ 2023-01-24T14:41:42.959 INFO:teuthology.orchestra.run.smithi035.stdout: Package Arch Version Repo Size 2023-01-24T14:41:42.959 INFO:teuthology.orchestra.run.smithi035.stdout:================================================================================ 2023-01-24T14:41:42.959 INFO:teuthology.orchestra.run.smithi035.stdout:Installing: 2023-01-24T14:41:42.960 INFO:teuthology.orchestra.run.smithi035.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-1954.g1a671c19.el8 ceph 157 k 2023-01-24T14:41:42.960 INFO:teuthology.orchestra.run.smithi035.stdout: 2023-01-24T14:41:42.960 INFO:teuthology.orchestra.run.smithi035.stdout:Transaction Summary 2023-01-24T14:41:42.960 INFO:teuthology.orchestra.run.smithi035.stdout:================================================================================ 2023-01-24T14:41:42.960 INFO:teuthology.orchestra.run.smithi035.stdout:Install 1 Package 2023-01-24T14:41:42.961 INFO:teuthology.orchestra.run.smithi035.stdout: 2023-01-24T14:41:42.961 INFO:teuthology.orchestra.run.smithi035.stdout:Total download size: 157 k 2023-01-24T14:41:42.961 INFO:teuthology.orchestra.run.smithi035.stdout:Installed size: 540 k 2023-01-24T14:41:42.962 INFO:teuthology.orchestra.run.smithi035.stdout:Downloading Packages: 2023-01-24T14:41:43.206 INFO:teuthology.orchestra.run.smithi027.stdout:rbd-nbd-18.0.0-1954.g1a671c19.el8.x86_64.rpm 682 kB/s | 177 kB 00:00 2023-01-24T14:41:43.207 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2023-01-24T14:41:43.207 INFO:teuthology.orchestra.run.smithi027.stdout:Total 678 kB/s | 177 kB 00:00 2023-01-24T14:41:43.207 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction check 2023-01-24T14:41:43.218 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction check succeeded. 2023-01-24T14:41:43.219 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction test 2023-01-24T14:41:43.241 INFO:teuthology.orchestra.run.smithi035.stdout:ceph-immutable-object-cache-18.0.0-1954.g1a671c 559 kB/s | 157 kB 00:00 2023-01-24T14:41:43.242 INFO:teuthology.orchestra.run.smithi035.stdout:-------------------------------------------------------------------------------- 2023-01-24T14:41:43.242 INFO:teuthology.orchestra.run.smithi035.stdout:Total 556 kB/s | 157 kB 00:00 2023-01-24T14:41:43.242 INFO:teuthology.orchestra.run.smithi035.stdout:Running transaction check 2023-01-24T14:41:43.254 INFO:teuthology.orchestra.run.smithi035.stdout:Transaction check succeeded. 2023-01-24T14:41:43.254 INFO:teuthology.orchestra.run.smithi035.stdout:Running transaction test 2023-01-24T14:41:43.275 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction test succeeded. 2023-01-24T14:41:43.278 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction 2023-01-24T14:41:43.307 INFO:teuthology.orchestra.run.smithi035.stdout:Transaction test succeeded. 2023-01-24T14:41:43.311 INFO:teuthology.orchestra.run.smithi035.stdout:Running transaction 2023-01-24T14:41:43.410 INFO:teuthology.orchestra.run.smithi027.stdout: Preparing : 1/1 2023-01-24T14:41:43.445 INFO:teuthology.orchestra.run.smithi035.stdout: Preparing : 1/1 2023-01-24T14:41:43.595 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : rbd-nbd-2:18.0.0-1954.g1a671c19.el8.x86_64 1/1 2023-01-24T14:41:43.607 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-1954.g1a671c19. 1/1 2023-01-24T14:41:43.909 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: rbd-nbd-2:18.0.0-1954.g1a671c19.el8.x86_64 1/1 2023-01-24T14:41:44.058 INFO:teuthology.orchestra.run.smithi035.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-1954.g1a671c19. 1/1 2023-01-24T14:41:44.403 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : rbd-nbd-2:18.0.0-1954.g1a671c19.el8.x86_64 1/1 2023-01-24T14:41:44.404 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-24T14:41:44.404 INFO:teuthology.orchestra.run.smithi027.stdout:Installed: 2023-01-24T14:41:44.404 INFO:teuthology.orchestra.run.smithi027.stdout: rbd-nbd-2:18.0.0-1954.g1a671c19.el8.x86_64 2023-01-24T14:41:44.404 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-24T14:41:44.405 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2023-01-24T14:41:44.544 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-1954.g1a671c19. 1/1 2023-01-24T14:41:44.545 INFO:teuthology.orchestra.run.smithi035.stdout: 2023-01-24T14:41:44.545 INFO:teuthology.orchestra.run.smithi035.stdout:Installed: 2023-01-24T14:41:44.545 INFO:teuthology.orchestra.run.smithi035.stdout: ceph-immutable-object-cache-2:18.0.0-1954.g1a671c19.el8.x86_64 2023-01-24T14:41:44.545 INFO:teuthology.orchestra.run.smithi035.stdout: 2023-01-24T14:41:44.546 INFO:teuthology.orchestra.run.smithi035.stdout:Complete! 2023-01-24T14:41:44.760 DEBUG:teuthology.orchestra.run.smithi035:> sudo yum -y install ceph-mgr 2023-01-24T14:41:44.931 DEBUG:teuthology.parallel:result is None 2023-01-24T14:41:45.516 INFO:teuthology.orchestra.run.smithi035.stdout:Last metadata expiration check: 0:13:36 ago on Tue 24 Jan 2023 02:28:09 PM UTC. 2023-01-24T14:41:46.639 INFO:teuthology.orchestra.run.smithi035.stdout:Package ceph-mgr-2:18.0.0-1954.g1a671c19.el8.x86_64 is already installed. 2023-01-24T14:41:46.749 INFO:teuthology.orchestra.run.smithi035.stdout:Dependencies resolved. 2023-01-24T14:41:46.750 INFO:teuthology.orchestra.run.smithi035.stdout:Nothing to do. 2023-01-24T14:41:46.750 INFO:teuthology.orchestra.run.smithi035.stdout:Complete! 2023-01-24T14:41:46.827 DEBUG:teuthology.orchestra.run.smithi035:> sudo yum -y install ceph-mgr-dashboard 2023-01-24T14:41:47.583 INFO:teuthology.orchestra.run.smithi035.stdout:Last metadata expiration check: 0:13:38 ago on Tue 24 Jan 2023 02:28:09 PM UTC. 2023-01-24T14:41:48.804 INFO:teuthology.orchestra.run.smithi035.stdout:Dependencies resolved. 2023-01-24T14:41:48.805 INFO:teuthology.orchestra.run.smithi035.stdout:================================================================================ 2023-01-24T14:41:48.806 INFO:teuthology.orchestra.run.smithi035.stdout: Package Arch Version Repository Size 2023-01-24T14:41:48.806 INFO:teuthology.orchestra.run.smithi035.stdout:================================================================================ 2023-01-24T14:41:48.806 INFO:teuthology.orchestra.run.smithi035.stdout:Installing: 2023-01-24T14:41:48.807 INFO:teuthology.orchestra.run.smithi035.stdout: ceph-mgr-dashboard noarch 2:18.0.0-1954.g1a671c19.el8 ceph-noarch 1.7 M 2023-01-24T14:41:48.807 INFO:teuthology.orchestra.run.smithi035.stdout:Installing dependencies: 2023-01-24T14:41:48.807 INFO:teuthology.orchestra.run.smithi035.stdout: ceph-grafana-dashboards noarch 2:18.0.0-1954.g1a671c19.el8 ceph-noarch 23 k 2023-01-24T14:41:48.807 INFO:teuthology.orchestra.run.smithi035.stdout: ceph-prometheus-alerts noarch 2:18.0.0-1954.g1a671c19.el8 ceph-noarch 15 k 2023-01-24T14:41:48.807 INFO:teuthology.orchestra.run.smithi035.stdout: python3-jwt noarch 1.6.1-2.el8 CentOS-BaseOS 43 k 2023-01-24T14:41:48.808 INFO:teuthology.orchestra.run.smithi035.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-01-24T14:41:48.808 INFO:teuthology.orchestra.run.smithi035.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-01-24T14:41:48.808 INFO:teuthology.orchestra.run.smithi035.stdout: 2023-01-24T14:41:48.808 INFO:teuthology.orchestra.run.smithi035.stdout:Transaction Summary 2023-01-24T14:41:48.809 INFO:teuthology.orchestra.run.smithi035.stdout:================================================================================ 2023-01-24T14:41:48.809 INFO:teuthology.orchestra.run.smithi035.stdout:Install 6 Packages 2023-01-24T14:41:48.809 INFO:teuthology.orchestra.run.smithi035.stdout: 2023-01-24T14:41:48.810 INFO:teuthology.orchestra.run.smithi035.stdout:Total download size: 2.0 M 2023-01-24T14:41:48.811 INFO:teuthology.orchestra.run.smithi035.stdout:Installed size: 8.2 M 2023-01-24T14:41:48.811 INFO:teuthology.orchestra.run.smithi035.stdout:Downloading Packages: 2023-01-24T14:41:48.968 INFO:teuthology.orchestra.run.smithi035.stdout:(1/6): ceph-grafana-dashboards-18.0.0-1954.g1a6 144 kB/s | 23 kB 00:00 2023-01-24T14:41:48.985 INFO:teuthology.orchestra.run.smithi035.stdout:(2/6): ceph-prometheus-alerts-18.0.0-1954.g1a67 82 kB/s | 15 kB 00:00 2023-01-24T14:41:49.027 INFO:teuthology.orchestra.run.smithi035.stdout:(3/6): python3-jwt-1.6.1-2.el8.noarch.rpm 725 kB/s | 43 kB 00:00 2023-01-24T14:41:49.086 INFO:teuthology.orchestra.run.smithi035.stdout:(4/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 335 kB/s | 34 kB 00:00 2023-01-24T14:41:49.211 INFO:teuthology.orchestra.run.smithi035.stdout:(5/6): ceph-mgr-dashboard-18.0.0-1954.g1a671c19 4.3 MB/s | 1.7 MB 00:00 2023-01-24T14:41:49.236 INFO:teuthology.orchestra.run.smithi035.stdout:(6/6): python3-routes-2.4.1-12.el8.noarch.rpm 942 kB/s | 196 kB 00:00 2023-01-24T14:41:49.238 INFO:teuthology.orchestra.run.smithi035.stdout:-------------------------------------------------------------------------------- 2023-01-24T14:41:49.239 INFO:teuthology.orchestra.run.smithi035.stdout:Total 4.7 MB/s | 2.0 MB 00:00 2023-01-24T14:41:49.249 INFO:teuthology.orchestra.run.smithi035.stdout:Running transaction check 2023-01-24T14:41:49.258 INFO:teuthology.orchestra.run.smithi035.stdout:Transaction check succeeded. 2023-01-24T14:41:49.258 INFO:teuthology.orchestra.run.smithi035.stdout:Running transaction test 2023-01-24T14:41:49.346 INFO:teuthology.orchestra.run.smithi035.stdout:Transaction test succeeded. 2023-01-24T14:41:49.349 INFO:teuthology.orchestra.run.smithi035.stdout:Running transaction 2023-01-24T14:41:49.622 INFO:teuthology.orchestra.run.smithi035.stdout: Preparing : 1/1 2023-01-24T14:41:49.889 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2023-01-24T14:41:50.108 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2023-01-24T14:41:50.317 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2023-01-24T14:41:50.454 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-1954.g1a671c19.el8.n 4/6 2023-01-24T14:41:50.883 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-1954.g1a671c19.el8. 5/6 2023-01-24T14:41:51.129 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-1954.g1a671c19.el8.noarc 6/6 2023-01-24T14:41:51.445 INFO:teuthology.orchestra.run.smithi035.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-1954.g1a671c19.el8.noarc 6/6 2023-01-24T14:41:51.445 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-1954.g1a671c19.el8. 1/6 2023-01-24T14:41:51.445 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-1954.g1a671c19.el8.noarc 2/6 2023-01-24T14:41:51.445 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-1954.g1a671c19.el8.n 3/6 2023-01-24T14:41:51.446 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2023-01-24T14:41:51.446 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2023-01-24T14:41:51.950 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2023-01-24T14:41:51.950 INFO:teuthology.orchestra.run.smithi035.stdout: 2023-01-24T14:41:51.951 INFO:teuthology.orchestra.run.smithi035.stdout:Installed: 2023-01-24T14:41:51.951 INFO:teuthology.orchestra.run.smithi035.stdout: ceph-grafana-dashboards-2:18.0.0-1954.g1a671c19.el8.noarch 2023-01-24T14:41:51.951 INFO:teuthology.orchestra.run.smithi035.stdout: ceph-mgr-dashboard-2:18.0.0-1954.g1a671c19.el8.noarch 2023-01-24T14:41:51.951 INFO:teuthology.orchestra.run.smithi035.stdout: ceph-prometheus-alerts-2:18.0.0-1954.g1a671c19.el8.noarch 2023-01-24T14:41:51.951 INFO:teuthology.orchestra.run.smithi035.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-01-24T14:41:51.952 INFO:teuthology.orchestra.run.smithi035.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-01-24T14:41:51.952 INFO:teuthology.orchestra.run.smithi035.stdout: python3-routes-2.4.1-12.el8.noarch 2023-01-24T14:41:51.952 INFO:teuthology.orchestra.run.smithi035.stdout: 2023-01-24T14:41:51.952 INFO:teuthology.orchestra.run.smithi035.stdout:Complete! 2023-01-24T14:41:52.050 DEBUG:teuthology.orchestra.run.smithi035:> sudo yum -y install ceph-mgr-diskprediction-local 2023-01-24T14:41:52.803 INFO:teuthology.orchestra.run.smithi035.stdout:Last metadata expiration check: 0:13:43 ago on Tue 24 Jan 2023 02:28:09 PM UTC. 2023-01-24T14:41:54.027 INFO:teuthology.orchestra.run.smithi035.stdout:Dependencies resolved. 2023-01-24T14:41:54.028 INFO:teuthology.orchestra.run.smithi035.stdout:===================================================================================================== 2023-01-24T14:41:54.028 INFO:teuthology.orchestra.run.smithi035.stdout: Package Arch Version Repository Size 2023-01-24T14:41:54.028 INFO:teuthology.orchestra.run.smithi035.stdout:===================================================================================================== 2023-01-24T14:41:54.029 INFO:teuthology.orchestra.run.smithi035.stdout:Installing: 2023-01-24T14:41:54.029 INFO:teuthology.orchestra.run.smithi035.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-1954.g1a671c19.el8 ceph-noarch 7.4 M 2023-01-24T14:41:54.029 INFO:teuthology.orchestra.run.smithi035.stdout:Installing dependencies: 2023-01-24T14:41:54.030 INFO:teuthology.orchestra.run.smithi035.stdout: libgfortran x86_64 8.5.0-18.el8 CentOS-BaseOS 645 k 2023-01-24T14:41:54.030 INFO:teuthology.orchestra.run.smithi035.stdout: libquadmath x86_64 8.5.0-18.el8 CentOS-BaseOS 172 k 2023-01-24T14:41:54.030 INFO:teuthology.orchestra.run.smithi035.stdout: openblas x86_64 0.3.15-4.el8 CentOS-AppStream 4.7 M 2023-01-24T14:41:54.030 INFO:teuthology.orchestra.run.smithi035.stdout: openblas-threads x86_64 0.3.15-4.el8 CentOS-AppStream 4.9 M 2023-01-24T14:41:54.030 INFO:teuthology.orchestra.run.smithi035.stdout: python3-numpy x86_64 1:1.14.3-10.el8 CentOS-AppStream 3.7 M 2023-01-24T14:41:54.031 INFO:teuthology.orchestra.run.smithi035.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 CentOS-AppStream 225 k 2023-01-24T14:41:54.031 INFO:teuthology.orchestra.run.smithi035.stdout: python3-scipy x86_64 1.0.0-21.module_el8.5.0+771+e5d9a225 CentOS-AppStream 14 M 2023-01-24T14:41:54.031 INFO:teuthology.orchestra.run.smithi035.stdout: 2023-01-24T14:41:54.031 INFO:teuthology.orchestra.run.smithi035.stdout:Transaction Summary 2023-01-24T14:41:54.032 INFO:teuthology.orchestra.run.smithi035.stdout:===================================================================================================== 2023-01-24T14:41:54.032 INFO:teuthology.orchestra.run.smithi035.stdout:Install 8 Packages 2023-01-24T14:41:54.032 INFO:teuthology.orchestra.run.smithi035.stdout: 2023-01-24T14:41:54.033 INFO:teuthology.orchestra.run.smithi035.stdout:Total download size: 35 M 2023-01-24T14:41:54.033 INFO:teuthology.orchestra.run.smithi035.stdout:Installed size: 210 M 2023-01-24T14:41:54.033 INFO:teuthology.orchestra.run.smithi035.stdout:Downloading Packages: 2023-01-24T14:41:54.192 INFO:teuthology.orchestra.run.smithi035.stdout:(1/8): openblas-0.3.15-4.el8.x86_64.rpm 29 MB/s | 4.7 MB 00:00 2023-01-24T14:41:54.351 INFO:teuthology.orchestra.run.smithi035.stdout:(2/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 24 MB/s | 3.7 MB 00:00 2023-01-24T14:41:54.384 INFO:teuthology.orchestra.run.smithi035.stdout:(3/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 6.7 MB/s | 225 kB 00:00 2023-01-24T14:41:54.493 INFO:teuthology.orchestra.run.smithi035.stdout:(4/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 11 MB/s | 4.9 MB 00:00 2023-01-24T14:41:54.552 INFO:teuthology.orchestra.run.smithi035.stdout:(5/8): libgfortran-8.5.0-18.el8.x86_64.rpm 11 MB/s | 645 kB 00:00 2023-01-24T14:41:54.585 INFO:teuthology.orchestra.run.smithi035.stdout:(6/8): libquadmath-8.5.0-18.el8.x86_64.rpm 5.1 MB/s | 172 kB 00:00 2023-01-24T14:41:54.944 INFO:teuthology.orchestra.run.smithi035.stdout:(7/8): python3-scipy-1.0.0-21.module_el8.5.0+77 24 MB/s | 14 MB 00:00 2023-01-24T14:41:57.411 INFO:teuthology.orchestra.run.smithi035.stdout:(8/8): ceph-mgr-diskprediction-local-18.0.0-195 2.2 MB/s | 7.4 MB 00:03 2023-01-24T14:41:57.414 INFO:teuthology.orchestra.run.smithi035.stdout:-------------------------------------------------------------------------------- 2023-01-24T14:41:57.415 INFO:teuthology.orchestra.run.smithi035.stdout:Total 10 MB/s | 35 MB 00:03 2023-01-24T14:41:57.657 INFO:teuthology.orchestra.run.smithi035.stdout:Running transaction check 2023-01-24T14:41:57.676 INFO:teuthology.orchestra.run.smithi035.stdout:Transaction check succeeded. 2023-01-24T14:41:57.676 INFO:teuthology.orchestra.run.smithi035.stdout:Running transaction test 2023-01-24T14:41:58.117 INFO:teuthology.orchestra.run.smithi035.stdout:Transaction test succeeded. 2023-01-24T14:41:58.120 INFO:teuthology.orchestra.run.smithi035.stdout:Running transaction 2023-01-24T14:41:58.699 INFO:teuthology.orchestra.run.smithi035.stdout: Preparing : 1/1 2023-01-24T14:41:58.887 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : libquadmath-8.5.0-18.el8.x86_64 1/8 2023-01-24T14:41:59.019 INFO:teuthology.orchestra.run.smithi035.stdout: Running scriptlet: libquadmath-8.5.0-18.el8.x86_64 1/8 2023-01-24T14:41:59.179 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : libgfortran-8.5.0-18.el8.x86_64 2/8 2023-01-24T14:41:59.813 INFO:teuthology.orchestra.run.smithi035.stdout: Running scriptlet: libgfortran-8.5.0-18.el8.x86_64 2/8 2023-01-24T14:41:59.988 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2023-01-24T14:42:00.642 INFO:teuthology.orchestra.run.smithi035.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2023-01-24T14:42:00.796 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2023-01-24T14:42:01.534 INFO:teuthology.orchestra.run.smithi035.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2023-01-24T14:42:01.920 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2023-01-24T14:42:02.221 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-01-24T14:42:04.110 INFO:teuthology.orchestra.run.smithi035.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-01-24T14:42:05.415 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 7/8 2023-01-24T14:42:05.637 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-1954.g1a671c1 8/8 2023-01-24T14:42:05.983 INFO:teuthology.orchestra.run.smithi035.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-1954.g1a671c1 8/8 2023-01-24T14:42:05.984 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-1954.g1a671c1 1/8 2023-01-24T14:42:05.984 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2023-01-24T14:42:05.984 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2023-01-24T14:42:05.984 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2023-01-24T14:42:05.985 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2023-01-24T14:42:05.985 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 6/8 2023-01-24T14:42:05.985 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : libgfortran-8.5.0-18.el8.x86_64 7/8 2023-01-24T14:42:06.630 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : libquadmath-8.5.0-18.el8.x86_64 8/8 2023-01-24T14:42:06.631 INFO:teuthology.orchestra.run.smithi035.stdout: 2023-01-24T14:42:06.631 INFO:teuthology.orchestra.run.smithi035.stdout:Installed: 2023-01-24T14:42:06.632 INFO:teuthology.orchestra.run.smithi035.stdout: ceph-mgr-diskprediction-local-2:18.0.0-1954.g1a671c19.el8.noarch 2023-01-24T14:42:06.632 INFO:teuthology.orchestra.run.smithi035.stdout: libgfortran-8.5.0-18.el8.x86_64 2023-01-24T14:42:06.632 INFO:teuthology.orchestra.run.smithi035.stdout: libquadmath-8.5.0-18.el8.x86_64 2023-01-24T14:42:06.633 INFO:teuthology.orchestra.run.smithi035.stdout: openblas-0.3.15-4.el8.x86_64 2023-01-24T14:42:06.633 INFO:teuthology.orchestra.run.smithi035.stdout: openblas-threads-0.3.15-4.el8.x86_64 2023-01-24T14:42:06.633 INFO:teuthology.orchestra.run.smithi035.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-01-24T14:42:06.633 INFO:teuthology.orchestra.run.smithi035.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-01-24T14:42:06.633 INFO:teuthology.orchestra.run.smithi035.stdout: python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x86_64 2023-01-24T14:42:06.634 INFO:teuthology.orchestra.run.smithi035.stdout: 2023-01-24T14:42:06.634 INFO:teuthology.orchestra.run.smithi035.stdout:Complete! 2023-01-24T14:42:06.745 DEBUG:teuthology.orchestra.run.smithi035:> sudo yum -y install ceph-mgr-rook 2023-01-24T14:42:07.505 INFO:teuthology.orchestra.run.smithi035.stdout:Last metadata expiration check: 0:13:58 ago on Tue 24 Jan 2023 02:28:09 PM UTC. 2023-01-24T14:42:08.722 INFO:teuthology.orchestra.run.smithi035.stdout:Dependencies resolved. 2023-01-24T14:42:08.724 INFO:teuthology.orchestra.run.smithi035.stdout:======================================================================================== 2023-01-24T14:42:08.724 INFO:teuthology.orchestra.run.smithi035.stdout: Package Arch Version Repository Size 2023-01-24T14:42:08.724 INFO:teuthology.orchestra.run.smithi035.stdout:======================================================================================== 2023-01-24T14:42:08.724 INFO:teuthology.orchestra.run.smithi035.stdout:Installing: 2023-01-24T14:42:08.725 INFO:teuthology.orchestra.run.smithi035.stdout: ceph-mgr-rook noarch 2:18.0.0-1954.g1a671c19.el8 ceph-noarch 47 k 2023-01-24T14:42:08.725 INFO:teuthology.orchestra.run.smithi035.stdout:Installing dependencies: 2023-01-24T14:42:08.725 INFO:teuthology.orchestra.run.smithi035.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-01-24T14:42:08.725 INFO:teuthology.orchestra.run.smithi035.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-01-24T14:42:08.725 INFO:teuthology.orchestra.run.smithi035.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-01-24T14:42:08.726 INFO:teuthology.orchestra.run.smithi035.stdout: python3-jsonpatch noarch 1.21-2.el8 CentOS-AppStream 27 k 2023-01-24T14:42:08.726 INFO:teuthology.orchestra.run.smithi035.stdout: python3-jsonpointer noarch 1.10-11.el8 CentOS-AppStream 20 k 2023-01-24T14:42:08.726 INFO:teuthology.orchestra.run.smithi035.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-01-24T14:42:08.726 INFO:teuthology.orchestra.run.smithi035.stdout: python3-oauthlib noarch 2.1.0-1.el8 CentOS-BaseOS 155 k 2023-01-24T14:42:08.726 INFO:teuthology.orchestra.run.smithi035.stdout: python3-pyasn1 noarch 0.3.7-6.el8 CentOS-AppStream 126 k 2023-01-24T14:42:08.727 INFO:teuthology.orchestra.run.smithi035.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 CentOS-AppStream 110 k 2023-01-24T14:42:08.727 INFO:teuthology.orchestra.run.smithi035.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 CentOS-BaseOS 43 k 2023-01-24T14:42:08.727 INFO:teuthology.orchestra.run.smithi035.stdout: python3-rsa noarch 4.8-1.el8 epel 61 k 2023-01-24T14:42:08.727 INFO:teuthology.orchestra.run.smithi035.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-01-24T14:42:08.728 INFO:teuthology.orchestra.run.smithi035.stdout: 2023-01-24T14:42:08.728 INFO:teuthology.orchestra.run.smithi035.stdout:Transaction Summary 2023-01-24T14:42:08.728 INFO:teuthology.orchestra.run.smithi035.stdout:======================================================================================== 2023-01-24T14:42:08.728 INFO:teuthology.orchestra.run.smithi035.stdout:Install 13 Packages 2023-01-24T14:42:08.728 INFO:teuthology.orchestra.run.smithi035.stdout: 2023-01-24T14:42:08.729 INFO:teuthology.orchestra.run.smithi035.stdout:Total download size: 1.9 M 2023-01-24T14:42:08.729 INFO:teuthology.orchestra.run.smithi035.stdout:Installed size: 25 M 2023-01-24T14:42:08.729 INFO:teuthology.orchestra.run.smithi035.stdout:Downloading Packages: 2023-01-24T14:42:08.811 INFO:teuthology.orchestra.run.smithi035.stdout:(1/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 318 kB/s | 27 kB 00:00 2023-01-24T14:42:08.846 INFO:teuthology.orchestra.run.smithi035.stdout:(2/13): python3-jsonpointer-1.10-11.el8.noarch. 170 kB/s | 20 kB 00:00 2023-01-24T14:42:08.870 INFO:teuthology.orchestra.run.smithi035.stdout:(3/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 2.3 MB/s | 126 kB 00:00 2023-01-24T14:42:08.896 INFO:teuthology.orchestra.run.smithi035.stdout:(4/13): python3-pyasn1-modules-0.3.7-6.el8.noar 2.1 MB/s | 110 kB 00:00 2023-01-24T14:42:08.921 INFO:teuthology.orchestra.run.smithi035.stdout:(5/13): python3-requests-oauthlib-1.0.0-1.el8.n 1.7 MB/s | 43 kB 00:00 2023-01-24T14:42:08.946 INFO:teuthology.orchestra.run.smithi035.stdout:(6/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 3.1 MB/s | 155 kB 00:00 2023-01-24T14:42:09.063 INFO:teuthology.orchestra.run.smithi035.stdout:(7/13): python3-cachetools-3.1.1-4.el8.noarch.r 232 kB/s | 33 kB 00:00 2023-01-24T14:42:09.089 INFO:teuthology.orchestra.run.smithi035.stdout:(8/13): python3-certifi-2018.10.15-7.el8.noarch 115 kB/s | 16 kB 00:00 2023-01-24T14:42:09.139 INFO:teuthology.orchestra.run.smithi035.stdout:(9/13): python3-google-auth-1.1.1-10.el8.noarch 1.1 MB/s | 81 kB 00:00 2023-01-24T14:42:09.206 INFO:teuthology.orchestra.run.smithi035.stdout:(10/13): ceph-mgr-rook-18.0.0-1954.g1a671c19.el 99 kB/s | 47 kB 00:00 2023-01-24T14:42:09.232 INFO:teuthology.orchestra.run.smithi035.stdout:(11/13): python3-rsa-4.8-1.el8.noarch.rpm 668 kB/s | 61 kB 00:00 2023-01-24T14:42:09.290 INFO:teuthology.orchestra.run.smithi035.stdout:(12/13): python3-websocket-client-0.56.0-5.el8. 730 kB/s | 61 kB 00:00 2023-01-24T14:42:09.657 INFO:teuthology.orchestra.run.smithi035.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 2.1 MB/s | 1.2 MB 00:00 2023-01-24T14:42:09.659 INFO:teuthology.orchestra.run.smithi035.stdout:-------------------------------------------------------------------------------- 2023-01-24T14:42:09.659 INFO:teuthology.orchestra.run.smithi035.stdout:Total 2.1 MB/s | 1.9 MB 00:00 2023-01-24T14:42:09.715 INFO:teuthology.orchestra.run.smithi035.stdout:Running transaction check 2023-01-24T14:42:09.731 INFO:teuthology.orchestra.run.smithi035.stdout:Transaction check succeeded. 2023-01-24T14:42:09.732 INFO:teuthology.orchestra.run.smithi035.stdout:Running transaction test 2023-01-24T14:42:09.839 INFO:teuthology.orchestra.run.smithi035.stdout:Transaction test succeeded. 2023-01-24T14:42:09.843 INFO:teuthology.orchestra.run.smithi035.stdout:Running transaction 2023-01-24T14:42:10.353 INFO:teuthology.orchestra.run.smithi035.stdout: Preparing : 1/1 2023-01-24T14:42:10.659 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2023-01-24T14:42:10.865 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2023-01-24T14:42:11.116 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : python3-rsa-4.8-1.el8.noarch 3/13 2023-01-24T14:42:11.555 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2023-01-24T14:42:11.779 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2023-01-24T14:42:12.040 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2023-01-24T14:42:12.410 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2023-01-24T14:42:12.645 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2023-01-24T14:42:13.760 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2023-01-24T14:42:14.110 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2023-01-24T14:42:14.311 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2023-01-24T14:42:14.531 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2023-01-24T14:42:14.702 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : ceph-mgr-rook-2:18.0.0-1954.g1a671c19.el8.noarch 13/13 2023-01-24T14:42:15.021 INFO:teuthology.orchestra.run.smithi035.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-1954.g1a671c19.el8.noarch 13/13 2023-01-24T14:42:15.022 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : ceph-mgr-rook-2:18.0.0-1954.g1a671c19.el8.noarch 1/13 2023-01-24T14:42:15.022 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2023-01-24T14:42:15.022 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2023-01-24T14:42:15.022 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2023-01-24T14:42:15.023 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2023-01-24T14:42:15.023 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2023-01-24T14:42:15.023 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2023-01-24T14:42:15.023 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2023-01-24T14:42:15.023 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2023-01-24T14:42:15.023 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2023-01-24T14:42:15.024 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2023-01-24T14:42:15.024 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : python3-rsa-4.8-1.el8.noarch 12/13 2023-01-24T14:42:15.617 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2023-01-24T14:42:15.617 INFO:teuthology.orchestra.run.smithi035.stdout: 2023-01-24T14:42:15.617 INFO:teuthology.orchestra.run.smithi035.stdout:Installed: 2023-01-24T14:42:15.618 INFO:teuthology.orchestra.run.smithi035.stdout: ceph-mgr-rook-2:18.0.0-1954.g1a671c19.el8.noarch 2023-01-24T14:42:15.618 INFO:teuthology.orchestra.run.smithi035.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-01-24T14:42:15.618 INFO:teuthology.orchestra.run.smithi035.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-01-24T14:42:15.618 INFO:teuthology.orchestra.run.smithi035.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-01-24T14:42:15.619 INFO:teuthology.orchestra.run.smithi035.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-01-24T14:42:15.619 INFO:teuthology.orchestra.run.smithi035.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-01-24T14:42:15.619 INFO:teuthology.orchestra.run.smithi035.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-01-24T14:42:15.619 INFO:teuthology.orchestra.run.smithi035.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-01-24T14:42:15.619 INFO:teuthology.orchestra.run.smithi035.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-01-24T14:42:15.619 INFO:teuthology.orchestra.run.smithi035.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-01-24T14:42:15.620 INFO:teuthology.orchestra.run.smithi035.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-01-24T14:42:15.620 INFO:teuthology.orchestra.run.smithi035.stdout: python3-rsa-4.8-1.el8.noarch 2023-01-24T14:42:15.620 INFO:teuthology.orchestra.run.smithi035.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-01-24T14:42:15.620 INFO:teuthology.orchestra.run.smithi035.stdout: 2023-01-24T14:42:15.620 INFO:teuthology.orchestra.run.smithi035.stdout:Complete! 2023-01-24T14:42:15.766 DEBUG:teuthology.orchestra.run.smithi035:> sudo yum -y install ceph-mgr-cephadm 2023-01-24T14:42:16.540 INFO:teuthology.orchestra.run.smithi035.stdout:Last metadata expiration check: 0:14:07 ago on Tue 24 Jan 2023 02:28:09 PM UTC. 2023-01-24T14:42:17.759 INFO:teuthology.orchestra.run.smithi035.stdout:Dependencies resolved. 2023-01-24T14:42:17.760 INFO:teuthology.orchestra.run.smithi035.stdout:================================================================================ 2023-01-24T14:42:17.760 INFO:teuthology.orchestra.run.smithi035.stdout: Package Arch Version Repository Size 2023-01-24T14:42:17.760 INFO:teuthology.orchestra.run.smithi035.stdout:================================================================================ 2023-01-24T14:42:17.760 INFO:teuthology.orchestra.run.smithi035.stdout:Installing: 2023-01-24T14:42:17.761 INFO:teuthology.orchestra.run.smithi035.stdout: ceph-mgr-cephadm noarch 2:18.0.0-1954.g1a671c19.el8 ceph-noarch 120 k 2023-01-24T14:42:17.761 INFO:teuthology.orchestra.run.smithi035.stdout:Installing dependencies: 2023-01-24T14:42:17.761 INFO:teuthology.orchestra.run.smithi035.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-01-24T14:42:17.761 INFO:teuthology.orchestra.run.smithi035.stdout: python3-babel noarch 2.5.1-7.el8 CentOS-AppStream 4.8 M 2023-01-24T14:42:17.762 INFO:teuthology.orchestra.run.smithi035.stdout: python3-jinja2 noarch 2.10.1-3.el8 CentOS-AppStream 538 k 2023-01-24T14:42:17.762 INFO:teuthology.orchestra.run.smithi035.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-01-24T14:42:17.762 INFO:teuthology.orchestra.run.smithi035.stdout: 2023-01-24T14:42:17.762 INFO:teuthology.orchestra.run.smithi035.stdout:Transaction Summary 2023-01-24T14:42:17.762 INFO:teuthology.orchestra.run.smithi035.stdout:================================================================================ 2023-01-24T14:42:17.763 INFO:teuthology.orchestra.run.smithi035.stdout:Install 5 Packages 2023-01-24T14:42:17.763 INFO:teuthology.orchestra.run.smithi035.stdout: 2023-01-24T14:42:17.763 INFO:teuthology.orchestra.run.smithi035.stdout:Total download size: 6.0 M 2023-01-24T14:42:17.764 INFO:teuthology.orchestra.run.smithi035.stdout:Installed size: 26 M 2023-01-24T14:42:17.764 INFO:teuthology.orchestra.run.smithi035.stdout:Downloading Packages: 2023-01-24T14:42:17.841 INFO:teuthology.orchestra.run.smithi035.stdout:(1/5): python3-jinja2-2.10.1-3.el8.noarch.rpm 6.7 MB/s | 538 kB 00:00 2023-01-24T14:42:17.966 INFO:teuthology.orchestra.run.smithi035.stdout:(2/5): python3-babel-2.5.1-7.el8.noarch.rpm 23 MB/s | 4.8 MB 00:00 2023-01-24T14:42:18.100 INFO:teuthology.orchestra.run.smithi035.stdout:(3/5): python3-natsort-7.1.1-2.el8.noarch.rpm 452 kB/s | 60 kB 00:00 2023-01-24T14:42:18.217 INFO:teuthology.orchestra.run.smithi035.stdout:(4/5): python3-asyncssh-2.7.0-2.el8.noarch.rpm 1.2 MB/s | 476 kB 00:00 2023-01-24T14:42:18.251 INFO:teuthology.orchestra.run.smithi035.stdout:(5/5): ceph-mgr-cephadm-18.0.0-1954.g1a671c19.e 246 kB/s | 120 kB 00:00 2023-01-24T14:42:18.253 INFO:teuthology.orchestra.run.smithi035.stdout:-------------------------------------------------------------------------------- 2023-01-24T14:42:18.253 INFO:teuthology.orchestra.run.smithi035.stdout:Total 12 MB/s | 6.0 MB 00:00 2023-01-24T14:42:18.307 INFO:teuthology.orchestra.run.smithi035.stdout:Running transaction check 2023-01-24T14:42:18.318 INFO:teuthology.orchestra.run.smithi035.stdout:Transaction check succeeded. 2023-01-24T14:42:18.319 INFO:teuthology.orchestra.run.smithi035.stdout:Running transaction test 2023-01-24T14:42:18.455 INFO:teuthology.orchestra.run.smithi035.stdout:Transaction test succeeded. 2023-01-24T14:42:18.458 INFO:teuthology.orchestra.run.smithi035.stdout:Running transaction 2023-01-24T14:42:18.752 INFO:teuthology.orchestra.run.smithi035.stdout: Preparing : 1/1 2023-01-24T14:42:19.118 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 1/5 2023-01-24T14:42:20.160 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 2/5 2023-01-24T14:42:20.574 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 3/5 2023-01-24T14:42:20.858 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 4/5 2023-01-24T14:42:21.076 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-1954.g1a671c19.el8.noarch 5/5 2023-01-24T14:42:21.395 INFO:teuthology.orchestra.run.smithi035.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-1954.g1a671c19.el8.noarch 5/5 2023-01-24T14:42:21.395 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-1954.g1a671c19.el8.noarch 1/5 2023-01-24T14:42:21.396 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/5 2023-01-24T14:42:21.396 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 3/5 2023-01-24T14:42:21.396 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 4/5 2023-01-24T14:42:21.945 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 5/5 2023-01-24T14:42:21.945 INFO:teuthology.orchestra.run.smithi035.stdout: 2023-01-24T14:42:21.946 INFO:teuthology.orchestra.run.smithi035.stdout:Installed: 2023-01-24T14:42:21.946 INFO:teuthology.orchestra.run.smithi035.stdout: ceph-mgr-cephadm-2:18.0.0-1954.g1a671c19.el8.noarch 2023-01-24T14:42:21.946 INFO:teuthology.orchestra.run.smithi035.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-01-24T14:42:21.946 INFO:teuthology.orchestra.run.smithi035.stdout: python3-babel-2.5.1-7.el8.noarch 2023-01-24T14:42:21.946 INFO:teuthology.orchestra.run.smithi035.stdout: python3-jinja2-2.10.1-3.el8.noarch 2023-01-24T14:42:21.947 INFO:teuthology.orchestra.run.smithi035.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-01-24T14:42:21.947 INFO:teuthology.orchestra.run.smithi035.stdout: 2023-01-24T14:42:21.947 INFO:teuthology.orchestra.run.smithi035.stdout:Complete! 2023-01-24T14:42:22.273 DEBUG:teuthology.orchestra.run.smithi035:> sudo yum -y install ceph-fuse 2023-01-24T14:42:23.033 INFO:teuthology.orchestra.run.smithi035.stdout:Last metadata expiration check: 0:14:14 ago on Tue 24 Jan 2023 02:28:09 PM UTC. 2023-01-24T14:42:24.264 INFO:teuthology.orchestra.run.smithi035.stdout:Dependencies resolved. 2023-01-24T14:42:24.265 INFO:teuthology.orchestra.run.smithi035.stdout:================================================================================ 2023-01-24T14:42:24.265 INFO:teuthology.orchestra.run.smithi035.stdout: Package Arch Version Repository Size 2023-01-24T14:42:24.265 INFO:teuthology.orchestra.run.smithi035.stdout:================================================================================ 2023-01-24T14:42:24.266 INFO:teuthology.orchestra.run.smithi035.stdout:Installing: 2023-01-24T14:42:24.266 INFO:teuthology.orchestra.run.smithi035.stdout: ceph-fuse x86_64 2:18.0.0-1954.g1a671c19.el8 ceph 814 k 2023-01-24T14:42:24.266 INFO:teuthology.orchestra.run.smithi035.stdout:Installing dependencies: 2023-01-24T14:42:24.266 INFO:teuthology.orchestra.run.smithi035.stdout: fuse x86_64 2.9.7-16.el8 CentOS-BaseOS 83 k 2023-01-24T14:42:24.266 INFO:teuthology.orchestra.run.smithi035.stdout: 2023-01-24T14:42:24.267 INFO:teuthology.orchestra.run.smithi035.stdout:Transaction Summary 2023-01-24T14:42:24.267 INFO:teuthology.orchestra.run.smithi035.stdout:================================================================================ 2023-01-24T14:42:24.267 INFO:teuthology.orchestra.run.smithi035.stdout:Install 2 Packages 2023-01-24T14:42:24.267 INFO:teuthology.orchestra.run.smithi035.stdout: 2023-01-24T14:42:24.268 INFO:teuthology.orchestra.run.smithi035.stdout:Total download size: 897 k 2023-01-24T14:42:24.268 INFO:teuthology.orchestra.run.smithi035.stdout:Installed size: 3.3 M 2023-01-24T14:42:24.268 INFO:teuthology.orchestra.run.smithi035.stdout:Downloading Packages: 2023-01-24T14:42:24.343 INFO:teuthology.orchestra.run.smithi035.stdout:(1/2): fuse-2.9.7-16.el8.x86_64.rpm 1.1 MB/s | 83 kB 00:00 2023-01-24T14:42:24.926 INFO:teuthology.orchestra.run.smithi035.stdout:(2/2): ceph-fuse-18.0.0-1954.g1a671c19.el8.x86_ 1.2 MB/s | 814 kB 00:00 2023-01-24T14:42:24.927 INFO:teuthology.orchestra.run.smithi035.stdout:-------------------------------------------------------------------------------- 2023-01-24T14:42:24.928 INFO:teuthology.orchestra.run.smithi035.stdout:Total 1.3 MB/s | 897 kB 00:00 2023-01-24T14:42:24.939 INFO:teuthology.orchestra.run.smithi035.stdout:Running transaction check 2023-01-24T14:42:24.961 INFO:teuthology.orchestra.run.smithi035.stdout:Transaction check succeeded. 2023-01-24T14:42:24.962 INFO:teuthology.orchestra.run.smithi035.stdout:Running transaction test 2023-01-24T14:42:25.064 INFO:teuthology.orchestra.run.smithi035.stdout:Transaction test succeeded. 2023-01-24T14:42:25.068 INFO:teuthology.orchestra.run.smithi035.stdout:Running transaction 2023-01-24T14:42:25.286 INFO:teuthology.orchestra.run.smithi035.stdout: Preparing : 1/1 2023-01-24T14:42:25.548 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : fuse-2.9.7-16.el8.x86_64 1/2 2023-01-24T14:42:25.784 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : ceph-fuse-2:18.0.0-1954.g1a671c19.el8.x86_64 2/2 2023-01-24T14:42:26.225 INFO:teuthology.orchestra.run.smithi035.stdout: Running scriptlet: ceph-fuse-2:18.0.0-1954.g1a671c19.el8.x86_64 2/2 2023-01-24T14:42:26.225 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : ceph-fuse-2:18.0.0-1954.g1a671c19.el8.x86_64 1/2 2023-01-24T14:42:26.746 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : fuse-2.9.7-16.el8.x86_64 2/2 2023-01-24T14:42:26.747 INFO:teuthology.orchestra.run.smithi035.stdout: 2023-01-24T14:42:26.747 INFO:teuthology.orchestra.run.smithi035.stdout:Installed: 2023-01-24T14:42:26.747 INFO:teuthology.orchestra.run.smithi035.stdout: ceph-fuse-2:18.0.0-1954.g1a671c19.el8.x86_64 fuse-2.9.7-16.el8.x86_64 2023-01-24T14:42:26.747 INFO:teuthology.orchestra.run.smithi035.stdout: 2023-01-24T14:42:26.747 INFO:teuthology.orchestra.run.smithi035.stdout:Complete! 2023-01-24T14:42:26.982 DEBUG:teuthology.orchestra.run.smithi035:> sudo yum -y install ceph-volume 2023-01-24T14:42:27.755 INFO:teuthology.orchestra.run.smithi035.stdout:Last metadata expiration check: 0:14:18 ago on Tue 24 Jan 2023 02:28:09 PM UTC. 2023-01-24T14:42:28.998 INFO:teuthology.orchestra.run.smithi035.stdout:Dependencies resolved. 2023-01-24T14:42:28.999 INFO:teuthology.orchestra.run.smithi035.stdout:================================================================================ 2023-01-24T14:42:28.999 INFO:teuthology.orchestra.run.smithi035.stdout: Package Arch Version Repository Size 2023-01-24T14:42:29.000 INFO:teuthology.orchestra.run.smithi035.stdout:================================================================================ 2023-01-24T14:42:29.000 INFO:teuthology.orchestra.run.smithi035.stdout:Installing: 2023-01-24T14:42:29.001 INFO:teuthology.orchestra.run.smithi035.stdout: ceph-volume noarch 2:18.0.0-1954.g1a671c19.el8 ceph-noarch 263 k 2023-01-24T14:42:29.001 INFO:teuthology.orchestra.run.smithi035.stdout: 2023-01-24T14:42:29.001 INFO:teuthology.orchestra.run.smithi035.stdout:Transaction Summary 2023-01-24T14:42:29.002 INFO:teuthology.orchestra.run.smithi035.stdout:================================================================================ 2023-01-24T14:42:29.002 INFO:teuthology.orchestra.run.smithi035.stdout:Install 1 Package 2023-01-24T14:42:29.003 INFO:teuthology.orchestra.run.smithi035.stdout: 2023-01-24T14:42:29.003 INFO:teuthology.orchestra.run.smithi035.stdout:Total download size: 263 k 2023-01-24T14:42:29.003 INFO:teuthology.orchestra.run.smithi035.stdout:Installed size: 1.3 M 2023-01-24T14:42:29.004 INFO:teuthology.orchestra.run.smithi035.stdout:Downloading Packages: 2023-01-24T14:42:29.597 INFO:teuthology.orchestra.run.smithi035.stdout:ceph-volume-18.0.0-1954.g1a671c19.el8.noarch.rp 440 kB/s | 263 kB 00:00 2023-01-24T14:42:29.598 INFO:teuthology.orchestra.run.smithi035.stdout:-------------------------------------------------------------------------------- 2023-01-24T14:42:29.598 INFO:teuthology.orchestra.run.smithi035.stdout:Total 439 kB/s | 263 kB 00:00 2023-01-24T14:42:29.599 INFO:teuthology.orchestra.run.smithi035.stdout:Running transaction check 2023-01-24T14:42:29.610 INFO:teuthology.orchestra.run.smithi035.stdout:Transaction check succeeded. 2023-01-24T14:42:29.610 INFO:teuthology.orchestra.run.smithi035.stdout:Running transaction test 2023-01-24T14:42:29.715 INFO:teuthology.orchestra.run.smithi035.stdout:Transaction test succeeded. 2023-01-24T14:42:29.718 INFO:teuthology.orchestra.run.smithi035.stdout:Running transaction 2023-01-24T14:42:29.917 INFO:teuthology.orchestra.run.smithi035.stdout: Preparing : 1/1 2023-01-24T14:42:30.209 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : ceph-volume-2:18.0.0-1954.g1a671c19.el8.noarch 1/1 2023-01-24T14:42:30.659 INFO:teuthology.orchestra.run.smithi035.stdout: Running scriptlet: ceph-volume-2:18.0.0-1954.g1a671c19.el8.noarch 1/1 2023-01-24T14:42:31.153 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : ceph-volume-2:18.0.0-1954.g1a671c19.el8.noarch 1/1 2023-01-24T14:42:31.154 INFO:teuthology.orchestra.run.smithi035.stdout: 2023-01-24T14:42:31.154 INFO:teuthology.orchestra.run.smithi035.stdout:Installed: 2023-01-24T14:42:31.154 INFO:teuthology.orchestra.run.smithi035.stdout: ceph-volume-2:18.0.0-1954.g1a671c19.el8.noarch 2023-01-24T14:42:31.154 INFO:teuthology.orchestra.run.smithi035.stdout: 2023-01-24T14:42:31.154 INFO:teuthology.orchestra.run.smithi035.stdout:Complete! 2023-01-24T14:42:31.332 DEBUG:teuthology.orchestra.run.smithi035:> sudo yum -y install librados-devel 2023-01-24T14:42:32.109 INFO:teuthology.orchestra.run.smithi035.stdout:Last metadata expiration check: 0:14:23 ago on Tue 24 Jan 2023 02:28:09 PM UTC. 2023-01-24T14:42:33.341 INFO:teuthology.orchestra.run.smithi035.stdout:Dependencies resolved. 2023-01-24T14:42:33.341 INFO:teuthology.orchestra.run.smithi035.stdout:================================================================================ 2023-01-24T14:42:33.342 INFO:teuthology.orchestra.run.smithi035.stdout: Package Arch Version Repo Size 2023-01-24T14:42:33.342 INFO:teuthology.orchestra.run.smithi035.stdout:================================================================================ 2023-01-24T14:42:33.342 INFO:teuthology.orchestra.run.smithi035.stdout:Installing: 2023-01-24T14:42:33.342 INFO:teuthology.orchestra.run.smithi035.stdout: librados-devel x86_64 2:18.0.0-1954.g1a671c19.el8 ceph 139 k 2023-01-24T14:42:33.342 INFO:teuthology.orchestra.run.smithi035.stdout: 2023-01-24T14:42:33.342 INFO:teuthology.orchestra.run.smithi035.stdout:Transaction Summary 2023-01-24T14:42:33.343 INFO:teuthology.orchestra.run.smithi035.stdout:================================================================================ 2023-01-24T14:42:33.343 INFO:teuthology.orchestra.run.smithi035.stdout:Install 1 Package 2023-01-24T14:42:33.343 INFO:teuthology.orchestra.run.smithi035.stdout: 2023-01-24T14:42:33.344 INFO:teuthology.orchestra.run.smithi035.stdout:Total download size: 139 k 2023-01-24T14:42:33.344 INFO:teuthology.orchestra.run.smithi035.stdout:Installed size: 512 k 2023-01-24T14:42:33.344 INFO:teuthology.orchestra.run.smithi035.stdout:Downloading Packages: 2023-01-24T14:42:33.963 INFO:teuthology.orchestra.run.smithi035.stdout:librados-devel-18.0.0-1954.g1a671c19.el8.x86_64 223 kB/s | 139 kB 00:00 2023-01-24T14:42:33.964 INFO:teuthology.orchestra.run.smithi035.stdout:-------------------------------------------------------------------------------- 2023-01-24T14:42:33.964 INFO:teuthology.orchestra.run.smithi035.stdout:Total 223 kB/s | 139 kB 00:00 2023-01-24T14:42:33.965 INFO:teuthology.orchestra.run.smithi035.stdout:Running transaction check 2023-01-24T14:42:33.970 INFO:teuthology.orchestra.run.smithi035.stdout:Transaction check succeeded. 2023-01-24T14:42:33.971 INFO:teuthology.orchestra.run.smithi035.stdout:Running transaction test 2023-01-24T14:42:34.029 INFO:teuthology.orchestra.run.smithi035.stdout:Transaction test succeeded. 2023-01-24T14:42:34.032 INFO:teuthology.orchestra.run.smithi035.stdout:Running transaction 2023-01-24T14:42:34.159 INFO:teuthology.orchestra.run.smithi035.stdout: Preparing : 1/1 2023-01-24T14:42:34.365 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : librados-devel-2:18.0.0-1954.g1a671c19.el8.x86_64 1/1 2023-01-24T14:42:34.672 INFO:teuthology.orchestra.run.smithi035.stdout: Running scriptlet: librados-devel-2:18.0.0-1954.g1a671c19.el8.x86_64 1/1 2023-01-24T14:42:35.175 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : librados-devel-2:18.0.0-1954.g1a671c19.el8.x86_64 1/1 2023-01-24T14:42:35.175 INFO:teuthology.orchestra.run.smithi035.stdout: 2023-01-24T14:42:35.175 INFO:teuthology.orchestra.run.smithi035.stdout:Installed: 2023-01-24T14:42:35.175 INFO:teuthology.orchestra.run.smithi035.stdout: librados-devel-2:18.0.0-1954.g1a671c19.el8.x86_64 2023-01-24T14:42:35.176 INFO:teuthology.orchestra.run.smithi035.stdout: 2023-01-24T14:42:35.176 INFO:teuthology.orchestra.run.smithi035.stdout:Complete! 2023-01-24T14:42:35.400 DEBUG:teuthology.orchestra.run.smithi035:> sudo yum -y install libcephfs2 2023-01-24T14:42:36.160 INFO:teuthology.orchestra.run.smithi035.stdout:Last metadata expiration check: 0:14:27 ago on Tue 24 Jan 2023 02:28:09 PM UTC. 2023-01-24T14:42:37.269 INFO:teuthology.orchestra.run.smithi035.stdout:Package libcephfs2-2:18.0.0-1954.g1a671c19.el8.x86_64 is already installed. 2023-01-24T14:42:37.381 INFO:teuthology.orchestra.run.smithi035.stdout:Dependencies resolved. 2023-01-24T14:42:37.382 INFO:teuthology.orchestra.run.smithi035.stdout:Nothing to do. 2023-01-24T14:42:37.382 INFO:teuthology.orchestra.run.smithi035.stdout:Complete! 2023-01-24T14:42:37.475 DEBUG:teuthology.orchestra.run.smithi035:> sudo yum -y install libcephfs-devel 2023-01-24T14:42:38.244 INFO:teuthology.orchestra.run.smithi035.stdout:Last metadata expiration check: 0:14:29 ago on Tue 24 Jan 2023 02:28:09 PM UTC. 2023-01-24T14:42:39.464 INFO:teuthology.orchestra.run.smithi035.stdout:Dependencies resolved. 2023-01-24T14:42:39.465 INFO:teuthology.orchestra.run.smithi035.stdout:================================================================================ 2023-01-24T14:42:39.466 INFO:teuthology.orchestra.run.smithi035.stdout: Package Arch Version Repo Size 2023-01-24T14:42:39.466 INFO:teuthology.orchestra.run.smithi035.stdout:================================================================================ 2023-01-24T14:42:39.466 INFO:teuthology.orchestra.run.smithi035.stdout:Installing: 2023-01-24T14:42:39.466 INFO:teuthology.orchestra.run.smithi035.stdout: libcephfs-devel x86_64 2:18.0.0-1954.g1a671c19.el8 ceph 30 k 2023-01-24T14:42:39.466 INFO:teuthology.orchestra.run.smithi035.stdout: 2023-01-24T14:42:39.467 INFO:teuthology.orchestra.run.smithi035.stdout:Transaction Summary 2023-01-24T14:42:39.467 INFO:teuthology.orchestra.run.smithi035.stdout:================================================================================ 2023-01-24T14:42:39.467 INFO:teuthology.orchestra.run.smithi035.stdout:Install 1 Package 2023-01-24T14:42:39.467 INFO:teuthology.orchestra.run.smithi035.stdout: 2023-01-24T14:42:39.467 INFO:teuthology.orchestra.run.smithi035.stdout:Total download size: 30 k 2023-01-24T14:42:39.467 INFO:teuthology.orchestra.run.smithi035.stdout:Installed size: 137 k 2023-01-24T14:42:39.468 INFO:teuthology.orchestra.run.smithi035.stdout:Downloading Packages: 2023-01-24T14:42:39.777 INFO:teuthology.orchestra.run.smithi035.stdout:libcephfs-devel-18.0.0-1954.g1a671c19.el8.x86_6 98 kB/s | 30 kB 00:00 2023-01-24T14:42:39.778 INFO:teuthology.orchestra.run.smithi035.stdout:-------------------------------------------------------------------------------- 2023-01-24T14:42:39.778 INFO:teuthology.orchestra.run.smithi035.stdout:Total 98 kB/s | 30 kB 00:00 2023-01-24T14:42:39.778 INFO:teuthology.orchestra.run.smithi035.stdout:Running transaction check 2023-01-24T14:42:39.782 INFO:teuthology.orchestra.run.smithi035.stdout:Transaction check succeeded. 2023-01-24T14:42:39.782 INFO:teuthology.orchestra.run.smithi035.stdout:Running transaction test 2023-01-24T14:42:39.799 INFO:teuthology.orchestra.run.smithi035.stdout:Transaction test succeeded. 2023-01-24T14:42:39.803 INFO:teuthology.orchestra.run.smithi035.stdout:Running transaction 2023-01-24T14:42:39.868 INFO:teuthology.orchestra.run.smithi035.stdout: Preparing : 1/1 2023-01-24T14:42:40.131 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : libcephfs-devel-2:18.0.0-1954.g1a671c19.el8.x86_64 1/1 2023-01-24T14:42:40.422 INFO:teuthology.orchestra.run.smithi035.stdout: Running scriptlet: libcephfs-devel-2:18.0.0-1954.g1a671c19.el8.x86_64 1/1 2023-01-24T14:42:40.901 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : libcephfs-devel-2:18.0.0-1954.g1a671c19.el8.x86_64 1/1 2023-01-24T14:42:40.901 INFO:teuthology.orchestra.run.smithi035.stdout: 2023-01-24T14:42:40.901 INFO:teuthology.orchestra.run.smithi035.stdout:Installed: 2023-01-24T14:42:40.902 INFO:teuthology.orchestra.run.smithi035.stdout: libcephfs-devel-2:18.0.0-1954.g1a671c19.el8.x86_64 2023-01-24T14:42:40.902 INFO:teuthology.orchestra.run.smithi035.stdout: 2023-01-24T14:42:40.902 INFO:teuthology.orchestra.run.smithi035.stdout:Complete! 2023-01-24T14:42:41.008 DEBUG:teuthology.orchestra.run.smithi035:> sudo yum -y install librados2 2023-01-24T14:42:41.761 INFO:teuthology.orchestra.run.smithi035.stdout:Last metadata expiration check: 0:14:32 ago on Tue 24 Jan 2023 02:28:09 PM UTC. 2023-01-24T14:42:42.872 INFO:teuthology.orchestra.run.smithi035.stdout:Package librados2-2:18.0.0-1954.g1a671c19.el8.x86_64 is already installed. 2023-01-24T14:42:42.986 INFO:teuthology.orchestra.run.smithi035.stdout:Dependencies resolved. 2023-01-24T14:42:42.987 INFO:teuthology.orchestra.run.smithi035.stdout:Nothing to do. 2023-01-24T14:42:42.987 INFO:teuthology.orchestra.run.smithi035.stdout:Complete! 2023-01-24T14:42:43.078 DEBUG:teuthology.orchestra.run.smithi035:> sudo yum -y install librbd1 2023-01-24T14:42:43.840 INFO:teuthology.orchestra.run.smithi035.stdout:Last metadata expiration check: 0:14:34 ago on Tue 24 Jan 2023 02:28:09 PM UTC. 2023-01-24T14:42:44.951 INFO:teuthology.orchestra.run.smithi035.stdout:Package librbd1-2:18.0.0-1954.g1a671c19.el8.x86_64 is already installed. 2023-01-24T14:42:45.066 INFO:teuthology.orchestra.run.smithi035.stdout:Dependencies resolved. 2023-01-24T14:42:45.067 INFO:teuthology.orchestra.run.smithi035.stdout:Nothing to do. 2023-01-24T14:42:45.067 INFO:teuthology.orchestra.run.smithi035.stdout:Complete! 2023-01-24T14:42:45.154 DEBUG:teuthology.orchestra.run.smithi035:> sudo yum -y install python3-rados 2023-01-24T14:42:45.925 INFO:teuthology.orchestra.run.smithi035.stdout:Last metadata expiration check: 0:14:36 ago on Tue 24 Jan 2023 02:28:09 PM UTC. 2023-01-24T14:42:47.033 INFO:teuthology.orchestra.run.smithi035.stdout:Package python3-rados-2:18.0.0-1954.g1a671c19.el8.x86_64 is already installed. 2023-01-24T14:42:47.145 INFO:teuthology.orchestra.run.smithi035.stdout:Dependencies resolved. 2023-01-24T14:42:47.148 INFO:teuthology.orchestra.run.smithi035.stdout:Nothing to do. 2023-01-24T14:42:47.149 INFO:teuthology.orchestra.run.smithi035.stdout:Complete! 2023-01-24T14:42:47.236 DEBUG:teuthology.orchestra.run.smithi035:> sudo yum -y install python3-rgw 2023-01-24T14:42:47.993 INFO:teuthology.orchestra.run.smithi035.stdout:Last metadata expiration check: 0:14:38 ago on Tue 24 Jan 2023 02:28:09 PM UTC. 2023-01-24T14:42:49.116 INFO:teuthology.orchestra.run.smithi035.stdout:Package python3-rgw-2:18.0.0-1954.g1a671c19.el8.x86_64 is already installed. 2023-01-24T14:42:49.227 INFO:teuthology.orchestra.run.smithi035.stdout:Dependencies resolved. 2023-01-24T14:42:49.228 INFO:teuthology.orchestra.run.smithi035.stdout:Nothing to do. 2023-01-24T14:42:49.229 INFO:teuthology.orchestra.run.smithi035.stdout:Complete! 2023-01-24T14:42:49.327 DEBUG:teuthology.orchestra.run.smithi035:> sudo yum -y install python3-cephfs 2023-01-24T14:42:50.096 INFO:teuthology.orchestra.run.smithi035.stdout:Last metadata expiration check: 0:14:41 ago on Tue 24 Jan 2023 02:28:09 PM UTC. 2023-01-24T14:42:51.209 INFO:teuthology.orchestra.run.smithi035.stdout:Package python3-cephfs-2:18.0.0-1954.g1a671c19.el8.x86_64 is already installed. 2023-01-24T14:42:51.318 INFO:teuthology.orchestra.run.smithi035.stdout:Dependencies resolved. 2023-01-24T14:42:51.319 INFO:teuthology.orchestra.run.smithi035.stdout:Nothing to do. 2023-01-24T14:42:51.319 INFO:teuthology.orchestra.run.smithi035.stdout:Complete! 2023-01-24T14:42:51.418 DEBUG:teuthology.orchestra.run.smithi035:> sudo yum -y install python3-rbd 2023-01-24T14:42:52.176 INFO:teuthology.orchestra.run.smithi035.stdout:Last metadata expiration check: 0:14:43 ago on Tue 24 Jan 2023 02:28:09 PM UTC. 2023-01-24T14:42:53.289 INFO:teuthology.orchestra.run.smithi035.stdout:Package python3-rbd-2:18.0.0-1954.g1a671c19.el8.x86_64 is already installed. 2023-01-24T14:42:53.402 INFO:teuthology.orchestra.run.smithi035.stdout:Dependencies resolved. 2023-01-24T14:42:53.403 INFO:teuthology.orchestra.run.smithi035.stdout:Nothing to do. 2023-01-24T14:42:53.404 INFO:teuthology.orchestra.run.smithi035.stdout:Complete! 2023-01-24T14:42:53.585 DEBUG:teuthology.orchestra.run.smithi035:> sudo yum -y install rbd-fuse 2023-01-24T14:42:54.339 INFO:teuthology.orchestra.run.smithi035.stdout:Last metadata expiration check: 0:14:45 ago on Tue 24 Jan 2023 02:28:09 PM UTC. 2023-01-24T14:42:55.563 INFO:teuthology.orchestra.run.smithi035.stdout:Dependencies resolved. 2023-01-24T14:42:55.564 INFO:teuthology.orchestra.run.smithi035.stdout:================================================================================ 2023-01-24T14:42:55.565 INFO:teuthology.orchestra.run.smithi035.stdout: Package Architecture Version Repository Size 2023-01-24T14:42:55.565 INFO:teuthology.orchestra.run.smithi035.stdout:================================================================================ 2023-01-24T14:42:55.565 INFO:teuthology.orchestra.run.smithi035.stdout:Installing: 2023-01-24T14:42:55.565 INFO:teuthology.orchestra.run.smithi035.stdout: rbd-fuse x86_64 2:18.0.0-1954.g1a671c19.el8 ceph 91 k 2023-01-24T14:42:55.565 INFO:teuthology.orchestra.run.smithi035.stdout: 2023-01-24T14:42:55.566 INFO:teuthology.orchestra.run.smithi035.stdout:Transaction Summary 2023-01-24T14:42:55.566 INFO:teuthology.orchestra.run.smithi035.stdout:================================================================================ 2023-01-24T14:42:55.566 INFO:teuthology.orchestra.run.smithi035.stdout:Install 1 Package 2023-01-24T14:42:55.566 INFO:teuthology.orchestra.run.smithi035.stdout: 2023-01-24T14:42:55.566 INFO:teuthology.orchestra.run.smithi035.stdout:Total download size: 91 k 2023-01-24T14:42:55.567 INFO:teuthology.orchestra.run.smithi035.stdout:Installed size: 303 k 2023-01-24T14:42:55.567 INFO:teuthology.orchestra.run.smithi035.stdout:Downloading Packages: 2023-01-24T14:42:55.793 INFO:teuthology.orchestra.run.smithi035.stdout:rbd-fuse-18.0.0-1954.g1a671c19.el8.x86_64.rpm 400 kB/s | 91 kB 00:00 2023-01-24T14:42:55.794 INFO:teuthology.orchestra.run.smithi035.stdout:-------------------------------------------------------------------------------- 2023-01-24T14:42:55.794 INFO:teuthology.orchestra.run.smithi035.stdout:Total 398 kB/s | 91 kB 00:00 2023-01-24T14:42:55.794 INFO:teuthology.orchestra.run.smithi035.stdout:Running transaction check 2023-01-24T14:42:55.804 INFO:teuthology.orchestra.run.smithi035.stdout:Transaction check succeeded. 2023-01-24T14:42:55.805 INFO:teuthology.orchestra.run.smithi035.stdout:Running transaction test 2023-01-24T14:42:55.862 INFO:teuthology.orchestra.run.smithi035.stdout:Transaction test succeeded. 2023-01-24T14:42:55.865 INFO:teuthology.orchestra.run.smithi035.stdout:Running transaction 2023-01-24T14:42:55.994 INFO:teuthology.orchestra.run.smithi035.stdout: Preparing : 1/1 2023-01-24T14:42:56.221 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : rbd-fuse-2:18.0.0-1954.g1a671c19.el8.x86_64 1/1 2023-01-24T14:42:56.536 INFO:teuthology.orchestra.run.smithi035.stdout: Running scriptlet: rbd-fuse-2:18.0.0-1954.g1a671c19.el8.x86_64 1/1 2023-01-24T14:42:57.053 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : rbd-fuse-2:18.0.0-1954.g1a671c19.el8.x86_64 1/1 2023-01-24T14:42:57.053 INFO:teuthology.orchestra.run.smithi035.stdout: 2023-01-24T14:42:57.053 INFO:teuthology.orchestra.run.smithi035.stdout:Installed: 2023-01-24T14:42:57.053 INFO:teuthology.orchestra.run.smithi035.stdout: rbd-fuse-2:18.0.0-1954.g1a671c19.el8.x86_64 2023-01-24T14:42:57.053 INFO:teuthology.orchestra.run.smithi035.stdout: 2023-01-24T14:42:57.054 INFO:teuthology.orchestra.run.smithi035.stdout:Complete! 2023-01-24T14:42:57.254 DEBUG:teuthology.orchestra.run.smithi035:> sudo yum -y install rbd-mirror 2023-01-24T14:42:58.017 INFO:teuthology.orchestra.run.smithi035.stdout:Last metadata expiration check: 0:14:49 ago on Tue 24 Jan 2023 02:28:09 PM UTC. 2023-01-24T14:42:59.236 INFO:teuthology.orchestra.run.smithi035.stdout:Dependencies resolved. 2023-01-24T14:42:59.237 INFO:teuthology.orchestra.run.smithi035.stdout:================================================================================ 2023-01-24T14:42:59.237 INFO:teuthology.orchestra.run.smithi035.stdout: Package Arch Version Repo Size 2023-01-24T14:42:59.238 INFO:teuthology.orchestra.run.smithi035.stdout:================================================================================ 2023-01-24T14:42:59.238 INFO:teuthology.orchestra.run.smithi035.stdout:Installing: 2023-01-24T14:42:59.238 INFO:teuthology.orchestra.run.smithi035.stdout: rbd-mirror x86_64 2:18.0.0-1954.g1a671c19.el8 ceph 3.3 M 2023-01-24T14:42:59.238 INFO:teuthology.orchestra.run.smithi035.stdout: 2023-01-24T14:42:59.238 INFO:teuthology.orchestra.run.smithi035.stdout:Transaction Summary 2023-01-24T14:42:59.239 INFO:teuthology.orchestra.run.smithi035.stdout:================================================================================ 2023-01-24T14:42:59.239 INFO:teuthology.orchestra.run.smithi035.stdout:Install 1 Package 2023-01-24T14:42:59.239 INFO:teuthology.orchestra.run.smithi035.stdout: 2023-01-24T14:42:59.239 INFO:teuthology.orchestra.run.smithi035.stdout:Total download size: 3.3 M 2023-01-24T14:42:59.239 INFO:teuthology.orchestra.run.smithi035.stdout:Installed size: 18 M 2023-01-24T14:42:59.240 INFO:teuthology.orchestra.run.smithi035.stdout:Downloading Packages: 2023-01-24T14:42:59.863 INFO:teuthology.orchestra.run.smithi035.stdout:rbd-mirror-18.0.0-1954.g1a671c19.el8.x86_64.rpm 5.3 MB/s | 3.3 MB 00:00 2023-01-24T14:42:59.864 INFO:teuthology.orchestra.run.smithi035.stdout:-------------------------------------------------------------------------------- 2023-01-24T14:42:59.865 INFO:teuthology.orchestra.run.smithi035.stdout:Total 5.3 MB/s | 3.3 MB 00:00 2023-01-24T14:42:59.865 INFO:teuthology.orchestra.run.smithi035.stdout:Running transaction check 2023-01-24T14:42:59.877 INFO:teuthology.orchestra.run.smithi035.stdout:Transaction check succeeded. 2023-01-24T14:42:59.877 INFO:teuthology.orchestra.run.smithi035.stdout:Running transaction test 2023-01-24T14:42:59.945 INFO:teuthology.orchestra.run.smithi035.stdout:Transaction test succeeded. 2023-01-24T14:42:59.948 INFO:teuthology.orchestra.run.smithi035.stdout:Running transaction 2023-01-24T14:43:00.424 INFO:teuthology.orchestra.run.smithi035.stdout: Preparing : 1/1 2023-01-24T14:43:00.605 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : rbd-mirror-2:18.0.0-1954.g1a671c19.el8.x86_64 1/1 2023-01-24T14:43:01.062 INFO:teuthology.orchestra.run.smithi035.stdout: Running scriptlet: rbd-mirror-2:18.0.0-1954.g1a671c19.el8.x86_64 1/1 2023-01-24T14:43:01.560 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : rbd-mirror-2:18.0.0-1954.g1a671c19.el8.x86_64 1/1 2023-01-24T14:43:01.561 INFO:teuthology.orchestra.run.smithi035.stdout: 2023-01-24T14:43:01.561 INFO:teuthology.orchestra.run.smithi035.stdout:Installed: 2023-01-24T14:43:01.561 INFO:teuthology.orchestra.run.smithi035.stdout: rbd-mirror-2:18.0.0-1954.g1a671c19.el8.x86_64 2023-01-24T14:43:01.561 INFO:teuthology.orchestra.run.smithi035.stdout: 2023-01-24T14:43:01.562 INFO:teuthology.orchestra.run.smithi035.stdout:Complete! 2023-01-24T14:43:01.790 DEBUG:teuthology.orchestra.run.smithi035:> sudo yum -y install rbd-nbd 2023-01-24T14:43:02.553 INFO:teuthology.orchestra.run.smithi035.stdout:Last metadata expiration check: 0:14:53 ago on Tue 24 Jan 2023 02:28:09 PM UTC. 2023-01-24T14:43:03.770 INFO:teuthology.orchestra.run.smithi035.stdout:Dependencies resolved. 2023-01-24T14:43:03.771 INFO:teuthology.orchestra.run.smithi035.stdout:================================================================================ 2023-01-24T14:43:03.771 INFO:teuthology.orchestra.run.smithi035.stdout: Package Architecture Version Repository Size 2023-01-24T14:43:03.772 INFO:teuthology.orchestra.run.smithi035.stdout:================================================================================ 2023-01-24T14:43:03.772 INFO:teuthology.orchestra.run.smithi035.stdout:Installing: 2023-01-24T14:43:03.772 INFO:teuthology.orchestra.run.smithi035.stdout: rbd-nbd x86_64 2:18.0.0-1954.g1a671c19.el8 ceph 177 k 2023-01-24T14:43:03.772 INFO:teuthology.orchestra.run.smithi035.stdout: 2023-01-24T14:43:03.773 INFO:teuthology.orchestra.run.smithi035.stdout:Transaction Summary 2023-01-24T14:43:03.773 INFO:teuthology.orchestra.run.smithi035.stdout:================================================================================ 2023-01-24T14:43:03.773 INFO:teuthology.orchestra.run.smithi035.stdout:Install 1 Package 2023-01-24T14:43:03.773 INFO:teuthology.orchestra.run.smithi035.stdout: 2023-01-24T14:43:03.773 INFO:teuthology.orchestra.run.smithi035.stdout:Total download size: 177 k 2023-01-24T14:43:03.773 INFO:teuthology.orchestra.run.smithi035.stdout:Installed size: 626 k 2023-01-24T14:43:03.774 INFO:teuthology.orchestra.run.smithi035.stdout:Downloading Packages: 2023-01-24T14:43:03.976 INFO:teuthology.orchestra.run.smithi035.stdout:rbd-nbd-18.0.0-1954.g1a671c19.el8.x86_64.rpm 868 kB/s | 177 kB 00:00 2023-01-24T14:43:03.977 INFO:teuthology.orchestra.run.smithi035.stdout:-------------------------------------------------------------------------------- 2023-01-24T14:43:03.977 INFO:teuthology.orchestra.run.smithi035.stdout:Total 863 kB/s | 177 kB 00:00 2023-01-24T14:43:03.977 INFO:teuthology.orchestra.run.smithi035.stdout:Running transaction check 2023-01-24T14:43:03.989 INFO:teuthology.orchestra.run.smithi035.stdout:Transaction check succeeded. 2023-01-24T14:43:03.989 INFO:teuthology.orchestra.run.smithi035.stdout:Running transaction test 2023-01-24T14:43:04.044 INFO:teuthology.orchestra.run.smithi035.stdout:Transaction test succeeded. 2023-01-24T14:43:04.048 INFO:teuthology.orchestra.run.smithi035.stdout:Running transaction 2023-01-24T14:43:04.200 INFO:teuthology.orchestra.run.smithi035.stdout: Preparing : 1/1 2023-01-24T14:43:04.386 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : rbd-nbd-2:18.0.0-1954.g1a671c19.el8.x86_64 1/1 2023-01-24T14:43:04.703 INFO:teuthology.orchestra.run.smithi035.stdout: Running scriptlet: rbd-nbd-2:18.0.0-1954.g1a671c19.el8.x86_64 1/1 2023-01-24T14:43:05.218 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : rbd-nbd-2:18.0.0-1954.g1a671c19.el8.x86_64 1/1 2023-01-24T14:43:05.218 INFO:teuthology.orchestra.run.smithi035.stdout: 2023-01-24T14:43:05.219 INFO:teuthology.orchestra.run.smithi035.stdout:Installed: 2023-01-24T14:43:05.219 INFO:teuthology.orchestra.run.smithi035.stdout: rbd-nbd-2:18.0.0-1954.g1a671c19.el8.x86_64 2023-01-24T14:43:05.219 INFO:teuthology.orchestra.run.smithi035.stdout: 2023-01-24T14:43:05.219 INFO:teuthology.orchestra.run.smithi035.stdout:Complete! 2023-01-24T14:43:05.422 DEBUG:teuthology.parallel:result is None 2023-01-24T14:43:05.423 DEBUG:teuthology.orchestra.run.smithi027:> sudo lsb_release -is 2023-01-24T14:43:05.472 INFO:teuthology.orchestra.run.smithi027.stdout:CentOSStream 2023-01-24T14:43:05.474 DEBUG:teuthology.misc:System to be installed: CentOSStream 2023-01-24T14:43:05.475 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F8%2Fx86_64&sha1=1a671c198559f54389cd5afa22085a0c7b29a1ef 2023-01-24T14:43:05.656 DEBUG:teuthology.orchestra.run.smithi027:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-01-24T14:43:05.681 INFO:teuthology.orchestra.run.smithi027.stdout:18.0.0-1954.g1a671c19.el8 2023-01-24T14:43:05.682 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-1954.g1a671c19.el8 2023-01-24T14:43:05.682 INFO:teuthology.task.install:The correct ceph version 18.0.0-1954.g1a671c19 is installed. 2023-01-24T14:43:05.683 DEBUG:teuthology.orchestra.run.smithi035:> sudo lsb_release -is 2023-01-24T14:43:05.723 INFO:teuthology.orchestra.run.smithi035.stdout:CentOSStream 2023-01-24T14:43:05.724 DEBUG:teuthology.misc:System to be installed: CentOSStream 2023-01-24T14:43:05.725 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F8%2Fx86_64&sha1=1a671c198559f54389cd5afa22085a0c7b29a1ef 2023-01-24T14:43:05.893 DEBUG:teuthology.orchestra.run.smithi035:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-01-24T14:43:05.917 INFO:teuthology.orchestra.run.smithi035.stdout:18.0.0-1954.g1a671c19.el8 2023-01-24T14:43:05.918 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-1954.g1a671c19.el8 2023-01-24T14:43:05.919 INFO:teuthology.task.install:The correct ceph version 18.0.0-1954.g1a671c19 is installed. 2023-01-24T14:43:05.919 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-01-24T14:43:05.920 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2023-01-24T14:43:05.920 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-01-24T14:43:05.953 DEBUG:teuthology.orchestra.run.smithi035:> set -ex 2023-01-24T14:43:05.953 DEBUG:teuthology.orchestra.run.smithi035:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-01-24T14:43:05.987 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-01-24T14:43:05.988 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2023-01-24T14:43:05.988 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd of=/usr/bin/daemon-helper 2023-01-24T14:43:06.024 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-01-24T14:43:06.094 DEBUG:teuthology.orchestra.run.smithi035:> set -ex 2023-01-24T14:43:06.095 DEBUG:teuthology.orchestra.run.smithi035:> sudo dd of=/usr/bin/daemon-helper 2023-01-24T14:43:06.123 DEBUG:teuthology.orchestra.run.smithi035:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-01-24T14:43:06.191 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-01-24T14:43:06.192 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2023-01-24T14:43:06.192 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd of=/usr/bin/adjust-ulimits 2023-01-24T14:43:06.222 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-01-24T14:43:06.291 DEBUG:teuthology.orchestra.run.smithi035:> set -ex 2023-01-24T14:43:06.292 DEBUG:teuthology.orchestra.run.smithi035:> sudo dd of=/usr/bin/adjust-ulimits 2023-01-24T14:43:06.321 DEBUG:teuthology.orchestra.run.smithi035:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-01-24T14:43:06.390 INFO:teuthology.run_tasks:Running task ceph... 2023-01-24T14:43:06.461 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2023-01-24T14:43:06.462 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod 777 /var/log/ceph 2023-01-24T14:43:06.465 DEBUG:teuthology.orchestra.run.smithi035:> sudo chmod 777 /var/log/ceph 2023-01-24T14:43:06.493 INFO:tasks.ceph:Disabling ceph logrotate... 2023-01-24T14:43:06.494 DEBUG:teuthology.orchestra.run.smithi027:> sudo rm -f -- /etc/logrotate.d/ceph 2023-01-24T14:43:06.534 DEBUG:teuthology.orchestra.run.smithi035:> sudo rm -f -- /etc/logrotate.d/ceph 2023-01-24T14:43:06.562 INFO:tasks.ceph:Creating extra log directories... 2023-01-24T14:43:06.563 DEBUG:teuthology.orchestra.run.smithi027:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-01-24T14:43:06.602 DEBUG:teuthology.orchestra.run.smithi035:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-01-24T14:43:06.640 INFO:tasks.ceph:Creating ceph cluster ceph... 2023-01-24T14:43:06.641 INFO:tasks.ceph:config {'conf': {'mon': {'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10, 'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon warn on pool no app': False}, 'osd': {'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'osd blocked scrub grace period': 3600, 'osd class default list': '*', 'osd class load list': '*', 'osd shutdown pgref assert': True}, 'client': {'debug ms': 1}, 'global': {'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'reached quota', 'but it is still running', 'overall HEALTH_', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-01-24T14:43:06.641 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/rzarzynski-2023-01-24_14:08:39-crimson-rados-main-distro-default-smithi/7135310', 'branch': 'main', 'description': 'crimson-rados/basic/{centos_latest clusters/fixed-2 crimson_qa_overrides deploy/ceph tasks/rados_api_tests}', 'email': None, 'first_in_suite': False, 'job_id': '7135310', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'rzarzynski-2023-01-24_14:08:39-crimson-rados-main-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'os_type': 'centos', 'os_version': '8.stream', 'overrides': {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'client': {'debug ms': 1}, 'global': {'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon warn on pool no app': False}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd blocked scrub grace period': 3600, 'osd class default list': '*', 'osd class load list': '*', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'reached quota', 'but it is still running', 'overall HEALTH_', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)'], 'sha1': '1a671c198559f54389cd5afa22085a0c7b29a1ef'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}}}, 'install': {'ceph': {'flavor': 'crimson', 'sha1': '1a671c198559f54389cd5afa22085a0c7b29a1ef'}}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'workunit': {'branch': 'main', 'env': {'CRIMSON_COMPAT': '1'}, 'sha1': '510284b66513490445619d1430aa869868c71a09'}}, 'owner': 'scheduled_rzarzynski@teuthology', 'priority': 60, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'osd.0', 'osd.1', 'client.0', 'node-exporter.a'], ['mgr.x', 'osd.2', 'osd.3', 'client.1', 'prometheus.a', 'node-exporter.b']], 'sha1': '1a671c198559f54389cd5afa22085a0c7b29a1ef', 'sleep_before_teardown': 0, 'suite': 'crimson-rados', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_510284b66513490445619d1430aa869868c71a09/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '510284b66513490445619d1430aa869868c71a09', 'targets': {'smithi027.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCyyKjOLmVFo1yODYdugRvmA4PwF5LOyll0nMHXIdtr5y8mJCfNBjSVho1aO7Lj0wsEcUYvj6sJubW/NQF8tRXEyewW7ueIkPWYZhpn+F4XU9/I/3spaTpil9gzzUo09gyWbpoPrRcp+KfD2Yd41gbkXVahA/ckqjKFzWt9WoI2O9yf9xG5A3Iqrk+X2NINWRjZvl/OBN2eG2U7NqtHIUuAcrrNMHviybyW3iCx6JIG+tyhBEsfz9ynrs81Z3VXpqzxYbMwZzUy6jR0rK7s8DOmlJrtneeuV1vP7ZQL529Mot1eEAIC6tu7+DDADy9VKa42zxGV+BLj90yzKCxjoEG3N+g7k1XPynu8+NpThviPHHu/xkJtIM8ARF1FihI+zqmmEthyGxBEu4hhP6UI7gcdSfCXj7Z/Xe+vHEJDRGvfRD5Qu6UBtXovn/OsEdASb63WB6MmtIm9H9iVq7dKgMqIlhsSkyPJcBCof8o+YIZOVhGM9/vLOM0OghreMPERH0s=', 'smithi035.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC0U35SOhGJEcCl3gsD7NS9turfhFdIoVqcbI+HP7UJ9k9QXoGr8FaC86jZTiqo53O8iPTfHawLYATSYEb2fHnIcqPAjGOesshjL5uJW2eQ3UN9LcDMpAD2VLbqHY64AvfgTGnnOP7uvBt9Aq1pmGneE6f0uzaee3PWKKr7DqtZpMbEPzSAzwsy/4+IQmdDRMpopzHRbFhI1sYUe0Z/tPazDvreeINHFtBx+T8M764LG0YKHWvLdkoE6k12BDv0lPlZbd4lwKrDmiejQZIZL+3RczovsnH2eidLdDj0aYGVdFRYfUBY5kNaU08TlKe/TJ1zD2P2/KemmjxHErqMJ6wOHqZ+iZylsczlruglSqZKtNeVECl+S0wj5Fk6EtKGkzEfWs1jRiKS9U1Zota+jdLLT7STBXHl42v1FMB1xsc4FvIr5sAk7wOHq7nauvXeZTpoLpn11VqDPKUmjWGt7JklTkMa8zjhIkHMdu7D3qtBj+SDhg/eX2n5wSvVnlVhOe8='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'conf': {'mon': {'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10, 'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon warn on pool no app': False}, 'osd': {'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'osd blocked scrub grace period': 3600, 'osd class default list': '*', 'osd class load list': '*', 'osd shutdown pgref assert': True}, 'client': {'debug ms': 1}, 'global': {'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'reached quota', 'but it is still running', 'overall HEALTH_', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)'], 'sha1': '1a671c198559f54389cd5afa22085a0c7b29a1ef', 'cluster': 'ceph'}}, {'workunit': {'clients': {'client.0': ['rados/test.sh', 'rados/test_pool_quota.sh']}}}], 'teuthology_branch': 'main', 'teuthology_sha1': '8e1bba12e584b7ae912e92e94afe9c075ab51fa3', 'timestamp': '2023-01-24_14:08:39', 'tube': 'smithi', 'user': 'rzarzynski', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.17735'} 2023-01-24T14:43:06.642 DEBUG:teuthology.orchestra.run.smithi027:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-01-24T14:43:06.671 DEBUG:teuthology.orchestra.run.smithi035:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-01-24T14:43:06.690 DEBUG:teuthology.orchestra.run.smithi027:> sudo install -d -m0777 -- /var/run/ceph 2023-01-24T14:43:06.730 DEBUG:teuthology.orchestra.run.smithi035:> sudo install -d -m0777 -- /var/run/ceph 2023-01-24T14:43:06.758 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2023-01-24T14:43:06.758 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/scratch_devs of=/dev/stdout 2023-01-24T14:43:06.813 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-01-24T14:43:06.814 DEBUG:teuthology.orchestra.run.smithi027:> stat /dev/vg_nvme/lv_1 2023-01-24T14:43:06.871 INFO:teuthology.orchestra.run.smithi027.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-01-24T14:43:06.871 INFO:teuthology.orchestra.run.smithi027.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-01-24T14:43:06.872 INFO:teuthology.orchestra.run.smithi027.stdout:Device: 6h/6d Inode: 369100 Links: 1 2023-01-24T14:43:06.872 INFO:teuthology.orchestra.run.smithi027.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-01-24T14:43:06.873 INFO:teuthology.orchestra.run.smithi027.stdout:Context: system_u:object_r:device_t:s0 2023-01-24T14:43:06.873 INFO:teuthology.orchestra.run.smithi027.stdout:Access: 2023-01-24 14:41:39.860287819 +0000 2023-01-24T14:43:06.873 INFO:teuthology.orchestra.run.smithi027.stdout:Modify: 2023-01-24 14:27:12.422937563 +0000 2023-01-24T14:43:06.873 INFO:teuthology.orchestra.run.smithi027.stdout:Change: 2023-01-24 14:27:12.422937563 +0000 2023-01-24T14:43:06.873 INFO:teuthology.orchestra.run.smithi027.stdout: Birth: - 2023-01-24T14:43:06.874 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-01-24T14:43:06.938 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records in 2023-01-24T14:43:06.938 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records out 2023-01-24T14:43:06.938 INFO:teuthology.orchestra.run.smithi027.stderr:512 bytes copied, 0.000234012 s, 2.2 MB/s 2023-01-24T14:43:06.939 DEBUG:teuthology.orchestra.run.smithi027:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-01-24T14:43:06.998 DEBUG:teuthology.orchestra.run.smithi027:> stat /dev/vg_nvme/lv_2 2023-01-24T14:43:07.058 INFO:teuthology.orchestra.run.smithi027.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-01-24T14:43:07.059 INFO:teuthology.orchestra.run.smithi027.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-01-24T14:43:07.059 INFO:teuthology.orchestra.run.smithi027.stdout:Device: 6h/6d Inode: 373888 Links: 1 2023-01-24T14:43:07.059 INFO:teuthology.orchestra.run.smithi027.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-01-24T14:43:07.059 INFO:teuthology.orchestra.run.smithi027.stdout:Context: system_u:object_r:device_t:s0 2023-01-24T14:43:07.060 INFO:teuthology.orchestra.run.smithi027.stdout:Access: 2023-01-24 14:41:39.860287819 +0000 2023-01-24T14:43:07.060 INFO:teuthology.orchestra.run.smithi027.stdout:Modify: 2023-01-24 14:27:12.948924092 +0000 2023-01-24T14:43:07.060 INFO:teuthology.orchestra.run.smithi027.stdout:Change: 2023-01-24 14:27:12.948924092 +0000 2023-01-24T14:43:07.060 INFO:teuthology.orchestra.run.smithi027.stdout: Birth: - 2023-01-24T14:43:07.061 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-01-24T14:43:07.124 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records in 2023-01-24T14:43:07.125 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records out 2023-01-24T14:43:07.125 INFO:teuthology.orchestra.run.smithi027.stderr:512 bytes copied, 0.000210407 s, 2.4 MB/s 2023-01-24T14:43:07.126 DEBUG:teuthology.orchestra.run.smithi027:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-01-24T14:43:07.186 DEBUG:teuthology.orchestra.run.smithi027:> stat /dev/vg_nvme/lv_3 2023-01-24T14:43:07.245 INFO:teuthology.orchestra.run.smithi027.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-01-24T14:43:07.245 INFO:teuthology.orchestra.run.smithi027.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-01-24T14:43:07.245 INFO:teuthology.orchestra.run.smithi027.stdout:Device: 6h/6d Inode: 370107 Links: 1 2023-01-24T14:43:07.246 INFO:teuthology.orchestra.run.smithi027.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-01-24T14:43:07.246 INFO:teuthology.orchestra.run.smithi027.stdout:Context: system_u:object_r:device_t:s0 2023-01-24T14:43:07.246 INFO:teuthology.orchestra.run.smithi027.stdout:Access: 2023-01-24 14:41:39.860287819 +0000 2023-01-24T14:43:07.246 INFO:teuthology.orchestra.run.smithi027.stdout:Modify: 2023-01-24 14:27:13.456911082 +0000 2023-01-24T14:43:07.246 INFO:teuthology.orchestra.run.smithi027.stdout:Change: 2023-01-24 14:27:13.456911082 +0000 2023-01-24T14:43:07.246 INFO:teuthology.orchestra.run.smithi027.stdout: Birth: - 2023-01-24T14:43:07.247 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-01-24T14:43:07.312 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records in 2023-01-24T14:43:07.313 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records out 2023-01-24T14:43:07.313 INFO:teuthology.orchestra.run.smithi027.stderr:512 bytes copied, 0.00021663 s, 2.4 MB/s 2023-01-24T14:43:07.314 DEBUG:teuthology.orchestra.run.smithi027:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-01-24T14:43:07.373 DEBUG:teuthology.orchestra.run.smithi027:> stat /dev/vg_nvme/lv_4 2023-01-24T14:43:07.432 INFO:teuthology.orchestra.run.smithi027.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-01-24T14:43:07.433 INFO:teuthology.orchestra.run.smithi027.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-01-24T14:43:07.433 INFO:teuthology.orchestra.run.smithi027.stdout:Device: 6h/6d Inode: 370262 Links: 1 2023-01-24T14:43:07.433 INFO:teuthology.orchestra.run.smithi027.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-01-24T14:43:07.433 INFO:teuthology.orchestra.run.smithi027.stdout:Context: system_u:object_r:device_t:s0 2023-01-24T14:43:07.434 INFO:teuthology.orchestra.run.smithi027.stdout:Access: 2023-01-24 14:41:39.860287819 +0000 2023-01-24T14:43:07.434 INFO:teuthology.orchestra.run.smithi027.stdout:Modify: 2023-01-24 14:27:13.978897713 +0000 2023-01-24T14:43:07.434 INFO:teuthology.orchestra.run.smithi027.stdout:Change: 2023-01-24 14:27:13.978897713 +0000 2023-01-24T14:43:07.434 INFO:teuthology.orchestra.run.smithi027.stdout: Birth: - 2023-01-24T14:43:07.435 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-01-24T14:43:07.500 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records in 2023-01-24T14:43:07.500 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records out 2023-01-24T14:43:07.501 INFO:teuthology.orchestra.run.smithi027.stderr:512 bytes copied, 0.000185084 s, 2.8 MB/s 2023-01-24T14:43:07.502 DEBUG:teuthology.orchestra.run.smithi027:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-01-24T14:43:07.561 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2'} 2023-01-24T14:43:07.562 DEBUG:teuthology.orchestra.run.smithi035:> set -ex 2023-01-24T14:43:07.562 DEBUG:teuthology.orchestra.run.smithi035:> dd if=/scratch_devs of=/dev/stdout 2023-01-24T14:43:07.584 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-01-24T14:43:07.584 DEBUG:teuthology.orchestra.run.smithi035:> stat /dev/vg_nvme/lv_1 2023-01-24T14:43:07.645 INFO:teuthology.orchestra.run.smithi035.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-01-24T14:43:07.645 INFO:teuthology.orchestra.run.smithi035.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-01-24T14:43:07.645 INFO:teuthology.orchestra.run.smithi035.stdout:Device: 6h/6d Inode: 370584 Links: 1 2023-01-24T14:43:07.646 INFO:teuthology.orchestra.run.smithi035.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-01-24T14:43:07.646 INFO:teuthology.orchestra.run.smithi035.stdout:Context: system_u:object_r:device_t:s0 2023-01-24T14:43:07.646 INFO:teuthology.orchestra.run.smithi035.stdout:Access: 2023-01-24 14:43:00.786677365 +0000 2023-01-24T14:43:07.646 INFO:teuthology.orchestra.run.smithi035.stdout:Modify: 2023-01-24 14:27:20.177323980 +0000 2023-01-24T14:43:07.646 INFO:teuthology.orchestra.run.smithi035.stdout:Change: 2023-01-24 14:27:20.177323980 +0000 2023-01-24T14:43:07.647 INFO:teuthology.orchestra.run.smithi035.stdout: Birth: - 2023-01-24T14:43:07.647 DEBUG:teuthology.orchestra.run.smithi035:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-01-24T14:43:07.716 INFO:teuthology.orchestra.run.smithi035.stderr:1+0 records in 2023-01-24T14:43:07.716 INFO:teuthology.orchestra.run.smithi035.stderr:1+0 records out 2023-01-24T14:43:07.716 INFO:teuthology.orchestra.run.smithi035.stderr:512 bytes copied, 0.000341194 s, 1.5 MB/s 2023-01-24T14:43:07.717 DEBUG:teuthology.orchestra.run.smithi035:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-01-24T14:43:07.780 DEBUG:teuthology.orchestra.run.smithi035:> stat /dev/vg_nvme/lv_2 2023-01-24T14:43:07.840 INFO:teuthology.orchestra.run.smithi035.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-01-24T14:43:07.841 INFO:teuthology.orchestra.run.smithi035.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-01-24T14:43:07.841 INFO:teuthology.orchestra.run.smithi035.stdout:Device: 6h/6d Inode: 372317 Links: 1 2023-01-24T14:43:07.841 INFO:teuthology.orchestra.run.smithi035.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-01-24T14:43:07.841 INFO:teuthology.orchestra.run.smithi035.stdout:Context: system_u:object_r:device_t:s0 2023-01-24T14:43:07.841 INFO:teuthology.orchestra.run.smithi035.stdout:Access: 2023-01-24 14:43:00.787677344 +0000 2023-01-24T14:43:07.842 INFO:teuthology.orchestra.run.smithi035.stdout:Modify: 2023-01-24 14:27:20.700313292 +0000 2023-01-24T14:43:07.842 INFO:teuthology.orchestra.run.smithi035.stdout:Change: 2023-01-24 14:27:20.700313292 +0000 2023-01-24T14:43:07.842 INFO:teuthology.orchestra.run.smithi035.stdout: Birth: - 2023-01-24T14:43:07.842 DEBUG:teuthology.orchestra.run.smithi035:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-01-24T14:43:07.911 INFO:teuthology.orchestra.run.smithi035.stderr:1+0 records in 2023-01-24T14:43:07.911 INFO:teuthology.orchestra.run.smithi035.stderr:1+0 records out 2023-01-24T14:43:07.911 INFO:teuthology.orchestra.run.smithi035.stderr:512 bytes copied, 0.000210563 s, 2.4 MB/s 2023-01-24T14:43:07.912 DEBUG:teuthology.orchestra.run.smithi035:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-01-24T14:43:07.975 DEBUG:teuthology.orchestra.run.smithi035:> stat /dev/vg_nvme/lv_3 2023-01-24T14:43:08.035 INFO:teuthology.orchestra.run.smithi035.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-01-24T14:43:08.036 INFO:teuthology.orchestra.run.smithi035.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-01-24T14:43:08.036 INFO:teuthology.orchestra.run.smithi035.stdout:Device: 6h/6d Inode: 373346 Links: 1 2023-01-24T14:43:08.036 INFO:teuthology.orchestra.run.smithi035.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-01-24T14:43:08.036 INFO:teuthology.orchestra.run.smithi035.stdout:Context: system_u:object_r:device_t:s0 2023-01-24T14:43:08.037 INFO:teuthology.orchestra.run.smithi035.stdout:Access: 2023-01-24 14:43:00.787677344 +0000 2023-01-24T14:43:08.037 INFO:teuthology.orchestra.run.smithi035.stdout:Modify: 2023-01-24 14:27:21.226302543 +0000 2023-01-24T14:43:08.037 INFO:teuthology.orchestra.run.smithi035.stdout:Change: 2023-01-24 14:27:21.226302543 +0000 2023-01-24T14:43:08.037 INFO:teuthology.orchestra.run.smithi035.stdout: Birth: - 2023-01-24T14:43:08.038 DEBUG:teuthology.orchestra.run.smithi035:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-01-24T14:43:08.105 INFO:teuthology.orchestra.run.smithi035.stderr:1+0 records in 2023-01-24T14:43:08.106 INFO:teuthology.orchestra.run.smithi035.stderr:1+0 records out 2023-01-24T14:43:08.106 INFO:teuthology.orchestra.run.smithi035.stderr:512 bytes copied, 0.000245114 s, 2.1 MB/s 2023-01-24T14:43:08.107 DEBUG:teuthology.orchestra.run.smithi035:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-01-24T14:43:08.168 DEBUG:teuthology.orchestra.run.smithi035:> stat /dev/vg_nvme/lv_4 2023-01-24T14:43:08.230 INFO:teuthology.orchestra.run.smithi035.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-01-24T14:43:08.230 INFO:teuthology.orchestra.run.smithi035.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-01-24T14:43:08.231 INFO:teuthology.orchestra.run.smithi035.stdout:Device: 6h/6d Inode: 366486 Links: 1 2023-01-24T14:43:08.231 INFO:teuthology.orchestra.run.smithi035.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-01-24T14:43:08.231 INFO:teuthology.orchestra.run.smithi035.stdout:Context: system_u:object_r:device_t:s0 2023-01-24T14:43:08.232 INFO:teuthology.orchestra.run.smithi035.stdout:Access: 2023-01-24 14:43:00.787677344 +0000 2023-01-24T14:43:08.232 INFO:teuthology.orchestra.run.smithi035.stdout:Modify: 2023-01-24 14:27:21.755291733 +0000 2023-01-24T14:43:08.232 INFO:teuthology.orchestra.run.smithi035.stdout:Change: 2023-01-24 14:27:21.755291733 +0000 2023-01-24T14:43:08.232 INFO:teuthology.orchestra.run.smithi035.stdout: Birth: - 2023-01-24T14:43:08.233 DEBUG:teuthology.orchestra.run.smithi035:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-01-24T14:43:08.299 INFO:teuthology.orchestra.run.smithi035.stderr:1+0 records in 2023-01-24T14:43:08.299 INFO:teuthology.orchestra.run.smithi035.stderr:1+0 records out 2023-01-24T14:43:08.299 INFO:teuthology.orchestra.run.smithi035.stderr:512 bytes copied, 0.000200687 s, 2.6 MB/s 2023-01-24T14:43:08.300 DEBUG:teuthology.orchestra.run.smithi035:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-01-24T14:43:08.363 INFO:tasks.ceph:osd dev map: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.3': '/dev/vg_nvme/lv_2'} 2023-01-24T14:43:08.363 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi027.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2'}, Remote(name='ubuntu@smithi035.front.sepia.ceph.com'): {'osd.2': '/dev/vg_nvme/lv_1', 'osd.3': '/dev/vg_nvme/lv_2'}} 2023-01-24T14:43:08.364 INFO:tasks.ceph:Generating config... 2023-01-24T14:43:08.365 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2023-01-24T14:43:08.365 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2023-01-24T14:43:08.366 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2023-01-24T14:43:08.366 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2023-01-24T14:43:08.366 INFO:tasks.ceph:[mon] paxos service trim min = 10 2023-01-24T14:43:08.366 INFO:tasks.ceph:[mon] debug mon = 20 2023-01-24T14:43:08.367 INFO:tasks.ceph:[mon] debug ms = 1 2023-01-24T14:43:08.367 INFO:tasks.ceph:[mon] debug paxos = 20 2023-01-24T14:43:08.367 INFO:tasks.ceph:[mon] mon warn on pool no app = False 2023-01-24T14:43:08.367 INFO:tasks.ceph:[osd] debug monc = 20 2023-01-24T14:43:08.368 INFO:tasks.ceph:[osd] debug ms = 1 2023-01-24T14:43:08.368 INFO:tasks.ceph:[osd] debug osd = 20 2023-01-24T14:43:08.368 INFO:tasks.ceph:[osd] osd blocked scrub grace period = 3600 2023-01-24T14:43:08.368 INFO:tasks.ceph:[osd] osd class default list = * 2023-01-24T14:43:08.369 INFO:tasks.ceph:[osd] osd class load list = * 2023-01-24T14:43:08.369 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2023-01-24T14:43:08.370 INFO:tasks.ceph:[client] debug ms = 1 2023-01-24T14:43:08.370 INFO:tasks.ceph:[global] ms client mode = crc 2023-01-24T14:43:08.370 INFO:tasks.ceph:[global] ms cluster mode = crc 2023-01-24T14:43:08.370 INFO:tasks.ceph:[global] ms mon client mode = crc 2023-01-24T14:43:08.371 INFO:tasks.ceph:[global] ms mon cluster mode = crc 2023-01-24T14:43:08.371 INFO:tasks.ceph:[global] ms mon service mode = crc 2023-01-24T14:43:08.371 INFO:tasks.ceph:[global] ms service mode = crc 2023-01-24T14:43:08.372 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-01-24T14:43:08.372 INFO:tasks.ceph:[mgr] debug ms = 1 2023-01-24T14:43:08.372 INFO:tasks.ceph:Setting up mon.a... 2023-01-24T14:43:08.373 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2023-01-24T14:43:08.433 INFO:teuthology.orchestra.run.smithi027.stdout:creating /etc/ceph/ceph.keyring 2023-01-24T14:43:08.437 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2023-01-24T14:43:08.491 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-01-24T14:43:08.519 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.27')] 2023-01-24T14:43:08.519 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.27', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'osd blocked scrub grace period': 3600, 'osd class default list': '*', 'osd class load list': '*', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10, 'mon warn on pool no app': False}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug ms': 1}, 'mon.a': {}} 2023-01-24T14:43:08.520 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2023-01-24T14:43:08.521 DEBUG:teuthology.orchestra.run.smithi027:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-01-24T14:43:08.578 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.27 --print /home/ubuntu/cephtest/ceph.monmap 2023-01-24T14:43:08.669 INFO:teuthology.orchestra.run.smithi027.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2023-01-24T14:43:08.669 INFO:teuthology.orchestra.run.smithi027.stdout:monmaptool: generated fsid 5e79a879-b90d-445c-b439-aec3d80a8b8a 2023-01-24T14:43:08.670 INFO:teuthology.orchestra.run.smithi027.stdout:setting min_mon_release = pacific 2023-01-24T14:43:08.670 INFO:teuthology.orchestra.run.smithi027.stdout:epoch 0 2023-01-24T14:43:08.670 INFO:teuthology.orchestra.run.smithi027.stdout:fsid 5e79a879-b90d-445c-b439-aec3d80a8b8a 2023-01-24T14:43:08.670 INFO:teuthology.orchestra.run.smithi027.stdout:last_changed 2023-01-24T14:43:08.668963+0000 2023-01-24T14:43:08.670 INFO:teuthology.orchestra.run.smithi027.stdout:created 2023-01-24T14:43:08.668963+0000 2023-01-24T14:43:08.671 INFO:teuthology.orchestra.run.smithi027.stdout:min_mon_release 16 (pacific) 2023-01-24T14:43:08.671 INFO:teuthology.orchestra.run.smithi027.stdout:election_strategy: 1 2023-01-24T14:43:08.671 INFO:teuthology.orchestra.run.smithi027.stdout:0: [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] mon.a 2023-01-24T14:43:08.671 INFO:teuthology.orchestra.run.smithi027.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (1 monitors) 2023-01-24T14:43:08.672 DEBUG:teuthology.orchestra.run.smithi027:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-01-24T14:43:08.691 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 5e79a879-b90d-445c-b439-aec3d80a8b8a... 2023-01-24T14:43:08.692 DEBUG:teuthology.orchestra.run.smithi027:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-01-24T14:43:08.734 DEBUG:teuthology.orchestra.run.smithi035:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-01-24T14:43:08.778 INFO:teuthology.orchestra.run.smithi027.stdout:[global] 2023-01-24T14:43:08.778 INFO:teuthology.orchestra.run.smithi027.stdout: chdir = "" 2023-01-24T14:43:08.778 INFO:teuthology.orchestra.run.smithi027.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-01-24T14:43:08.779 INFO:teuthology.orchestra.run.smithi027.stdout: auth supported = cephx 2023-01-24T14:43:08.779 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-24T14:43:08.779 INFO:teuthology.orchestra.run.smithi027.stdout: filestore xattr use omap = true 2023-01-24T14:43:08.779 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-24T14:43:08.779 INFO:teuthology.orchestra.run.smithi027.stdout: mon clock drift allowed = 1.000 2023-01-24T14:43:08.780 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-24T14:43:08.780 INFO:teuthology.orchestra.run.smithi027.stdout: osd crush chooseleaf type = 0 2023-01-24T14:43:08.780 INFO:teuthology.orchestra.run.smithi027.stdout: auth debug = true 2023-01-24T14:43:08.780 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-24T14:43:08.781 INFO:teuthology.orchestra.run.smithi027.stdout: ms die on old message = true 2023-01-24T14:43:08.781 INFO:teuthology.orchestra.run.smithi027.stdout: ms die on bug = true 2023-01-24T14:43:08.781 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-24T14:43:08.781 INFO:teuthology.orchestra.run.smithi027.stdout: mon max pg per osd = 10000 # >= luminous 2023-01-24T14:43:08.781 INFO:teuthology.orchestra.run.smithi027.stdout: mon pg warn max object skew = 0 2023-01-24T14:43:08.782 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-24T14:43:08.782 INFO:teuthology.orchestra.run.smithi027.stdout: # disable pg_autoscaler by default for new pools 2023-01-24T14:43:08.782 INFO:teuthology.orchestra.run.smithi027.stdout: osd_pool_default_pg_autoscale_mode = off 2023-01-24T14:43:08.782 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-24T14:43:08.782 INFO:teuthology.orchestra.run.smithi027.stdout: osd pool default size = 2 2023-01-24T14:43:08.783 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-24T14:43:08.783 INFO:teuthology.orchestra.run.smithi027.stdout: mon osd allow primary affinity = true 2023-01-24T14:43:08.783 INFO:teuthology.orchestra.run.smithi027.stdout: mon osd allow pg remap = true 2023-01-24T14:43:08.783 INFO:teuthology.orchestra.run.smithi027.stdout: mon warn on legacy crush tunables = false 2023-01-24T14:43:08.783 INFO:teuthology.orchestra.run.smithi027.stdout: mon warn on crush straw calc version zero = false 2023-01-24T14:43:08.783 INFO:teuthology.orchestra.run.smithi027.stdout: mon warn on no sortbitwise = false 2023-01-24T14:43:08.784 INFO:teuthology.orchestra.run.smithi027.stdout: mon warn on osd down out interval zero = false 2023-01-24T14:43:08.784 INFO:teuthology.orchestra.run.smithi027.stdout: mon warn on too few osds = false 2023-01-24T14:43:08.784 INFO:teuthology.orchestra.run.smithi027.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-01-24T14:43:08.784 INFO:teuthology.orchestra.run.smithi027.stdout: mon_warn_on_pool_no_redundancy = false 2023-01-24T14:43:08.785 INFO:teuthology.orchestra.run.smithi027.stdout: mon_allow_pool_size_one = true 2023-01-24T14:43:08.785 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-24T14:43:08.785 INFO:teuthology.orchestra.run.smithi027.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-01-24T14:43:08.785 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-24T14:43:08.785 INFO:teuthology.orchestra.run.smithi027.stdout: osd default data pool replay window = 5 2023-01-24T14:43:08.785 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-24T14:43:08.786 INFO:teuthology.orchestra.run.smithi027.stdout: mon allow pool delete = true 2023-01-24T14:43:08.786 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-24T14:43:08.786 INFO:teuthology.orchestra.run.smithi027.stdout: mon cluster log file level = debug 2023-01-24T14:43:08.786 INFO:teuthology.orchestra.run.smithi027.stdout: debug asserts on shutdown = true 2023-01-24T14:43:08.786 INFO:teuthology.orchestra.run.smithi027.stdout: mon health detail to clog = false 2023-01-24T14:43:08.787 INFO:teuthology.orchestra.run.smithi027.stdout: mon host = 172.21.15.27 2023-01-24T14:43:08.787 INFO:teuthology.orchestra.run.smithi027.stdout: ms client mode = crc 2023-01-24T14:43:08.787 INFO:teuthology.orchestra.run.smithi027.stdout: ms cluster mode = crc 2023-01-24T14:43:08.787 INFO:teuthology.orchestra.run.smithi027.stdout: ms mon client mode = crc 2023-01-24T14:43:08.787 INFO:teuthology.orchestra.run.smithi027.stdout: ms mon cluster mode = crc 2023-01-24T14:43:08.788 INFO:teuthology.orchestra.run.smithi027.stdout: ms mon service mode = crc 2023-01-24T14:43:08.788 INFO:teuthology.orchestra.run.smithi027.stdout: ms service mode = crc 2023-01-24T14:43:08.788 INFO:teuthology.orchestra.run.smithi027.stdout: fsid = 5e79a879-b90d-445c-b439-aec3d80a8b8a 2023-01-24T14:43:08.788 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-24T14:43:08.788 INFO:teuthology.orchestra.run.smithi027.stdout:[osd] 2023-01-24T14:43:08.788 INFO:teuthology.orchestra.run.smithi027.stdout: osd journal size = 100 2023-01-24T14:43:08.789 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-24T14:43:08.789 INFO:teuthology.orchestra.run.smithi027.stdout: osd scrub load threshold = 5.0 2023-01-24T14:43:08.789 INFO:teuthology.orchestra.run.smithi027.stdout: osd scrub max interval = 600 2023-01-24T14:43:08.789 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-24T14:43:08.789 INFO:teuthology.orchestra.run.smithi027.stdout: osd recover clone overlap = true 2023-01-24T14:43:08.790 INFO:teuthology.orchestra.run.smithi027.stdout: osd recovery max chunk = 1048576 2023-01-24T14:43:08.790 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-24T14:43:08.790 INFO:teuthology.orchestra.run.smithi027.stdout: osd debug shutdown = true 2023-01-24T14:43:08.790 INFO:teuthology.orchestra.run.smithi027.stdout: osd debug op order = true 2023-01-24T14:43:08.790 INFO:teuthology.orchestra.run.smithi027.stdout: osd debug verify stray on activate = true 2023-01-24T14:43:08.791 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-24T14:43:08.791 INFO:teuthology.orchestra.run.smithi027.stdout: osd open classes on start = true 2023-01-24T14:43:08.791 INFO:teuthology.orchestra.run.smithi027.stdout: osd debug pg log writeout = true 2023-01-24T14:43:08.791 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-24T14:43:08.791 INFO:teuthology.orchestra.run.smithi027.stdout: osd deep scrub update digest min age = 30 2023-01-24T14:43:08.792 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-24T14:43:08.792 INFO:teuthology.orchestra.run.smithi027.stdout: osd map max advance = 10 2023-01-24T14:43:08.792 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-24T14:43:08.792 INFO:teuthology.orchestra.run.smithi027.stdout: journal zero on create = true 2023-01-24T14:43:08.792 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-24T14:43:08.793 INFO:teuthology.orchestra.run.smithi027.stdout: filestore ondisk finisher threads = 3 2023-01-24T14:43:08.793 INFO:teuthology.orchestra.run.smithi027.stdout: filestore apply finisher threads = 3 2023-01-24T14:43:08.793 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-24T14:43:08.793 INFO:teuthology.orchestra.run.smithi027.stdout: bdev debug aio = true 2023-01-24T14:43:08.793 INFO:teuthology.orchestra.run.smithi027.stdout: osd debug misdirected ops = true 2023-01-24T14:43:08.794 INFO:teuthology.orchestra.run.smithi027.stdout: debug monc = 20 2023-01-24T14:43:08.794 INFO:teuthology.orchestra.run.smithi027.stdout: debug ms = 1 2023-01-24T14:43:08.794 INFO:teuthology.orchestra.run.smithi027.stdout: debug osd = 20 2023-01-24T14:43:08.794 INFO:teuthology.orchestra.run.smithi027.stdout: osd blocked scrub grace period = 3600 2023-01-24T14:43:08.794 INFO:teuthology.orchestra.run.smithi027.stdout: osd class default list = * 2023-01-24T14:43:08.795 INFO:teuthology.orchestra.run.smithi027.stdout: osd class load list = * 2023-01-24T14:43:08.795 INFO:teuthology.orchestra.run.smithi027.stdout: osd shutdown pgref assert = True 2023-01-24T14:43:08.795 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-24T14:43:08.795 INFO:teuthology.orchestra.run.smithi027.stdout:[mgr] 2023-01-24T14:43:08.795 INFO:teuthology.orchestra.run.smithi027.stdout: debug ms = 1 2023-01-24T14:43:08.796 INFO:teuthology.orchestra.run.smithi027.stdout: debug mgr = 20 2023-01-24T14:43:08.796 INFO:teuthology.orchestra.run.smithi027.stdout: debug mon = 20 2023-01-24T14:43:08.796 INFO:teuthology.orchestra.run.smithi027.stdout: debug auth = 20 2023-01-24T14:43:08.796 INFO:teuthology.orchestra.run.smithi027.stdout: mon reweight min pgs per osd = 4 2023-01-24T14:43:08.796 INFO:teuthology.orchestra.run.smithi027.stdout: mon reweight min bytes per osd = 10 2023-01-24T14:43:08.797 INFO:teuthology.orchestra.run.smithi027.stdout: mgr/telemetry/nag = false 2023-01-24T14:43:08.797 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-24T14:43:08.797 INFO:teuthology.orchestra.run.smithi027.stdout:[mon] 2023-01-24T14:43:08.797 INFO:teuthology.orchestra.run.smithi027.stdout: debug ms = 1 2023-01-24T14:43:08.798 INFO:teuthology.orchestra.run.smithi027.stdout: debug mon = 20 2023-01-24T14:43:08.798 INFO:teuthology.orchestra.run.smithi027.stdout: debug paxos = 20 2023-01-24T14:43:08.798 INFO:teuthology.orchestra.run.smithi027.stdout: debug auth = 20 2023-01-24T14:43:08.799 INFO:teuthology.orchestra.run.smithi027.stdout: mon data avail warn = 5 2023-01-24T14:43:08.799 INFO:teuthology.orchestra.run.smithi027.stdout: mon mgr mkfs grace = 240 2023-01-24T14:43:08.799 INFO:teuthology.orchestra.run.smithi027.stdout: mon reweight min pgs per osd = 4 2023-01-24T14:43:08.799 INFO:teuthology.orchestra.run.smithi027.stdout: mon osd reporter subtree level = osd 2023-01-24T14:43:08.799 INFO:teuthology.orchestra.run.smithi027.stdout: mon osd prime pg temp = true 2023-01-24T14:43:08.800 INFO:teuthology.orchestra.run.smithi027.stdout: mon reweight min bytes per osd = 10 2023-01-24T14:43:08.800 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-24T14:43:08.800 INFO:teuthology.orchestra.run.smithi027.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-01-24T14:43:08.800 INFO:teuthology.orchestra.run.smithi027.stdout: auth mon ticket ttl = 660 # 11m 2023-01-24T14:43:08.801 INFO:teuthology.orchestra.run.smithi027.stdout: auth service ticket ttl = 240 # 4m 2023-01-24T14:43:08.801 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-24T14:43:08.801 INFO:teuthology.orchestra.run.smithi027.stdout: # don't complain about insecure global_id in the test suite 2023-01-24T14:43:08.801 INFO:teuthology.orchestra.run.smithi027.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-01-24T14:43:08.801 INFO:teuthology.orchestra.run.smithi027.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-01-24T14:43:08.802 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-24T14:43:08.802 INFO:teuthology.orchestra.run.smithi027.stdout: # 1m isn't quite enough 2023-01-24T14:43:08.802 INFO:teuthology.orchestra.run.smithi027.stdout: mon_down_mkfs_grace = 2m 2023-01-24T14:43:08.802 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-24T14:43:08.802 INFO:teuthology.orchestra.run.smithi027.stdout: mon_warn_on_filestore_osds = false 2023-01-24T14:43:08.802 INFO:teuthology.orchestra.run.smithi027.stdout: mon min osdmap epochs = 50 2023-01-24T14:43:08.803 INFO:teuthology.orchestra.run.smithi027.stdout: mon osdmap full prune interval = 2 2023-01-24T14:43:08.803 INFO:teuthology.orchestra.run.smithi027.stdout: mon osdmap full prune min = 15 2023-01-24T14:43:08.803 INFO:teuthology.orchestra.run.smithi027.stdout: mon osdmap full prune txsize = 2 2023-01-24T14:43:08.803 INFO:teuthology.orchestra.run.smithi027.stdout: paxos service trim min = 10 2023-01-24T14:43:08.803 INFO:teuthology.orchestra.run.smithi027.stdout: mon warn on pool no app = False 2023-01-24T14:43:08.804 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-24T14:43:08.804 INFO:teuthology.orchestra.run.smithi027.stdout:[client] 2023-01-24T14:43:08.804 INFO:teuthology.orchestra.run.smithi027.stdout: rgw cache enabled = true 2023-01-24T14:43:08.804 INFO:teuthology.orchestra.run.smithi027.stdout: rgw enable ops log = true 2023-01-24T14:43:08.804 INFO:teuthology.orchestra.run.smithi027.stdout: rgw enable usage log = true 2023-01-24T14:43:08.805 INFO:teuthology.orchestra.run.smithi027.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-01-24T14:43:08.805 INFO:teuthology.orchestra.run.smithi027.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-01-24T14:43:08.805 INFO:teuthology.orchestra.run.smithi027.stdout: debug ms = 1 2023-01-24T14:43:08.805 INFO:teuthology.orchestra.run.smithi027.stdout:[mon.a] 2023-01-24T14:43:08.807 INFO:teuthology.orchestra.run.smithi035.stdout:[global] 2023-01-24T14:43:08.807 INFO:teuthology.orchestra.run.smithi035.stdout: chdir = "" 2023-01-24T14:43:08.807 INFO:teuthology.orchestra.run.smithi035.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-01-24T14:43:08.807 INFO:teuthology.orchestra.run.smithi035.stdout: auth supported = cephx 2023-01-24T14:43:08.807 INFO:teuthology.orchestra.run.smithi035.stdout: 2023-01-24T14:43:08.808 INFO:teuthology.orchestra.run.smithi035.stdout: filestore xattr use omap = true 2023-01-24T14:43:08.808 INFO:teuthology.orchestra.run.smithi035.stdout: 2023-01-24T14:43:08.808 INFO:teuthology.orchestra.run.smithi035.stdout: mon clock drift allowed = 1.000 2023-01-24T14:43:08.808 INFO:teuthology.orchestra.run.smithi035.stdout: 2023-01-24T14:43:08.808 INFO:teuthology.orchestra.run.smithi035.stdout: osd crush chooseleaf type = 0 2023-01-24T14:43:08.809 INFO:teuthology.orchestra.run.smithi035.stdout: auth debug = true 2023-01-24T14:43:08.809 INFO:teuthology.orchestra.run.smithi035.stdout: 2023-01-24T14:43:08.809 INFO:teuthology.orchestra.run.smithi035.stdout: ms die on old message = true 2023-01-24T14:43:08.809 INFO:teuthology.orchestra.run.smithi035.stdout: ms die on bug = true 2023-01-24T14:43:08.809 INFO:teuthology.orchestra.run.smithi035.stdout: 2023-01-24T14:43:08.810 INFO:teuthology.orchestra.run.smithi035.stdout: mon max pg per osd = 10000 # >= luminous 2023-01-24T14:43:08.810 INFO:teuthology.orchestra.run.smithi035.stdout: mon pg warn max object skew = 0 2023-01-24T14:43:08.810 INFO:teuthology.orchestra.run.smithi035.stdout: 2023-01-24T14:43:08.810 INFO:teuthology.orchestra.run.smithi035.stdout: # disable pg_autoscaler by default for new pools 2023-01-24T14:43:08.810 INFO:teuthology.orchestra.run.smithi035.stdout: osd_pool_default_pg_autoscale_mode = off 2023-01-24T14:43:08.810 INFO:teuthology.orchestra.run.smithi035.stdout: 2023-01-24T14:43:08.811 INFO:teuthology.orchestra.run.smithi035.stdout: osd pool default size = 2 2023-01-24T14:43:08.811 INFO:teuthology.orchestra.run.smithi035.stdout: 2023-01-24T14:43:08.811 INFO:teuthology.orchestra.run.smithi035.stdout: mon osd allow primary affinity = true 2023-01-24T14:43:08.811 INFO:teuthology.orchestra.run.smithi035.stdout: mon osd allow pg remap = true 2023-01-24T14:43:08.811 INFO:teuthology.orchestra.run.smithi035.stdout: mon warn on legacy crush tunables = false 2023-01-24T14:43:08.812 INFO:teuthology.orchestra.run.smithi035.stdout: mon warn on crush straw calc version zero = false 2023-01-24T14:43:08.812 INFO:teuthology.orchestra.run.smithi035.stdout: mon warn on no sortbitwise = false 2023-01-24T14:43:08.812 INFO:teuthology.orchestra.run.smithi035.stdout: mon warn on osd down out interval zero = false 2023-01-24T14:43:08.812 INFO:teuthology.orchestra.run.smithi035.stdout: mon warn on too few osds = false 2023-01-24T14:43:08.813 INFO:teuthology.orchestra.run.smithi035.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-01-24T14:43:08.813 INFO:teuthology.orchestra.run.smithi035.stdout: mon_warn_on_pool_no_redundancy = false 2023-01-24T14:43:08.813 INFO:teuthology.orchestra.run.smithi035.stdout: mon_allow_pool_size_one = true 2023-01-24T14:43:08.813 INFO:teuthology.orchestra.run.smithi035.stdout: 2023-01-24T14:43:08.813 INFO:teuthology.orchestra.run.smithi035.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-01-24T14:43:08.814 INFO:teuthology.orchestra.run.smithi035.stdout: 2023-01-24T14:43:08.814 INFO:teuthology.orchestra.run.smithi035.stdout: osd default data pool replay window = 5 2023-01-24T14:43:08.814 INFO:teuthology.orchestra.run.smithi035.stdout: 2023-01-24T14:43:08.814 INFO:teuthology.orchestra.run.smithi035.stdout: mon allow pool delete = true 2023-01-24T14:43:08.814 INFO:teuthology.orchestra.run.smithi035.stdout: 2023-01-24T14:43:08.815 INFO:teuthology.orchestra.run.smithi035.stdout: mon cluster log file level = debug 2023-01-24T14:43:08.815 INFO:teuthology.orchestra.run.smithi035.stdout: debug asserts on shutdown = true 2023-01-24T14:43:08.815 INFO:teuthology.orchestra.run.smithi035.stdout: mon health detail to clog = false 2023-01-24T14:43:08.815 INFO:teuthology.orchestra.run.smithi035.stdout: mon host = 172.21.15.27 2023-01-24T14:43:08.815 INFO:teuthology.orchestra.run.smithi035.stdout: ms client mode = crc 2023-01-24T14:43:08.816 INFO:teuthology.orchestra.run.smithi035.stdout: ms cluster mode = crc 2023-01-24T14:43:08.816 INFO:teuthology.orchestra.run.smithi035.stdout: ms mon client mode = crc 2023-01-24T14:43:08.816 INFO:teuthology.orchestra.run.smithi035.stdout: ms mon cluster mode = crc 2023-01-24T14:43:08.816 INFO:teuthology.orchestra.run.smithi035.stdout: ms mon service mode = crc 2023-01-24T14:43:08.816 INFO:teuthology.orchestra.run.smithi035.stdout: ms service mode = crc 2023-01-24T14:43:08.817 INFO:teuthology.orchestra.run.smithi035.stdout: fsid = 5e79a879-b90d-445c-b439-aec3d80a8b8a 2023-01-24T14:43:08.817 INFO:teuthology.orchestra.run.smithi035.stdout: 2023-01-24T14:43:08.817 INFO:teuthology.orchestra.run.smithi035.stdout:[osd] 2023-01-24T14:43:08.817 INFO:teuthology.orchestra.run.smithi035.stdout: osd journal size = 100 2023-01-24T14:43:08.817 INFO:teuthology.orchestra.run.smithi035.stdout: 2023-01-24T14:43:08.817 INFO:teuthology.orchestra.run.smithi035.stdout: osd scrub load threshold = 5.0 2023-01-24T14:43:08.818 INFO:teuthology.orchestra.run.smithi035.stdout: osd scrub max interval = 600 2023-01-24T14:43:08.818 INFO:teuthology.orchestra.run.smithi035.stdout: 2023-01-24T14:43:08.818 INFO:teuthology.orchestra.run.smithi035.stdout: osd recover clone overlap = true 2023-01-24T14:43:08.818 INFO:teuthology.orchestra.run.smithi035.stdout: osd recovery max chunk = 1048576 2023-01-24T14:43:08.818 INFO:teuthology.orchestra.run.smithi035.stdout: 2023-01-24T14:43:08.818 INFO:teuthology.orchestra.run.smithi035.stdout: osd debug shutdown = true 2023-01-24T14:43:08.819 INFO:teuthology.orchestra.run.smithi035.stdout: osd debug op order = true 2023-01-24T14:43:08.819 INFO:teuthology.orchestra.run.smithi035.stdout: osd debug verify stray on activate = true 2023-01-24T14:43:08.819 INFO:teuthology.orchestra.run.smithi035.stdout: 2023-01-24T14:43:08.819 INFO:teuthology.orchestra.run.smithi035.stdout: osd open classes on start = true 2023-01-24T14:43:08.819 INFO:teuthology.orchestra.run.smithi035.stdout: osd debug pg log writeout = true 2023-01-24T14:43:08.820 INFO:teuthology.orchestra.run.smithi035.stdout: 2023-01-24T14:43:08.820 INFO:teuthology.orchestra.run.smithi035.stdout: osd deep scrub update digest min age = 30 2023-01-24T14:43:08.820 INFO:teuthology.orchestra.run.smithi035.stdout: 2023-01-24T14:43:08.820 INFO:teuthology.orchestra.run.smithi035.stdout: osd map max advance = 10 2023-01-24T14:43:08.820 INFO:teuthology.orchestra.run.smithi035.stdout: 2023-01-24T14:43:08.821 INFO:teuthology.orchestra.run.smithi035.stdout: journal zero on create = true 2023-01-24T14:43:08.821 INFO:teuthology.orchestra.run.smithi035.stdout: 2023-01-24T14:43:08.821 INFO:teuthology.orchestra.run.smithi035.stdout: filestore ondisk finisher threads = 3 2023-01-24T14:43:08.821 INFO:teuthology.orchestra.run.smithi035.stdout: filestore apply finisher threads = 3 2023-01-24T14:43:08.821 INFO:teuthology.orchestra.run.smithi035.stdout: 2023-01-24T14:43:08.822 INFO:teuthology.orchestra.run.smithi035.stdout: bdev debug aio = true 2023-01-24T14:43:08.822 INFO:teuthology.orchestra.run.smithi035.stdout: osd debug misdirected ops = true 2023-01-24T14:43:08.822 INFO:teuthology.orchestra.run.smithi035.stdout: debug monc = 20 2023-01-24T14:43:08.822 INFO:teuthology.orchestra.run.smithi035.stdout: debug ms = 1 2023-01-24T14:43:08.822 INFO:teuthology.orchestra.run.smithi035.stdout: debug osd = 20 2023-01-24T14:43:08.822 INFO:teuthology.orchestra.run.smithi035.stdout: osd blocked scrub grace period = 3600 2023-01-24T14:43:08.823 INFO:teuthology.orchestra.run.smithi035.stdout: osd class default list = * 2023-01-24T14:43:08.823 INFO:teuthology.orchestra.run.smithi035.stdout: osd class load list = * 2023-01-24T14:43:08.823 INFO:teuthology.orchestra.run.smithi035.stdout: osd shutdown pgref assert = True 2023-01-24T14:43:08.823 INFO:teuthology.orchestra.run.smithi035.stdout: 2023-01-24T14:43:08.823 INFO:teuthology.orchestra.run.smithi035.stdout:[mgr] 2023-01-24T14:43:08.823 INFO:teuthology.orchestra.run.smithi035.stdout: debug ms = 1 2023-01-24T14:43:08.824 INFO:teuthology.orchestra.run.smithi035.stdout: debug mgr = 20 2023-01-24T14:43:08.824 INFO:teuthology.orchestra.run.smithi035.stdout: debug mon = 20 2023-01-24T14:43:08.824 INFO:teuthology.orchestra.run.smithi035.stdout: debug auth = 20 2023-01-24T14:43:08.824 INFO:teuthology.orchestra.run.smithi035.stdout: mon reweight min pgs per osd = 4 2023-01-24T14:43:08.824 INFO:teuthology.orchestra.run.smithi035.stdout: mon reweight min bytes per osd = 10 2023-01-24T14:43:08.825 INFO:teuthology.orchestra.run.smithi035.stdout: mgr/telemetry/nag = false 2023-01-24T14:43:08.825 INFO:teuthology.orchestra.run.smithi035.stdout: 2023-01-24T14:43:08.825 INFO:teuthology.orchestra.run.smithi035.stdout:[mon] 2023-01-24T14:43:08.825 INFO:teuthology.orchestra.run.smithi035.stdout: debug ms = 1 2023-01-24T14:43:08.825 INFO:teuthology.orchestra.run.smithi035.stdout: debug mon = 20 2023-01-24T14:43:08.826 INFO:teuthology.orchestra.run.smithi035.stdout: debug paxos = 20 2023-01-24T14:43:08.826 INFO:teuthology.orchestra.run.smithi035.stdout: debug auth = 20 2023-01-24T14:43:08.826 INFO:teuthology.orchestra.run.smithi035.stdout: mon data avail warn = 5 2023-01-24T14:43:08.826 INFO:teuthology.orchestra.run.smithi035.stdout: mon mgr mkfs grace = 240 2023-01-24T14:43:08.826 INFO:teuthology.orchestra.run.smithi035.stdout: mon reweight min pgs per osd = 4 2023-01-24T14:43:08.826 INFO:teuthology.orchestra.run.smithi035.stdout: mon osd reporter subtree level = osd 2023-01-24T14:43:08.827 INFO:teuthology.orchestra.run.smithi035.stdout: mon osd prime pg temp = true 2023-01-24T14:43:08.827 INFO:teuthology.orchestra.run.smithi035.stdout: mon reweight min bytes per osd = 10 2023-01-24T14:43:08.827 INFO:teuthology.orchestra.run.smithi035.stdout: 2023-01-24T14:43:08.827 INFO:teuthology.orchestra.run.smithi035.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-01-24T14:43:08.827 INFO:teuthology.orchestra.run.smithi035.stdout: auth mon ticket ttl = 660 # 11m 2023-01-24T14:43:08.827 INFO:teuthology.orchestra.run.smithi035.stdout: auth service ticket ttl = 240 # 4m 2023-01-24T14:43:08.828 INFO:teuthology.orchestra.run.smithi035.stdout: 2023-01-24T14:43:08.828 INFO:teuthology.orchestra.run.smithi035.stdout: # don't complain about insecure global_id in the test suite 2023-01-24T14:43:08.828 INFO:teuthology.orchestra.run.smithi035.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-01-24T14:43:08.828 INFO:teuthology.orchestra.run.smithi035.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-01-24T14:43:08.828 INFO:teuthology.orchestra.run.smithi035.stdout: 2023-01-24T14:43:08.829 INFO:teuthology.orchestra.run.smithi035.stdout: # 1m isn't quite enough 2023-01-24T14:43:08.829 INFO:teuthology.orchestra.run.smithi035.stdout: mon_down_mkfs_grace = 2m 2023-01-24T14:43:08.829 INFO:teuthology.orchestra.run.smithi035.stdout: 2023-01-24T14:43:08.829 INFO:teuthology.orchestra.run.smithi035.stdout: mon_warn_on_filestore_osds = false 2023-01-24T14:43:08.829 INFO:teuthology.orchestra.run.smithi035.stdout: mon min osdmap epochs = 50 2023-01-24T14:43:08.830 INFO:teuthology.orchestra.run.smithi035.stdout: mon osdmap full prune interval = 2 2023-01-24T14:43:08.830 INFO:teuthology.orchestra.run.smithi035.stdout: mon osdmap full prune min = 15 2023-01-24T14:43:08.830 INFO:teuthology.orchestra.run.smithi035.stdout: mon osdmap full prune txsize = 2 2023-01-24T14:43:08.830 INFO:teuthology.orchestra.run.smithi035.stdout: paxos service trim min = 10 2023-01-24T14:43:08.830 INFO:teuthology.orchestra.run.smithi035.stdout: mon warn on pool no app = False 2023-01-24T14:43:08.831 INFO:teuthology.orchestra.run.smithi035.stdout: 2023-01-24T14:43:08.831 INFO:teuthology.orchestra.run.smithi035.stdout:[client] 2023-01-24T14:43:08.831 INFO:teuthology.orchestra.run.smithi035.stdout: rgw cache enabled = true 2023-01-24T14:43:08.831 INFO:teuthology.orchestra.run.smithi035.stdout: rgw enable ops log = true 2023-01-24T14:43:08.831 INFO:teuthology.orchestra.run.smithi035.stdout: rgw enable usage log = true 2023-01-24T14:43:08.831 INFO:teuthology.orchestra.run.smithi035.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-01-24T14:43:08.832 INFO:teuthology.orchestra.run.smithi035.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-01-24T14:43:08.832 INFO:teuthology.orchestra.run.smithi035.stdout: debug ms = 1 2023-01-24T14:43:08.832 INFO:teuthology.orchestra.run.smithi035.stdout:[mon.a] 2023-01-24T14:43:08.832 INFO:tasks.ceph:Creating admin key on mon.a... 2023-01-24T14:43:08.833 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2023-01-24T14:43:08.913 INFO:tasks.ceph:Copying monmap to all nodes... 2023-01-24T14:43:08.913 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2023-01-24T14:43:08.914 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2023-01-24T14:43:08.932 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2023-01-24T14:43:08.933 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2023-01-24T14:43:08.990 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi027.front.sepia.ceph.com 2023-01-24T14:43:08.990 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2023-01-24T14:43:08.991 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd of=/etc/ceph/ceph.keyring 2023-01-24T14:43:08.991 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-01-24T14:43:09.068 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2023-01-24T14:43:09.068 DEBUG:teuthology.orchestra.run.smithi027:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-01-24T14:43:09.126 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi035.front.sepia.ceph.com 2023-01-24T14:43:09.127 DEBUG:teuthology.orchestra.run.smithi035:> set -ex 2023-01-24T14:43:09.127 DEBUG:teuthology.orchestra.run.smithi035:> sudo dd of=/etc/ceph/ceph.keyring 2023-01-24T14:43:09.127 DEBUG:teuthology.orchestra.run.smithi035:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-01-24T14:43:09.171 DEBUG:teuthology.orchestra.run.smithi035:> set -ex 2023-01-24T14:43:09.172 DEBUG:teuthology.orchestra.run.smithi035:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-01-24T14:43:09.230 INFO:tasks.ceph:Setting up mon nodes... 2023-01-24T14:43:09.231 INFO:tasks.ceph:Setting up mgr nodes... 2023-01-24T14:43:09.231 DEBUG:teuthology.orchestra.run.smithi035:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2023-01-24T14:43:09.347 INFO:teuthology.orchestra.run.smithi035.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2023-01-24T14:43:09.350 INFO:tasks.ceph:Setting up mds nodes... 2023-01-24T14:43:09.350 INFO:tasks.ceph_client:Setting up client nodes... 2023-01-24T14:43:09.351 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-01-24T14:43:09.403 INFO:teuthology.orchestra.run.smithi027.stdout:creating /etc/ceph/ceph.client.0.keyring 2023-01-24T14:43:09.416 DEBUG:teuthology.orchestra.run.smithi035:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2023-01-24T14:43:09.469 INFO:teuthology.orchestra.run.smithi035.stdout:creating /etc/ceph/ceph.client.1.keyring 2023-01-24T14:43:09.482 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-01-24T14:43:09.483 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi027.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2'}, Remote(name='ubuntu@smithi035.front.sepia.ceph.com'): {'osd.2': '/dev/vg_nvme/lv_1', 'osd.3': '/dev/vg_nvme/lv_2'}} 2023-01-24T14:43:09.483 DEBUG:teuthology.orchestra.run.smithi027:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2023-01-24T14:43:09.511 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2'} 2023-01-24T14:43:09.511 INFO:tasks.ceph:role: osd.0 2023-01-24T14:43:09.512 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi027.front.sepia.ceph.com 2023-01-24T14:43:09.512 DEBUG:teuthology.orchestra.run.smithi027:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-01-24T14:43:09.587 INFO:teuthology.orchestra.run.smithi027.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-01-24T14:43:09.587 INFO:teuthology.orchestra.run.smithi027.stdout: = sectsz=512 attr=2, projid32bit=1 2023-01-24T14:43:09.587 INFO:teuthology.orchestra.run.smithi027.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-01-24T14:43:09.588 INFO:teuthology.orchestra.run.smithi027.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-01-24T14:43:09.588 INFO:teuthology.orchestra.run.smithi027.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-01-24T14:43:09.588 INFO:teuthology.orchestra.run.smithi027.stdout: = sunit=0 swidth=0 blks 2023-01-24T14:43:09.588 INFO:teuthology.orchestra.run.smithi027.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-01-24T14:43:09.589 INFO:teuthology.orchestra.run.smithi027.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-01-24T14:43:09.589 INFO:teuthology.orchestra.run.smithi027.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-01-24T14:43:09.589 INFO:teuthology.orchestra.run.smithi027.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-01-24T14:43:10.014 INFO:teuthology.orchestra.run.smithi027.stdout:Discarding blocks...Done. 2023-01-24T14:43:10.016 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi027.front.sepia.ceph.com -o noatime 2023-01-24T14:43:10.016 DEBUG:teuthology.orchestra.run.smithi027:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2023-01-24T14:43:10.190 DEBUG:teuthology.orchestra.run.smithi027:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2023-01-24T14:43:10.227 DEBUG:teuthology.orchestra.run.smithi027:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2023-01-24T14:43:10.296 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2'} 2023-01-24T14:43:10.296 INFO:tasks.ceph:role: osd.1 2023-01-24T14:43:10.297 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi027.front.sepia.ceph.com 2023-01-24T14:43:10.297 DEBUG:teuthology.orchestra.run.smithi027:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-01-24T14:43:10.372 INFO:teuthology.orchestra.run.smithi027.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-01-24T14:43:10.373 INFO:teuthology.orchestra.run.smithi027.stdout: = sectsz=512 attr=2, projid32bit=1 2023-01-24T14:43:10.373 INFO:teuthology.orchestra.run.smithi027.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-01-24T14:43:10.373 INFO:teuthology.orchestra.run.smithi027.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-01-24T14:43:10.374 INFO:teuthology.orchestra.run.smithi027.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-01-24T14:43:10.374 INFO:teuthology.orchestra.run.smithi027.stdout: = sunit=0 swidth=0 blks 2023-01-24T14:43:10.375 INFO:teuthology.orchestra.run.smithi027.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-01-24T14:43:10.375 INFO:teuthology.orchestra.run.smithi027.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-01-24T14:43:10.375 INFO:teuthology.orchestra.run.smithi027.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-01-24T14:43:10.376 INFO:teuthology.orchestra.run.smithi027.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-01-24T14:43:10.795 INFO:teuthology.orchestra.run.smithi027.stdout:Discarding blocks...Done. 2023-01-24T14:43:10.797 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi027.front.sepia.ceph.com -o noatime 2023-01-24T14:43:10.797 DEBUG:teuthology.orchestra.run.smithi027:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2023-01-24T14:43:10.905 DEBUG:teuthology.orchestra.run.smithi027:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2023-01-24T14:43:10.938 DEBUG:teuthology.orchestra.run.smithi027:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-01-24T14:43:11.794 INFO:teuthology.orchestra.run.smithi027.stderr:WARNING: debug mode. Not for benchmarking or production 2023-01-24T14:43:11.795 INFO:teuthology.orchestra.run.smithi027.stderr:INFO 2023-01-24 14:43:11,794 seastar - Reactor backend: epoll 2023-01-24T14:43:11.803 INFO:teuthology.orchestra.run.smithi027.stderr:INFO 2023-01-24 14:43:11,802 [shard 0] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-01-24T14:43:11.804 INFO:teuthology.orchestra.run.smithi027.stderr:INFO 2023-01-24 14:43:11,802 [shard 0] seastar - IO queue uses 0.75ms latency goal for device 0 2023-01-24T14:43:11.804 INFO:teuthology.orchestra.run.smithi027.stderr:INFO 2023-01-24 14:43:11,802 [shard 0] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-01-24T14:43:11.804 INFO:teuthology.orchestra.run.smithi027.stderr:INFO 2023-01-24 14:43:11,802 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-01-24T14:43:11.805 INFO:teuthology.orchestra.run.smithi027.stderr:==100385==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-01-24T14:43:11.812 INFO:teuthology.orchestra.run.smithi027.stderr:==100385==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7ffdb8188000; bottom 0x7fb8ed811000; size: 0x0044ca977000 (295456698368) 2023-01-24T14:43:11.812 INFO:teuthology.orchestra.run.smithi027.stderr:False positive error reports may follow 2023-01-24T14:43:11.812 INFO:teuthology.orchestra.run.smithi027.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-01-24T14:43:11.849 INFO:teuthology.orchestra.run.smithi027.stderr:INFO 2023-01-24 14:43:11,848 [shard 0] osd - bypassing the config fetch due to --no-mon-config 2023-01-24T14:43:11.849 INFO:teuthology.orchestra.run.smithi027.stderr:INFO 2023-01-24 14:43:11,848 [shard 0] osd - get_early_config: set --smp 1 2023-01-24T14:43:12.377 INFO:teuthology.orchestra.run.smithi027.stderr:WARNING: debug mode. Not for benchmarking or production 2023-01-24T14:43:12.377 INFO:teuthology.orchestra.run.smithi027.stderr:INFO 2023-01-24 14:43:12,377 seastar - Reactor backend: epoll 2023-01-24T14:43:12.385 INFO:teuthology.orchestra.run.smithi027.stderr:INFO 2023-01-24 14:43:12,385 [shard 0] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-01-24T14:43:12.385 INFO:teuthology.orchestra.run.smithi027.stderr:INFO 2023-01-24 14:43:12,385 [shard 0] seastar - IO queue uses 0.75ms latency goal for device 0 2023-01-24T14:43:12.386 INFO:teuthology.orchestra.run.smithi027.stderr:INFO 2023-01-24 14:43:12,385 [shard 0] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-01-24T14:43:12.386 INFO:teuthology.orchestra.run.smithi027.stderr:INFO 2023-01-24 14:43:12,385 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-01-24T14:43:12.387 INFO:teuthology.orchestra.run.smithi027.stderr:==100383==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-01-24T14:43:12.393 INFO:teuthology.orchestra.run.smithi027.stderr:==100383==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7ffdb8188000; bottom 0x7fb8ed80a000; size: 0x0044ca97e000 (295456727040) 2023-01-24T14:43:12.393 INFO:teuthology.orchestra.run.smithi027.stderr:False positive error reports may follow 2023-01-24T14:43:12.393 INFO:teuthology.orchestra.run.smithi027.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-01-24T14:43:12.462 INFO:teuthology.orchestra.run.smithi027.stderr:Reactor stalled for 32 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x3a0591d6 0x3a0a3fb1 0x3a06343c 0x39e3c956 0x39e60d8a 0x39e61e3e 0x39e65eb9 0x39e677e8 0x3614421b 0x361c9ca8 0x361e141d 0x361e91a7 0x445f3ec8 2023-01-24T14:43:12.462 INFO:teuthology.orchestra.run.smithi027.stderr:kernel callstack: 2023-01-24T14:43:15.360 INFO:teuthology.orchestra.run.smithi027.stdout:created object store /var/lib/ceph/osd/ceph-0 for osd.0 fsid 5e79a879-b90d-445c-b439-aec3d80a8b8a 2023-01-24T14:43:15.423 DEBUG:teuthology.orchestra.run.smithi027:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2023-01-24T14:43:15.452 DEBUG:teuthology.orchestra.run.smithi027:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-01-24T14:43:16.295 INFO:teuthology.orchestra.run.smithi027.stderr:WARNING: debug mode. Not for benchmarking or production 2023-01-24T14:43:16.296 INFO:teuthology.orchestra.run.smithi027.stderr:INFO 2023-01-24 14:43:16,295 seastar - Reactor backend: epoll 2023-01-24T14:43:16.304 INFO:teuthology.orchestra.run.smithi027.stderr:INFO 2023-01-24 14:43:16,303 [shard 0] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-01-24T14:43:16.304 INFO:teuthology.orchestra.run.smithi027.stderr:INFO 2023-01-24 14:43:16,303 [shard 0] seastar - IO queue uses 0.75ms latency goal for device 0 2023-01-24T14:43:16.305 INFO:teuthology.orchestra.run.smithi027.stderr:INFO 2023-01-24 14:43:16,303 [shard 0] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-01-24T14:43:16.305 INFO:teuthology.orchestra.run.smithi027.stderr:INFO 2023-01-24 14:43:16,303 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-01-24T14:43:16.306 INFO:teuthology.orchestra.run.smithi027.stderr:==101203==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-01-24T14:43:16.312 INFO:teuthology.orchestra.run.smithi027.stderr:==101203==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7ffe159e6000; bottom 0x7f5251cea000; size: 0x00abc3cfc000 (737724579840) 2023-01-24T14:43:16.313 INFO:teuthology.orchestra.run.smithi027.stderr:False positive error reports may follow 2023-01-24T14:43:16.313 INFO:teuthology.orchestra.run.smithi027.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-01-24T14:43:16.350 INFO:teuthology.orchestra.run.smithi027.stderr:INFO 2023-01-24 14:43:16,349 [shard 0] osd - bypassing the config fetch due to --no-mon-config 2023-01-24T14:43:16.350 INFO:teuthology.orchestra.run.smithi027.stderr:INFO 2023-01-24 14:43:16,349 [shard 0] osd - get_early_config: set --smp 1 2023-01-24T14:43:16.858 INFO:teuthology.orchestra.run.smithi027.stderr:WARNING: debug mode. Not for benchmarking or production 2023-01-24T14:43:16.858 INFO:teuthology.orchestra.run.smithi027.stderr:INFO 2023-01-24 14:43:16,858 seastar - Reactor backend: epoll 2023-01-24T14:43:16.866 INFO:teuthology.orchestra.run.smithi027.stderr:INFO 2023-01-24 14:43:16,866 [shard 0] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-01-24T14:43:16.867 INFO:teuthology.orchestra.run.smithi027.stderr:INFO 2023-01-24 14:43:16,866 [shard 0] seastar - IO queue uses 0.75ms latency goal for device 0 2023-01-24T14:43:16.867 INFO:teuthology.orchestra.run.smithi027.stderr:INFO 2023-01-24 14:43:16,866 [shard 0] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-01-24T14:43:16.867 INFO:teuthology.orchestra.run.smithi027.stderr:INFO 2023-01-24 14:43:16,866 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-01-24T14:43:16.868 INFO:teuthology.orchestra.run.smithi027.stderr:==101201==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-01-24T14:43:16.874 INFO:teuthology.orchestra.run.smithi027.stderr:==101201==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7ffe159e6000; bottom 0x7f5251ce3000; size: 0x00abc3d03000 (737724608512) 2023-01-24T14:43:16.874 INFO:teuthology.orchestra.run.smithi027.stderr:False positive error reports may follow 2023-01-24T14:43:16.875 INFO:teuthology.orchestra.run.smithi027.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-01-24T14:43:19.820 INFO:teuthology.orchestra.run.smithi027.stdout:created object store /var/lib/ceph/osd/ceph-1 for osd.1 fsid 5e79a879-b90d-445c-b439-aec3d80a8b8a 2023-01-24T14:43:19.910 DEBUG:teuthology.orchestra.run.smithi027:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2023-01-24T14:43:19.938 DEBUG:teuthology.orchestra.run.smithi035:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2023-01-24T14:43:19.972 INFO:tasks.ceph:roles_to_devs: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.3': '/dev/vg_nvme/lv_2'} 2023-01-24T14:43:19.972 INFO:tasks.ceph:role: osd.2 2023-01-24T14:43:19.973 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi035.front.sepia.ceph.com 2023-01-24T14:43:19.973 DEBUG:teuthology.orchestra.run.smithi035:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-01-24T14:43:20.052 INFO:teuthology.orchestra.run.smithi035.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-01-24T14:43:20.053 INFO:teuthology.orchestra.run.smithi035.stdout: = sectsz=512 attr=2, projid32bit=1 2023-01-24T14:43:20.053 INFO:teuthology.orchestra.run.smithi035.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-01-24T14:43:20.053 INFO:teuthology.orchestra.run.smithi035.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-01-24T14:43:20.053 INFO:teuthology.orchestra.run.smithi035.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-01-24T14:43:20.054 INFO:teuthology.orchestra.run.smithi035.stdout: = sunit=0 swidth=0 blks 2023-01-24T14:43:20.054 INFO:teuthology.orchestra.run.smithi035.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-01-24T14:43:20.054 INFO:teuthology.orchestra.run.smithi035.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-01-24T14:43:20.054 INFO:teuthology.orchestra.run.smithi035.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-01-24T14:43:20.054 INFO:teuthology.orchestra.run.smithi035.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-01-24T14:43:20.495 INFO:teuthology.orchestra.run.smithi035.stdout:Discarding blocks...Done. 2023-01-24T14:43:20.497 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi035.front.sepia.ceph.com -o noatime 2023-01-24T14:43:20.497 DEBUG:teuthology.orchestra.run.smithi035:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-2 2023-01-24T14:43:20.633 DEBUG:teuthology.orchestra.run.smithi035:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2023-01-24T14:43:20.671 DEBUG:teuthology.orchestra.run.smithi035:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2023-01-24T14:43:20.742 INFO:tasks.ceph:roles_to_devs: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.3': '/dev/vg_nvme/lv_2'} 2023-01-24T14:43:20.742 INFO:tasks.ceph:role: osd.3 2023-01-24T14:43:20.742 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi035.front.sepia.ceph.com 2023-01-24T14:43:20.743 DEBUG:teuthology.orchestra.run.smithi035:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-01-24T14:43:20.820 INFO:teuthology.orchestra.run.smithi035.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-01-24T14:43:20.821 INFO:teuthology.orchestra.run.smithi035.stdout: = sectsz=512 attr=2, projid32bit=1 2023-01-24T14:43:20.821 INFO:teuthology.orchestra.run.smithi035.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-01-24T14:43:20.821 INFO:teuthology.orchestra.run.smithi035.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-01-24T14:43:20.821 INFO:teuthology.orchestra.run.smithi035.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-01-24T14:43:20.822 INFO:teuthology.orchestra.run.smithi035.stdout: = sunit=0 swidth=0 blks 2023-01-24T14:43:20.822 INFO:teuthology.orchestra.run.smithi035.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-01-24T14:43:20.822 INFO:teuthology.orchestra.run.smithi035.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-01-24T14:43:20.822 INFO:teuthology.orchestra.run.smithi035.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-01-24T14:43:20.822 INFO:teuthology.orchestra.run.smithi035.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-01-24T14:43:21.261 INFO:teuthology.orchestra.run.smithi035.stdout:Discarding blocks...Done. 2023-01-24T14:43:21.263 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi035.front.sepia.ceph.com -o noatime 2023-01-24T14:43:21.263 DEBUG:teuthology.orchestra.run.smithi035:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-3 2023-01-24T14:43:21.353 DEBUG:teuthology.orchestra.run.smithi035:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2023-01-24T14:43:21.389 DEBUG:teuthology.orchestra.run.smithi035:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-01-24T14:43:22.243 INFO:teuthology.orchestra.run.smithi035.stderr:WARNING: debug mode. Not for benchmarking or production 2023-01-24T14:43:22.243 INFO:teuthology.orchestra.run.smithi035.stderr:INFO 2023-01-24 14:43:22,243 seastar - Reactor backend: epoll 2023-01-24T14:43:22.252 INFO:teuthology.orchestra.run.smithi035.stderr:INFO 2023-01-24 14:43:22,251 [shard 0] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-01-24T14:43:22.252 INFO:teuthology.orchestra.run.smithi035.stderr:INFO 2023-01-24 14:43:22,251 [shard 0] seastar - IO queue uses 0.75ms latency goal for device 0 2023-01-24T14:43:22.253 INFO:teuthology.orchestra.run.smithi035.stderr:INFO 2023-01-24 14:43:22,251 [shard 0] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-01-24T14:43:22.253 INFO:teuthology.orchestra.run.smithi035.stderr:INFO 2023-01-24 14:43:22,251 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-01-24T14:43:22.254 INFO:teuthology.orchestra.run.smithi035.stderr:==100167==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-01-24T14:43:22.260 INFO:teuthology.orchestra.run.smithi035.stderr:==100167==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7ffdadfc6000; bottom 0x7f6819f26000; size: 0x0095940a0000 (642433810432) 2023-01-24T14:43:22.261 INFO:teuthology.orchestra.run.smithi035.stderr:False positive error reports may follow 2023-01-24T14:43:22.261 INFO:teuthology.orchestra.run.smithi035.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-01-24T14:43:22.298 INFO:teuthology.orchestra.run.smithi035.stderr:INFO 2023-01-24 14:43:22,297 [shard 0] osd - bypassing the config fetch due to --no-mon-config 2023-01-24T14:43:22.299 INFO:teuthology.orchestra.run.smithi035.stderr:INFO 2023-01-24 14:43:22,297 [shard 0] osd - get_early_config: set --smp 1 2023-01-24T14:43:22.808 INFO:teuthology.orchestra.run.smithi035.stderr:WARNING: debug mode. Not for benchmarking or production 2023-01-24T14:43:22.809 INFO:teuthology.orchestra.run.smithi035.stderr:INFO 2023-01-24 14:43:22,809 seastar - Reactor backend: epoll 2023-01-24T14:43:22.817 INFO:teuthology.orchestra.run.smithi035.stderr:INFO 2023-01-24 14:43:22,817 [shard 0] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-01-24T14:43:22.817 INFO:teuthology.orchestra.run.smithi035.stderr:INFO 2023-01-24 14:43:22,817 [shard 0] seastar - IO queue uses 0.75ms latency goal for device 0 2023-01-24T14:43:22.818 INFO:teuthology.orchestra.run.smithi035.stderr:INFO 2023-01-24 14:43:22,817 [shard 0] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-01-24T14:43:22.818 INFO:teuthology.orchestra.run.smithi035.stderr:INFO 2023-01-24 14:43:22,817 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-01-24T14:43:22.819 INFO:teuthology.orchestra.run.smithi035.stderr:==100165==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-01-24T14:43:22.825 INFO:teuthology.orchestra.run.smithi035.stderr:==100165==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7ffdadfc6000; bottom 0x7f6819f1f000; size: 0x0095940a7000 (642433839104) 2023-01-24T14:43:22.825 INFO:teuthology.orchestra.run.smithi035.stderr:False positive error reports may follow 2023-01-24T14:43:22.826 INFO:teuthology.orchestra.run.smithi035.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-01-24T14:43:25.765 INFO:teuthology.orchestra.run.smithi035.stdout:created object store /var/lib/ceph/osd/ceph-2 for osd.2 fsid 5e79a879-b90d-445c-b439-aec3d80a8b8a 2023-01-24T14:43:25.829 DEBUG:teuthology.orchestra.run.smithi035:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2023-01-24T14:43:25.857 DEBUG:teuthology.orchestra.run.smithi035:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-01-24T14:43:26.703 INFO:teuthology.orchestra.run.smithi035.stderr:WARNING: debug mode. Not for benchmarking or production 2023-01-24T14:43:26.703 INFO:teuthology.orchestra.run.smithi035.stderr:INFO 2023-01-24 14:43:26,703 seastar - Reactor backend: epoll 2023-01-24T14:43:26.712 INFO:teuthology.orchestra.run.smithi035.stderr:INFO 2023-01-24 14:43:26,711 [shard 0] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-01-24T14:43:26.712 INFO:teuthology.orchestra.run.smithi035.stderr:INFO 2023-01-24 14:43:26,711 [shard 0] seastar - IO queue uses 0.75ms latency goal for device 0 2023-01-24T14:43:26.712 INFO:teuthology.orchestra.run.smithi035.stderr:INFO 2023-01-24 14:43:26,711 [shard 0] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-01-24T14:43:26.712 INFO:teuthology.orchestra.run.smithi035.stderr:INFO 2023-01-24 14:43:26,711 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-01-24T14:43:26.714 INFO:teuthology.orchestra.run.smithi035.stderr:==100985==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-01-24T14:43:26.720 INFO:teuthology.orchestra.run.smithi035.stderr:==100985==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7fffe06c4000; bottom 0x7fd34e9b4000; size: 0x002c91d10000 (191424954368) 2023-01-24T14:43:26.721 INFO:teuthology.orchestra.run.smithi035.stderr:False positive error reports may follow 2023-01-24T14:43:26.721 INFO:teuthology.orchestra.run.smithi035.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-01-24T14:43:26.757 INFO:teuthology.orchestra.run.smithi035.stderr:INFO 2023-01-24 14:43:26,757 [shard 0] osd - bypassing the config fetch due to --no-mon-config 2023-01-24T14:43:26.758 INFO:teuthology.orchestra.run.smithi035.stderr:INFO 2023-01-24 14:43:26,757 [shard 0] osd - get_early_config: set --smp 1 2023-01-24T14:43:27.264 INFO:teuthology.orchestra.run.smithi035.stderr:WARNING: debug mode. Not for benchmarking or production 2023-01-24T14:43:27.265 INFO:teuthology.orchestra.run.smithi035.stderr:INFO 2023-01-24 14:43:27,265 seastar - Reactor backend: epoll 2023-01-24T14:43:27.267 INFO:teuthology.orchestra.run.smithi035.stderr:INFO 2023-01-24 14:43:27,266 [shard 0] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-01-24T14:43:27.267 INFO:teuthology.orchestra.run.smithi035.stderr:INFO 2023-01-24 14:43:27,266 [shard 0] seastar - IO queue uses 0.75ms latency goal for device 0 2023-01-24T14:43:27.267 INFO:teuthology.orchestra.run.smithi035.stderr:INFO 2023-01-24 14:43:27,266 [shard 0] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-01-24T14:43:27.267 INFO:teuthology.orchestra.run.smithi035.stderr:INFO 2023-01-24 14:43:27,266 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-01-24T14:43:27.268 INFO:teuthology.orchestra.run.smithi035.stderr:==100983==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-01-24T14:43:27.275 INFO:teuthology.orchestra.run.smithi035.stderr:==100983==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7fffe06c4000; bottom 0x7fd34e9ad000; size: 0x002c91d17000 (191424983040) 2023-01-24T14:43:27.275 INFO:teuthology.orchestra.run.smithi035.stderr:False positive error reports may follow 2023-01-24T14:43:27.275 INFO:teuthology.orchestra.run.smithi035.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-01-24T14:43:30.217 INFO:teuthology.orchestra.run.smithi035.stdout:created object store /var/lib/ceph/osd/ceph-3 for osd.3 fsid 5e79a879-b90d-445c-b439-aec3d80a8b8a 2023-01-24T14:43:30.307 DEBUG:teuthology.orchestra.run.smithi035:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2023-01-24T14:43:30.335 INFO:tasks.ceph:Reading keys from all nodes... 2023-01-24T14:43:30.336 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2023-01-24T14:43:30.336 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2023-01-24T14:43:30.370 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2023-01-24T14:43:30.371 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2023-01-24T14:43:30.442 DEBUG:teuthology.orchestra.run.smithi035:> set -ex 2023-01-24T14:43:30.443 DEBUG:teuthology.orchestra.run.smithi035:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2023-01-24T14:43:30.471 DEBUG:teuthology.orchestra.run.smithi035:> set -ex 2023-01-24T14:43:30.471 DEBUG:teuthology.orchestra.run.smithi035:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2023-01-24T14:43:30.539 DEBUG:teuthology.orchestra.run.smithi035:> set -ex 2023-01-24T14:43:30.539 DEBUG:teuthology.orchestra.run.smithi035:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2023-01-24T14:43:30.607 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2023-01-24T14:43:30.607 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2023-01-24T14:43:30.628 DEBUG:teuthology.orchestra.run.smithi035:> set -ex 2023-01-24T14:43:30.628 DEBUG:teuthology.orchestra.run.smithi035:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2023-01-24T14:43:30.667 INFO:tasks.ceph:Adding keys to all mons... 2023-01-24T14:43:30.667 DEBUG:teuthology.orchestra.run.smithi027:> sudo tee -a /etc/ceph/ceph.keyring 2023-01-24T14:43:30.698 INFO:teuthology.orchestra.run.smithi027.stdout:[osd.0] 2023-01-24T14:43:30.698 INFO:teuthology.orchestra.run.smithi027.stdout: key = AQCA7s9jF2mNGxAA6gKu1mpsmh7GYfv8JQ1M6w== 2023-01-24T14:43:30.698 INFO:teuthology.orchestra.run.smithi027.stdout:[osd.1] 2023-01-24T14:43:30.698 INFO:teuthology.orchestra.run.smithi027.stdout: key = AQCE7s9j/uVbNxAAaYNxYkXE0SFDYzpMBwXNHw== 2023-01-24T14:43:30.699 INFO:teuthology.orchestra.run.smithi027.stdout:[mgr.x] 2023-01-24T14:43:30.699 INFO:teuthology.orchestra.run.smithi027.stdout: key = AQB97s9jOUCqFBAASQ9a0Lvqc3BU/uWUvhpjdg== 2023-01-24T14:43:30.699 INFO:teuthology.orchestra.run.smithi027.stdout:[osd.2] 2023-01-24T14:43:30.699 INFO:teuthology.orchestra.run.smithi027.stdout: key = AQCK7s9jhWlzNBAAqT0+TjThe96JTZr8Z332Xw== 2023-01-24T14:43:30.700 INFO:teuthology.orchestra.run.smithi027.stdout:[osd.3] 2023-01-24T14:43:30.700 INFO:teuthology.orchestra.run.smithi027.stdout: key = AQCP7s9jQm6hExAApQKWrZiYbS4MpB+E+k+LMg== 2023-01-24T14:43:30.700 INFO:teuthology.orchestra.run.smithi027.stdout:[client.0] 2023-01-24T14:43:30.700 INFO:teuthology.orchestra.run.smithi027.stdout: key = AQB97s9jyf0GGBAACQvtichfWvn7S3oC1ButWA== 2023-01-24T14:43:30.700 INFO:teuthology.orchestra.run.smithi027.stdout:[client.1] 2023-01-24T14:43:30.701 INFO:teuthology.orchestra.run.smithi027.stdout: key = AQB97s9j5ZH6GxAAmrkHHcl5G/sA8yuc9AuslQ== 2023-01-24T14:43:30.702 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-01-24T14:43:30.809 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-01-24T14:43:30.873 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-01-24T14:43:30.935 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-01-24T14:43:30.997 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-01-24T14:43:31.062 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-01-24T14:43:31.128 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-01-24T14:43:31.194 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-01-24T14:43:31.195 DEBUG:teuthology.orchestra.run.smithi027:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2023-01-24T14:43:31.223 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-01-24T14:43:31.341 DEBUG:teuthology.orchestra.run.smithi027:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2023-01-24T14:43:31.369 DEBUG:teuthology.orchestra.run.smithi027:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-01-24T14:43:31.427 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2023-01-24T14:43:31.428 INFO:tasks.ceph.mon.a:Restarting daemon 2023-01-24T14:43:31.428 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2023-01-24T14:43:31.469 INFO:tasks.ceph.mon.a:Started 2023-01-24T14:43:31.470 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2023-01-24T14:43:31.470 INFO:tasks.ceph.mgr.x:Restarting daemon 2023-01-24T14:43:31.471 DEBUG:teuthology.orchestra.run.smithi035:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2023-01-24T14:43:31.473 INFO:tasks.ceph.mgr.x:Started 2023-01-24T14:43:31.474 INFO:tasks.ceph:Setting crush tunables to default 2023-01-24T14:43:31.474 DEBUG:teuthology.orchestra.run.smithi027:> sudo ceph --cluster ceph osd crush tunables default 2023-01-24T14:43:31.611 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:31.607+0000 7fdfe296e700 1 Processor -- start 2023-01-24T14:43:31.612 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:31.607+0000 7fdfe296e700 1 -- start start 2023-01-24T14:43:31.612 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:31.607+0000 7fdfe296e700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fdfdc071090 0x7fdfdc135290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T14:43:31.612 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:31.608+0000 7fdfe296e700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfdc09a000 con 0x7fdfdc098c00 2023-01-24T14:43:31.613 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:31.608+0000 7fdfe296e700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdfdc098940 con 0x7fdfdc071090 2023-01-24T14:43:31.613 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:31.608+0000 7fdfdb7fe700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fdfdc071090 0x7fdfdc135290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T14:43:31.613 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:31.608+0000 7fdfdbfff700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fdfdc098c00 0x7fdfdc070980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:52046/0 (socket says 172.21.15.27:52046) 2023-01-24T14:43:31.613 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:31.608+0000 7fdfdbfff700 1 -- 172.21.15.27:0/1224495967 learned_addr learned my addr 172.21.15.27:0/1224495967 (peer_addr_for_me v1:172.21.15.27:0/0) 2023-01-24T14:43:31.614 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:31.608+0000 7fdfdb7fe700 1 -- 172.21.15.27:0/1224495967 >> v1:172.21.15.27:6789/0 conn(0x7fdfdc098c00 legacy=0x7fdfdc070980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T14:43:31.614 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:31.608+0000 7fdfdb7fe700 1 -- 172.21.15.27:0/1224495967 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfdc0990c0 con 0x7fdfdc071090 2023-01-24T14:43:31.615 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:31.608+0000 7fdfdb7fe700 1 --2- 172.21.15.27:0/1224495967 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdfdc071090 0x7fdfdc135290 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f34442483fb65d4 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T14:43:31.615 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:31.608+0000 7fdfdaffd700 1 -- 172.21.15.27:0/1224495967 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fdfd0009cc0 con 0x7fdfdc071090 2023-01-24T14:43:31.615 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:31.608+0000 7fdfdaffd700 1 -- 172.21.15.27:0/1224495967 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7fdfd0009e20 con 0x7fdfdc071090 2023-01-24T14:43:31.616 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:31.608+0000 7fdfdaffd700 1 -- 172.21.15.27:0/1224495967 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fdfd000e360 con 0x7fdfdc071090 2023-01-24T14:43:31.616 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:31.609+0000 7fdfe296e700 1 -- 172.21.15.27:0/1224495967 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdfdc071090 msgr2=0x7fdfdc135290 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T14:43:31.616 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:31.609+0000 7fdfe296e700 1 --2- 172.21.15.27:0/1224495967 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdfdc071090 0x7fdfdc135290 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T14:43:31.617 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:31.609+0000 7fdfe296e700 1 -- 172.21.15.27:0/1224495967 shutdown_connections 2023-01-24T14:43:31.617 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:31.609+0000 7fdfe296e700 1 --2- 172.21.15.27:0/1224495967 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdfdc071090 0x7fdfdc135290 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T14:43:31.617 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:31.609+0000 7fdfe296e700 1 -- 172.21.15.27:0/1224495967 >> 172.21.15.27:0/1224495967 conn(0x7fdfdc0a32e0 msgr2=0x7fdfdc09f180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T14:43:31.618 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:31.609+0000 7fdfe296e700 1 -- 172.21.15.27:0/1224495967 shutdown_connections 2023-01-24T14:43:31.618 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:31.609+0000 7fdfe296e700 1 -- 172.21.15.27:0/1224495967 wait complete. 2023-01-24T14:43:31.618 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:31.610+0000 7fdfe296e700 1 Processor -- start 2023-01-24T14:43:31.619 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:31.610+0000 7fdfe296e700 1 -- start start 2023-01-24T14:43:31.621 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:31.611+0000 7fdfe296e700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdfdc071090 0x7fdfdc06b3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T14:43:31.622 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:31.611+0000 7fdfe296e700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfdc130f00 con 0x7fdfdc071090 2023-01-24T14:43:31.622 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:31.611+0000 7fdfdbfff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdfdc071090 0x7fdfdc06b3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T14:43:31.623 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:31.611+0000 7fdfdbfff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdfdc071090 0x7fdfdc06b3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56924/0 (socket says 172.21.15.27:56924) 2023-01-24T14:43:31.623 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:31.611+0000 7fdfdbfff700 1 -- 172.21.15.27:0/2395519208 learned_addr learned my addr 172.21.15.27:0/2395519208 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-01-24T14:43:31.623 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:31.611+0000 7fdfdbfff700 1 -- 172.21.15.27:0/2395519208 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfdc09fb10 con 0x7fdfdc071090 2023-01-24T14:43:31.624 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:31.611+0000 7fdfdbfff700 1 --2- 172.21.15.27:0/2395519208 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdfdc071090 0x7fdfdc06b3c0 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T14:43:31.624 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:31.611+0000 7fdfd8ff9700 1 -- 172.21.15.27:0/2395519208 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fdfcc005a60 con 0x7fdfdc071090 2023-01-24T14:43:31.624 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:31.611+0000 7fdfd8ff9700 1 -- 172.21.15.27:0/2395519208 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7fdfcc005bc0 con 0x7fdfdc071090 2023-01-24T14:43:31.625 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:31.611+0000 7fdfe296e700 1 -- 172.21.15.27:0/2395519208 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdfdc099cf0 con 0x7fdfdc071090 2023-01-24T14:43:31.625 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:31.611+0000 7fdfd8ff9700 1 -- 172.21.15.27:0/2395519208 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fdfcc005e60 con 0x7fdfdc071090 2023-01-24T14:43:31.625 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:31.612+0000 7fdfe296e700 1 -- 172.21.15.27:0/2395519208 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdfdc0714a0 con 0x7fdfdc071090 2023-01-24T14:43:31.625 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:31.612+0000 7fdfd8ff9700 1 -- 172.21.15.27:0/2395519208 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 1) v1 ==== 656+0+0 (crc 0 0 0) 0x7fdfcc00caa0 con 0x7fdfdc071090 2023-01-24T14:43:31.626 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:31.612+0000 7fdfd8ff9700 1 -- 172.21.15.27:0/2395519208 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (crc 0 0 0) 0x7fdfcc00f610 con 0x7fdfdc071090 2023-01-24T14:43:31.626 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:31.612+0000 7fdfe296e700 1 -- 172.21.15.27:0/2395519208 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdfdc130830 con 0x7fdfdc071090 2023-01-24T14:43:31.626 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:31.615+0000 7fdfd8ff9700 1 -- 172.21.15.27:0/2395519208 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71752 (crc 0 0 0) 0x7fdfcc0070b0 con 0x7fdfdc071090 2023-01-24T14:43:31.692 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:31.690+0000 7fdfe296e700 1 -- 172.21.15.27:0/2395519208 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7fdfdc072170 con 0x7fdfdc071090 2023-01-24T14:43:31.694 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:31.693+0000 7fdfd8ff9700 1 -- 172.21.15.27:0/2395519208 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (crc 0 0 0) 0x7fdfcc007250 con 0x7fdfdc071090 2023-01-24T14:43:31.695 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:31.693+0000 7fdfe296e700 1 -- 172.21.15.27:0/2395519208 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdfdc071090 msgr2=0x7fdfdc06b3c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T14:43:31.695 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:31.693+0000 7fdfe296e700 1 --2- 172.21.15.27:0/2395519208 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdfdc071090 0x7fdfdc06b3c0 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T14:43:31.696 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:31.693+0000 7fdfe296e700 1 -- 172.21.15.27:0/2395519208 shutdown_connections 2023-01-24T14:43:31.696 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:31.693+0000 7fdfe296e700 1 --2- 172.21.15.27:0/2395519208 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdfdc071090 0x7fdfdc06b3c0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T14:43:31.696 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:31.693+0000 7fdfe296e700 1 -- 172.21.15.27:0/2395519208 >> 172.21.15.27:0/2395519208 conn(0x7fdfdc0a32e0 msgr2=0x7fdfdc0a0480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T14:43:31.696 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:31.693+0000 7fdfe296e700 1 -- 172.21.15.27:0/2395519208 shutdown_connections 2023-01-24T14:43:31.696 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:31.693+0000 7fdfe296e700 1 -- 172.21.15.27:0/2395519208 wait complete. 2023-01-24T14:43:31.697 INFO:teuthology.orchestra.run.smithi027.stderr:adjusted tunables profile to default 2023-01-24T14:43:31.707 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2023-01-24T14:43:31.707 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2023-01-24T14:43:31.708 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2023-01-24T14:43:31.735 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2023-01-24T14:43:31.736 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2023-01-24T14:43:31.804 DEBUG:teuthology.orchestra.run.smithi035:> set -ex 2023-01-24T14:43:31.804 DEBUG:teuthology.orchestra.run.smithi035:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2023-01-24T14:43:31.833 DEBUG:teuthology.orchestra.run.smithi035:> set -ex 2023-01-24T14:43:31.833 DEBUG:teuthology.orchestra.run.smithi035:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2023-01-24T14:43:31.901 DEBUG:teuthology.orchestra.run.smithi035:> sudo ceph --cluster ceph osd new ed53e235-e041-46ca-b000-fecf0c6ce916 0 2023-01-24T14:43:32.074 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.073+0000 7fa806835700 1 Processor -- start 2023-01-24T14:43:32.075 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.073+0000 7fa806835700 1 -- start start 2023-01-24T14:43:32.077 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.073+0000 7fa806835700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa800116770 0x7fa800118b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T14:43:32.077 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.074+0000 7fa806835700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa800119160 con 0x7fa800113a00 2023-01-24T14:43:32.077 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.074+0000 7fa806835700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa80011a320 con 0x7fa800116770 2023-01-24T14:43:32.077 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.074+0000 7fa7f7fff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa800116770 0x7fa800118b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T14:43:32.078 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.074+0000 7fa7fffff700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fa800113a00 0x7fa800113e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.35:42396/0 (socket says 172.21.15.35:42396) 2023-01-24T14:43:32.078 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.074+0000 7fa7fffff700 1 -- 172.21.15.35:0/632349143 learned_addr learned my addr 172.21.15.35:0/632349143 (peer_addr_for_me v1:172.21.15.35:0/0) 2023-01-24T14:43:32.078 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.074+0000 7fa7f7fff700 1 -- 172.21.15.35:0/632349143 >> v1:172.21.15.27:6789/0 conn(0x7fa800113a00 legacy=0x7fa800113e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T14:43:32.078 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.074+0000 7fa7f7fff700 1 -- 172.21.15.35:0/632349143 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa80011a490 con 0x7fa800116770 2023-01-24T14:43:32.078 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.074+0000 7fa7ff7fe700 1 -- 172.21.15.35:0/632349143 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1479113919 0 0) 0x7fa800119160 con 0x7fa800113a00 2023-01-24T14:43:32.079 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.074+0000 7fa7f7fff700 1 --2- 172.21.15.35:0/632349143 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa800116770 0x7fa800118b90 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f770985a249cbab server_cookie=0 in_seq=0 out_seq=0 2023-01-24T14:43:32.079 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.075+0000 7fa7ff7fe700 1 -- 172.21.15.35:0/632349143 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fa7e8009cd0 con 0x7fa800116770 2023-01-24T14:43:32.079 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.075+0000 7fa7ff7fe700 1 -- 172.21.15.35:0/632349143 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7fa7e8009e30 con 0x7fa800116770 2023-01-24T14:43:32.079 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.075+0000 7fa7ff7fe700 1 -- 172.21.15.35:0/632349143 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fa7e800e360 con 0x7fa800116770 2023-01-24T14:43:32.079 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.075+0000 7fa806835700 1 -- 172.21.15.35:0/632349143 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa800116770 msgr2=0x7fa800118b90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T14:43:32.080 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.075+0000 7fa806835700 1 --2- 172.21.15.35:0/632349143 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa800116770 0x7fa800118b90 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T14:43:32.080 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.075+0000 7fa806835700 1 -- 172.21.15.35:0/632349143 shutdown_connections 2023-01-24T14:43:32.080 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.075+0000 7fa806835700 1 --2- 172.21.15.35:0/632349143 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa800116770 0x7fa800118b90 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T14:43:32.080 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.075+0000 7fa806835700 1 -- 172.21.15.35:0/632349143 >> 172.21.15.35:0/632349143 conn(0x7fa80006f300 msgr2=0x7fa800071720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T14:43:32.080 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.075+0000 7fa806835700 1 -- 172.21.15.35:0/632349143 shutdown_connections 2023-01-24T14:43:32.081 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.075+0000 7fa806835700 1 -- 172.21.15.35:0/632349143 wait complete. 2023-01-24T14:43:32.083 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.077+0000 7fa806835700 1 Processor -- start 2023-01-24T14:43:32.083 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.077+0000 7fa806835700 1 -- start start 2023-01-24T14:43:32.083 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.077+0000 7fa806835700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa800113a00 0x7fa8001a7df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T14:43:32.083 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.077+0000 7fa806835700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8001a83c0 con 0x7fa800113a00 2023-01-24T14:43:32.084 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.077+0000 7fa7fffff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa800113a00 0x7fa8001a7df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T14:43:32.084 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.077+0000 7fa7fffff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa800113a00 0x7fa8001a7df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.35:51342/0 (socket says 172.21.15.35:51342) 2023-01-24T14:43:32.084 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.077+0000 7fa7fffff700 1 -- 172.21.15.35:0/3060287683 learned_addr learned my addr 172.21.15.35:0/3060287683 (peer_addr_for_me v2:172.21.15.35:0/0) 2023-01-24T14:43:32.084 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.078+0000 7fa7fffff700 1 -- 172.21.15.35:0/3060287683 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa800119610 con 0x7fa800113a00 2023-01-24T14:43:32.084 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.078+0000 7fa7fffff700 1 --2- 172.21.15.35:0/3060287683 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa800113a00 0x7fa8001a7df0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T14:43:32.085 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.078+0000 7fa7fd7fa700 1 -- 172.21.15.35:0/3060287683 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fa7f0004b80 con 0x7fa800113a00 2023-01-24T14:43:32.085 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.078+0000 7fa7fd7fa700 1 -- 172.21.15.35:0/3060287683 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7fa7f0004ce0 con 0x7fa800113a00 2023-01-24T14:43:32.085 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.078+0000 7fa806835700 1 -- 172.21.15.35:0/3060287683 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa800119480 con 0x7fa800113a00 2023-01-24T14:43:32.085 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.078+0000 7fa7fd7fa700 1 -- 172.21.15.35:0/3060287683 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fa7f000c630 con 0x7fa800113a00 2023-01-24T14:43:32.085 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.078+0000 7fa806835700 1 -- 172.21.15.35:0/3060287683 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8001197a0 con 0x7fa800113a00 2023-01-24T14:43:32.086 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.078+0000 7fa7fd7fa700 1 -- 172.21.15.35:0/3060287683 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 1) v1 ==== 656+0+0 (crc 0 0 0) 0x7fa7f000cb10 con 0x7fa800113a00 2023-01-24T14:43:32.086 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.078+0000 7fa7fd7fa700 1 -- 172.21.15.35:0/3060287683 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (crc 0 0 0) 0x7fa7f000d150 con 0x7fa800113a00 2023-01-24T14:43:32.086 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.079+0000 7fa806835700 1 -- 172.21.15.35:0/3060287683 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa800061e90 con 0x7fa800113a00 2023-01-24T14:43:32.086 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.081+0000 7fa7fd7fa700 1 -- 172.21.15.35:0/3060287683 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71752 (crc 0 0 0) 0x7fa7f000e1c0 con 0x7fa800113a00 2023-01-24T14:43:32.167 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.166+0000 7fa806835700 1 -- 172.21.15.35:0/3060287683 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd new", "uuid": "ed53e235-e041-46ca-b000-fecf0c6ce916", "id": 0} v 0) v1 -- 0x7fa800116770 con 0x7fa800113a00 2023-01-24T14:43:32.172 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.169+0000 7fa7fd7fa700 1 -- 172.21.15.35:0/3060287683 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "ed53e235-e041-46ca-b000-fecf0c6ce916", "id": 0}]=0 v3) v1 ==== 112+0+2 (crc 0 0 0) 0x7fa7f000e360 con 0x7fa800113a00 2023-01-24T14:43:32.172 INFO:teuthology.orchestra.run.smithi035.stdout:0 2023-01-24T14:43:32.173 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.170+0000 7fa806835700 1 -- 172.21.15.35:0/3060287683 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa800113a00 msgr2=0x7fa8001a7df0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T14:43:32.173 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.170+0000 7fa806835700 1 --2- 172.21.15.35:0/3060287683 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa800113a00 0x7fa8001a7df0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T14:43:32.174 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.170+0000 7fa806835700 1 -- 172.21.15.35:0/3060287683 shutdown_connections 2023-01-24T14:43:32.174 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.170+0000 7fa806835700 1 --2- 172.21.15.35:0/3060287683 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa800113a00 0x7fa8001a7df0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T14:43:32.174 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.170+0000 7fa806835700 1 -- 172.21.15.35:0/3060287683 >> 172.21.15.35:0/3060287683 conn(0x7fa80006f300 msgr2=0x7fa80006f710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T14:43:32.174 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.171+0000 7fa806835700 1 -- 172.21.15.35:0/3060287683 shutdown_connections 2023-01-24T14:43:32.174 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.171+0000 7fa806835700 1 -- 172.21.15.35:0/3060287683 wait complete. 2023-01-24T14:43:32.188 DEBUG:teuthology.orchestra.run.smithi035:> sudo ceph --cluster ceph osd new bbbf53aa-af7f-499d-a2d5-3ac39dcc810b 1 2023-01-24T14:43:32.321 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.318+0000 7fd350963700 1 Processor -- start 2023-01-24T14:43:32.321 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.318+0000 7fd350963700 1 -- start start 2023-01-24T14:43:32.322 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.319+0000 7fd350963700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd34c119ad0 0x7fd34c11bef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T14:43:32.322 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.319+0000 7fd350963700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd34c11c520 con 0x7fd34c116d50 2023-01-24T14:43:32.322 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.319+0000 7fd350963700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd34c11d6e0 con 0x7fd34c119ad0 2023-01-24T14:43:32.322 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.319+0000 7fd343fff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd34c119ad0 0x7fd34c11bef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T14:43:32.322 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.319+0000 7fd34a59c700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fd34c116d50 0x7fd34c117160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.35:42404/0 (socket says 172.21.15.35:42404) 2023-01-24T14:43:32.323 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.319+0000 7fd34a59c700 1 -- 172.21.15.35:0/2428012607 learned_addr learned my addr 172.21.15.35:0/2428012607 (peer_addr_for_me v1:172.21.15.35:0/0) 2023-01-24T14:43:32.323 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.319+0000 7fd349d9b700 1 -- 172.21.15.35:0/2428012607 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2787551529 0 0) 0x7fd34c11c520 con 0x7fd34c116d50 2023-01-24T14:43:32.323 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.319+0000 7fd349d9b700 1 -- 172.21.15.35:0/2428012607 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd338003580 con 0x7fd34c116d50 2023-01-24T14:43:32.323 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.319+0000 7fd343fff700 1 -- 172.21.15.35:0/2428012607 >> v1:172.21.15.27:6789/0 conn(0x7fd34c116d50 legacy=0x7fd34c117160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T14:43:32.323 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.320+0000 7fd343fff700 1 -- 172.21.15.35:0/2428012607 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd34c11d850 con 0x7fd34c119ad0 2023-01-24T14:43:32.324 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.320+0000 7fd343fff700 1 --2- 172.21.15.35:0/2428012607 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd34c119ad0 0x7fd34c11bef0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c54dd5e05cf4cd9f server_cookie=0 in_seq=0 out_seq=0 2023-01-24T14:43:32.324 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.320+0000 7fd349d9b700 1 -- 172.21.15.35:0/2428012607 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fd334009c80 con 0x7fd34c119ad0 2023-01-24T14:43:32.324 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.320+0000 7fd349d9b700 1 -- 172.21.15.35:0/2428012607 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7fd334009de0 con 0x7fd34c119ad0 2023-01-24T14:43:32.324 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.320+0000 7fd349d9b700 1 -- 172.21.15.35:0/2428012607 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fd33400e360 con 0x7fd34c119ad0 2023-01-24T14:43:32.326 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.320+0000 7fd350963700 1 -- 172.21.15.35:0/2428012607 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd34c119ad0 msgr2=0x7fd34c11bef0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T14:43:32.327 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.321+0000 7fd350963700 1 --2- 172.21.15.35:0/2428012607 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd34c119ad0 0x7fd34c11bef0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T14:43:32.327 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.321+0000 7fd350963700 1 -- 172.21.15.35:0/2428012607 shutdown_connections 2023-01-24T14:43:32.327 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.321+0000 7fd350963700 1 --2- 172.21.15.35:0/2428012607 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd34c119ad0 0x7fd34c11bef0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T14:43:32.327 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.321+0000 7fd350963700 1 -- 172.21.15.35:0/2428012607 >> 172.21.15.35:0/2428012607 conn(0x7fd34c06d9e0 msgr2=0x7fd34c10ed80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T14:43:32.327 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.321+0000 7fd350963700 1 -- 172.21.15.35:0/2428012607 shutdown_connections 2023-01-24T14:43:32.328 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.321+0000 7fd350963700 1 -- 172.21.15.35:0/2428012607 wait complete. 2023-01-24T14:43:32.328 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.322+0000 7fd350963700 1 Processor -- start 2023-01-24T14:43:32.328 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.322+0000 7fd350963700 1 -- start start 2023-01-24T14:43:32.328 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.322+0000 7fd350963700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd34c116d50 0x7fd34c1ab170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T14:43:32.328 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.322+0000 7fd350963700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd34c1ab740 con 0x7fd34c116d50 2023-01-24T14:43:32.329 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.322+0000 7fd34a59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd34c116d50 0x7fd34c1ab170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T14:43:32.329 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.322+0000 7fd34a59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd34c116d50 0x7fd34c1ab170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.35:51362/0 (socket says 172.21.15.35:51362) 2023-01-24T14:43:32.329 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.322+0000 7fd34a59c700 1 -- 172.21.15.35:0/3639183654 learned_addr learned my addr 172.21.15.35:0/3639183654 (peer_addr_for_me v2:172.21.15.35:0/0) 2023-01-24T14:43:32.329 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.322+0000 7fd34a59c700 1 -- 172.21.15.35:0/3639183654 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd34c11c9d0 con 0x7fd34c116d50 2023-01-24T14:43:32.329 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.322+0000 7fd34a59c700 1 --2- 172.21.15.35:0/3639183654 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd34c116d50 0x7fd34c1ab170 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T14:43:32.330 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.323+0000 7fd342ffd700 1 -- 172.21.15.35:0/3639183654 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fd33c005a60 con 0x7fd34c116d50 2023-01-24T14:43:32.330 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.323+0000 7fd342ffd700 1 -- 172.21.15.35:0/3639183654 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7fd33c005bc0 con 0x7fd34c116d50 2023-01-24T14:43:32.330 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.323+0000 7fd350963700 1 -- 172.21.15.35:0/3639183654 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd34c11c840 con 0x7fd34c116d50 2023-01-24T14:43:32.330 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.323+0000 7fd342ffd700 1 -- 172.21.15.35:0/3639183654 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fd33c005e60 con 0x7fd34c116d50 2023-01-24T14:43:32.330 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.323+0000 7fd350963700 1 -- 172.21.15.35:0/3639183654 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd34c11cb60 con 0x7fd34c116d50 2023-01-24T14:43:32.330 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.323+0000 7fd342ffd700 1 -- 172.21.15.35:0/3639183654 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 1) v1 ==== 656+0+0 (crc 0 0 0) 0x7fd33c00cb30 con 0x7fd34c116d50 2023-01-24T14:43:32.331 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.323+0000 7fd342ffd700 1 -- 172.21.15.35:0/3639183654 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (crc 0 0 0) 0x7fd33c00f6d0 con 0x7fd34c116d50 2023-01-24T14:43:32.331 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.324+0000 7fd350963700 1 -- 172.21.15.35:0/3639183654 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd34c061e90 con 0x7fd34c116d50 2023-01-24T14:43:32.331 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.326+0000 7fd342ffd700 1 -- 172.21.15.35:0/3639183654 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71752 (crc 0 0 0) 0x7fd33c007160 con 0x7fd34c116d50 2023-01-24T14:43:32.412 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.411+0000 7fd350963700 1 -- 172.21.15.35:0/3639183654 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd new", "uuid": "bbbf53aa-af7f-499d-a2d5-3ac39dcc810b", "id": 1} v 0) v1 -- 0x7fd34c119ad0 con 0x7fd34c116d50 2023-01-24T14:43:32.415 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.414+0000 7fd342ffd700 1 -- 172.21.15.35:0/3639183654 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "bbbf53aa-af7f-499d-a2d5-3ac39dcc810b", "id": 1}]=0 v4) v1 ==== 112+0+2 (crc 0 0 0) 0x7fd33c0073d0 con 0x7fd34c116d50 2023-01-24T14:43:32.415 INFO:teuthology.orchestra.run.smithi035.stdout:1 2023-01-24T14:43:32.416 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.415+0000 7fd350963700 1 -- 172.21.15.35:0/3639183654 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd34c116d50 msgr2=0x7fd34c1ab170 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T14:43:32.416 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.415+0000 7fd350963700 1 --2- 172.21.15.35:0/3639183654 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd34c116d50 0x7fd34c1ab170 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T14:43:32.417 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.415+0000 7fd350963700 1 -- 172.21.15.35:0/3639183654 shutdown_connections 2023-01-24T14:43:32.417 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.415+0000 7fd350963700 1 --2- 172.21.15.35:0/3639183654 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd34c116d50 0x7fd34c1ab170 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T14:43:32.417 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.415+0000 7fd350963700 1 -- 172.21.15.35:0/3639183654 >> 172.21.15.35:0/3639183654 conn(0x7fd34c06d9e0 msgr2=0x7fd34c11ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T14:43:32.417 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.415+0000 7fd350963700 1 -- 172.21.15.35:0/3639183654 shutdown_connections 2023-01-24T14:43:32.417 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.415+0000 7fd350963700 1 -- 172.21.15.35:0/3639183654 wait complete. 2023-01-24T14:43:32.430 DEBUG:teuthology.orchestra.run.smithi035:> sudo ceph --cluster ceph osd new 14ce6465-900d-4d15-95d8-115fcb049310 2 2023-01-24T14:43:32.549 INFO:tasks.ceph.mgr.x.smithi035.stderr:2023-01-24T14:43:32.548+0000 7f6a691e91c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-01-24T14:43:32.562 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.560+0000 7f69fd3a7700 1 Processor -- start 2023-01-24T14:43:32.562 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.561+0000 7f69fd3a7700 1 -- start start 2023-01-24T14:43:32.563 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.561+0000 7f69fd3a7700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f69f81177d0 0x7f69f8119bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T14:43:32.563 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.561+0000 7f69fd3a7700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69f811a270 con 0x7f69f8113a20 2023-01-24T14:43:32.564 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.561+0000 7f69fd3a7700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69f811b430 con 0x7f69f81177d0 2023-01-24T14:43:32.564 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.561+0000 7f69f6ffd700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f69f8113a20 0x7f69f8113e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.35:42412/0 (socket says 172.21.15.35:42412) 2023-01-24T14:43:32.564 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.561+0000 7f69f67fc700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f69f81177d0 0x7f69f8119bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T14:43:32.564 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.561+0000 7f69f6ffd700 1 -- 172.21.15.35:0/1243056740 learned_addr learned my addr 172.21.15.35:0/1243056740 (peer_addr_for_me v1:172.21.15.35:0/0) 2023-01-24T14:43:32.564 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.562+0000 7f69f5ffb700 1 -- 172.21.15.35:0/1243056740 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2975036420 0 0) 0x7f69f811a270 con 0x7f69f8113a20 2023-01-24T14:43:32.565 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.562+0000 7f69f5ffb700 1 -- 172.21.15.35:0/1243056740 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69e0003580 con 0x7f69f8113a20 2023-01-24T14:43:32.565 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.562+0000 7f69f67fc700 1 -- 172.21.15.35:0/1243056740 >> v1:172.21.15.27:6789/0 conn(0x7f69f8113a20 legacy=0x7f69f8113e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T14:43:32.565 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.562+0000 7f69f67fc700 1 -- 172.21.15.35:0/1243056740 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69f811b5a0 con 0x7f69f81177d0 2023-01-24T14:43:32.565 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.562+0000 7f69f67fc700 1 --2- 172.21.15.35:0/1243056740 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f69f81177d0 0x7f69f8119bf0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d217641032fa849d server_cookie=0 in_seq=0 out_seq=0 2023-01-24T14:43:32.565 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.562+0000 7f69f5ffb700 1 -- 172.21.15.35:0/1243056740 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f69ec009c80 con 0x7f69f81177d0 2023-01-24T14:43:32.565 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.562+0000 7f69f5ffb700 1 -- 172.21.15.35:0/1243056740 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f69ec009de0 con 0x7f69f81177d0 2023-01-24T14:43:32.566 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.562+0000 7f69f5ffb700 1 -- 172.21.15.35:0/1243056740 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f69ec00e360 con 0x7f69f81177d0 2023-01-24T14:43:32.568 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.563+0000 7f69fd3a7700 1 -- 172.21.15.35:0/1243056740 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f69f81177d0 msgr2=0x7f69f8119bf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T14:43:32.568 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.563+0000 7f69fd3a7700 1 --2- 172.21.15.35:0/1243056740 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f69f81177d0 0x7f69f8119bf0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T14:43:32.569 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.563+0000 7f69fd3a7700 1 -- 172.21.15.35:0/1243056740 shutdown_connections 2023-01-24T14:43:32.569 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.563+0000 7f69fd3a7700 1 --2- 172.21.15.35:0/1243056740 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f69f81177d0 0x7f69f8119bf0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T14:43:32.569 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.563+0000 7f69fd3a7700 1 -- 172.21.15.35:0/1243056740 >> 172.21.15.35:0/1243056740 conn(0x7f69f806f300 msgr2=0x7f69f8071720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T14:43:32.569 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.563+0000 7f69fd3a7700 1 -- 172.21.15.35:0/1243056740 shutdown_connections 2023-01-24T14:43:32.569 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.563+0000 7f69fd3a7700 1 -- 172.21.15.35:0/1243056740 wait complete. 2023-01-24T14:43:32.570 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.564+0000 7f69fd3a7700 1 Processor -- start 2023-01-24T14:43:32.570 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.564+0000 7f69fd3a7700 1 -- start start 2023-01-24T14:43:32.570 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.564+0000 7f69fd3a7700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f69f8113a20 0x7f69f81a8f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T14:43:32.570 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.564+0000 7f69fd3a7700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69f81a9530 con 0x7f69f8113a20 2023-01-24T14:43:32.570 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.564+0000 7f69f6ffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f69f8113a20 0x7f69f81a8f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T14:43:32.570 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.564+0000 7f69f6ffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f69f8113a20 0x7f69f81a8f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.35:51376/0 (socket says 172.21.15.35:51376) 2023-01-24T14:43:32.571 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.564+0000 7f69f6ffd700 1 -- 172.21.15.35:0/1665848 learned_addr learned my addr 172.21.15.35:0/1665848 (peer_addr_for_me v2:172.21.15.35:0/0) 2023-01-24T14:43:32.571 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.564+0000 7f69f6ffd700 1 -- 172.21.15.35:0/1665848 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69f811a720 con 0x7f69f8113a20 2023-01-24T14:43:32.571 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.564+0000 7f69f6ffd700 1 --2- 172.21.15.35:0/1665848 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f69f8113a20 0x7f69f81a8f60 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T14:43:32.571 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.565+0000 7f69df7fe700 1 -- 172.21.15.35:0/1665848 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f69e8005a60 con 0x7f69f8113a20 2023-01-24T14:43:32.571 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.565+0000 7f69df7fe700 1 -- 172.21.15.35:0/1665848 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f69e8005bc0 con 0x7f69f8113a20 2023-01-24T14:43:32.572 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.565+0000 7f69df7fe700 1 -- 172.21.15.35:0/1665848 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f69e8005e60 con 0x7f69f8113a20 2023-01-24T14:43:32.572 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.565+0000 7f69fd3a7700 1 -- 172.21.15.35:0/1665848 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69f811a590 con 0x7f69f8113a20 2023-01-24T14:43:32.572 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.565+0000 7f69fd3a7700 1 -- 172.21.15.35:0/1665848 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69f811a8b0 con 0x7f69f8113a20 2023-01-24T14:43:32.572 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.565+0000 7f69df7fe700 1 -- 172.21.15.35:0/1665848 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 1) v1 ==== 656+0+0 (crc 0 0 0) 0x7f69e800cb30 con 0x7f69f8113a20 2023-01-24T14:43:32.572 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.565+0000 7f69df7fe700 1 -- 172.21.15.35:0/1665848 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (crc 0 0 0) 0x7f69e800cdc0 con 0x7f69f8113a20 2023-01-24T14:43:32.573 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.566+0000 7f69fd3a7700 1 -- 172.21.15.35:0/1665848 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69f8061e90 con 0x7f69f8113a20 2023-01-24T14:43:32.573 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.568+0000 7f69df7fe700 1 -- 172.21.15.35:0/1665848 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71752 (crc 0 0 0) 0x7f69e8007160 con 0x7f69f8113a20 2023-01-24T14:43:32.623 INFO:tasks.ceph.mgr.x.smithi035.stderr:2023-01-24T14:43:32.622+0000 7f6a691e91c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-01-24T14:43:32.654 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.653+0000 7f69fd3a7700 1 -- 172.21.15.35:0/1665848 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd new", "uuid": "14ce6465-900d-4d15-95d8-115fcb049310", "id": 2} v 0) v1 -- 0x7f69f8073850 con 0x7f69f8113a20 2023-01-24T14:43:32.656 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.655+0000 7f69df7fe700 1 -- 172.21.15.35:0/1665848 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "14ce6465-900d-4d15-95d8-115fcb049310", "id": 2}]=0 v5) v1 ==== 112+0+2 (crc 0 0 0) 0x7f69e80073d0 con 0x7f69f8113a20 2023-01-24T14:43:32.657 INFO:teuthology.orchestra.run.smithi035.stdout:2 2023-01-24T14:43:32.658 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.656+0000 7f69fd3a7700 1 -- 172.21.15.35:0/1665848 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f69f8113a20 msgr2=0x7f69f81a8f60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T14:43:32.658 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.656+0000 7f69fd3a7700 1 --2- 172.21.15.35:0/1665848 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f69f8113a20 0x7f69f81a8f60 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T14:43:32.658 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.656+0000 7f69fd3a7700 1 -- 172.21.15.35:0/1665848 shutdown_connections 2023-01-24T14:43:32.658 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.656+0000 7f69fd3a7700 1 --2- 172.21.15.35:0/1665848 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f69f8113a20 0x7f69f81a8f60 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T14:43:32.659 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.656+0000 7f69fd3a7700 1 -- 172.21.15.35:0/1665848 >> 172.21.15.35:0/1665848 conn(0x7f69f806f300 msgr2=0x7f69f81157a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T14:43:32.659 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.656+0000 7f69fd3a7700 1 -- 172.21.15.35:0/1665848 shutdown_connections 2023-01-24T14:43:32.659 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.656+0000 7f69fd3a7700 1 -- 172.21.15.35:0/1665848 wait complete. 2023-01-24T14:43:32.669 DEBUG:teuthology.orchestra.run.smithi035:> sudo ceph --cluster ceph osd new cf19c994-afc8-4b3f-b6a3-7a4df2680f89 3 2023-01-24T14:43:32.713 INFO:tasks.ceph.mgr.x.smithi035.stderr:2023-01-24T14:43:32.712+0000 7f6a691e91c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-01-24T14:43:32.803 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.801+0000 7ffb9344f700 1 Processor -- start 2023-01-24T14:43:32.803 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.801+0000 7ffb9344f700 1 -- start start 2023-01-24T14:43:32.803 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.802+0000 7ffb9344f700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ffb8c114b70 0x7ffb8c114f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T14:43:32.804 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.802+0000 7ffb9344f700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb8c10da50 con 0x7ffb8c10cf30 2023-01-24T14:43:32.804 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.802+0000 7ffb9344f700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb8c10dbe0 con 0x7ffb8c114b70 2023-01-24T14:43:32.804 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.802+0000 7ffb911eb700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ffb8c114b70 0x7ffb8c114f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T14:43:32.805 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.802+0000 7ffb909ea700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7ffb8c10cf30 0x7ffb8c10d340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.35:42420/0 (socket says 172.21.15.35:42420) 2023-01-24T14:43:32.805 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.802+0000 7ffb909ea700 1 -- 172.21.15.35:0/3588864094 learned_addr learned my addr 172.21.15.35:0/3588864094 (peer_addr_for_me v1:172.21.15.35:0/0) 2023-01-24T14:43:32.807 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.802+0000 7ffb911eb700 1 -- 172.21.15.35:0/3588864094 >> v1:172.21.15.27:6789/0 conn(0x7ffb8c10cf30 legacy=0x7ffb8c10d340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T14:43:32.808 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.802+0000 7ffb911eb700 1 -- 172.21.15.35:0/3588864094 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb8c10dd50 con 0x7ffb8c114b70 2023-01-24T14:43:32.808 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.802+0000 7ffb911eb700 1 --2- 172.21.15.35:0/3588864094 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffb8c114b70 0x7ffb8c114f80 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc40af0c04488bc2 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T14:43:32.808 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.802+0000 7ffb83fff700 1 -- 172.21.15.35:0/3588864094 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7ffb88009cd0 con 0x7ffb8c114b70 2023-01-24T14:43:32.809 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.803+0000 7ffb83fff700 1 -- 172.21.15.35:0/3588864094 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7ffb88009e30 con 0x7ffb8c114b70 2023-01-24T14:43:32.809 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.803+0000 7ffb83fff700 1 -- 172.21.15.35:0/3588864094 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7ffb8800e360 con 0x7ffb8c114b70 2023-01-24T14:43:32.809 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.803+0000 7ffb9344f700 1 -- 172.21.15.35:0/3588864094 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffb8c114b70 msgr2=0x7ffb8c114f80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T14:43:32.809 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.803+0000 7ffb9344f700 1 --2- 172.21.15.35:0/3588864094 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffb8c114b70 0x7ffb8c114f80 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T14:43:32.810 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.803+0000 7ffb9344f700 1 -- 172.21.15.35:0/3588864094 shutdown_connections 2023-01-24T14:43:32.810 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.803+0000 7ffb9344f700 1 --2- 172.21.15.35:0/3588864094 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffb8c114b70 0x7ffb8c114f80 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T14:43:32.810 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.803+0000 7ffb9344f700 1 -- 172.21.15.35:0/3588864094 >> 172.21.15.35:0/3588864094 conn(0x7ffb8c06ea80 msgr2=0x7ffb8c070ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T14:43:32.810 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.803+0000 7ffb9344f700 1 -- 172.21.15.35:0/3588864094 shutdown_connections 2023-01-24T14:43:32.810 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.803+0000 7ffb9344f700 1 -- 172.21.15.35:0/3588864094 wait complete. 2023-01-24T14:43:32.811 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.804+0000 7ffb9344f700 1 Processor -- start 2023-01-24T14:43:32.811 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.804+0000 7ffb9344f700 1 -- start start 2023-01-24T14:43:32.811 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.804+0000 7ffb9344f700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffb8c10cf30 0x7ffb8c10e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T14:43:32.811 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.804+0000 7ffb9344f700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb8c10e890 con 0x7ffb8c10cf30 2023-01-24T14:43:32.811 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.805+0000 7ffb911eb700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffb8c10cf30 0x7ffb8c10e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T14:43:32.812 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.805+0000 7ffb911eb700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffb8c10cf30 0x7ffb8c10e350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.35:51390/0 (socket says 172.21.15.35:51390) 2023-01-24T14:43:32.812 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.805+0000 7ffb911eb700 1 -- 172.21.15.35:0/4182585146 learned_addr learned my addr 172.21.15.35:0/4182585146 (peer_addr_for_me v2:172.21.15.35:0/0) 2023-01-24T14:43:32.812 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.805+0000 7ffb911eb700 1 -- 172.21.15.35:0/4182585146 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb8c11ab60 con 0x7ffb8c10cf30 2023-01-24T14:43:32.812 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.805+0000 7ffb911eb700 1 --2- 172.21.15.35:0/4182585146 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffb8c10cf30 0x7ffb8c10e350 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T14:43:32.813 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.805+0000 7ffb81ffb700 1 -- 172.21.15.35:0/4182585146 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7ffb88013740 con 0x7ffb8c10cf30 2023-01-24T14:43:32.813 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.805+0000 7ffb81ffb700 1 -- 172.21.15.35:0/4182585146 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7ffb880138a0 con 0x7ffb8c10cf30 2023-01-24T14:43:32.813 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.805+0000 7ffb9344f700 1 -- 172.21.15.35:0/4182585146 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb8c11a9d0 con 0x7ffb8c10cf30 2023-01-24T14:43:32.813 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.805+0000 7ffb81ffb700 1 -- 172.21.15.35:0/4182585146 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7ffb88013b40 con 0x7ffb8c10cf30 2023-01-24T14:43:32.813 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.806+0000 7ffb9344f700 1 -- 172.21.15.35:0/4182585146 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb8c11acf0 con 0x7ffb8c10cf30 2023-01-24T14:43:32.814 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.806+0000 7ffb81ffb700 1 -- 172.21.15.35:0/4182585146 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 1) v1 ==== 656+0+0 (crc 0 0 0) 0x7ffb880123f0 con 0x7ffb8c10cf30 2023-01-24T14:43:32.814 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.806+0000 7ffb81ffb700 1 -- 172.21.15.35:0/4182585146 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (crc 0 0 0) 0x7ffb88018040 con 0x7ffb8c10cf30 2023-01-24T14:43:32.814 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.807+0000 7ffb9344f700 1 -- 172.21.15.35:0/4182585146 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb8c061e90 con 0x7ffb8c10cf30 2023-01-24T14:43:32.814 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.809+0000 7ffb81ffb700 1 -- 172.21.15.35:0/4182585146 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71752 (crc 0 0 0) 0x7ffb88012680 con 0x7ffb8c10cf30 2023-01-24T14:43:32.895 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.894+0000 7ffb9344f700 1 -- 172.21.15.35:0/4182585146 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd new", "uuid": "cf19c994-afc8-4b3f-b6a3-7a4df2680f89", "id": 3} v 0) v1 -- 0x7ffb8c10ee50 con 0x7ffb8c10cf30 2023-01-24T14:43:32.902 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.901+0000 7ffb81ffb700 1 -- 172.21.15.35:0/4182585146 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "cf19c994-afc8-4b3f-b6a3-7a4df2680f89", "id": 3}]=0 v6) v1 ==== 112+0+2 (crc 0 0 0) 0x7ffb88012820 con 0x7ffb8c10cf30 2023-01-24T14:43:32.903 INFO:teuthology.orchestra.run.smithi035.stdout:3 2023-01-24T14:43:32.904 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.903+0000 7ffb9344f700 1 -- 172.21.15.35:0/4182585146 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffb8c10cf30 msgr2=0x7ffb8c10e350 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T14:43:32.904 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.903+0000 7ffb9344f700 1 --2- 172.21.15.35:0/4182585146 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffb8c10cf30 0x7ffb8c10e350 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T14:43:32.904 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.903+0000 7ffb9344f700 1 -- 172.21.15.35:0/4182585146 shutdown_connections 2023-01-24T14:43:32.905 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.903+0000 7ffb9344f700 1 --2- 172.21.15.35:0/4182585146 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffb8c10cf30 0x7ffb8c10e350 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T14:43:32.905 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.903+0000 7ffb9344f700 1 -- 172.21.15.35:0/4182585146 >> 172.21.15.35:0/4182585146 conn(0x7ffb8c06ea80 msgr2=0x7ffb8c113460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T14:43:32.905 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.903+0000 7ffb9344f700 1 -- 172.21.15.35:0/4182585146 shutdown_connections 2023-01-24T14:43:32.905 INFO:teuthology.orchestra.run.smithi035.stderr:2023-01-24T14:43:32.903+0000 7ffb9344f700 1 -- 172.21.15.35:0/4182585146 wait complete. 2023-01-24T14:43:32.911 INFO:tasks.ceph.mgr.x.smithi035.stderr:2023-01-24T14:43:32.910+0000 7f6a691e91c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-01-24T14:43:32.916 INFO:tasks.ceph.osd.0:Restarting daemon 2023-01-24T14:43:32.916 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2023-01-24T14:43:32.919 INFO:tasks.ceph.osd.0:Started 2023-01-24T14:43:32.919 INFO:tasks.ceph.osd.1:Restarting daemon 2023-01-24T14:43:32.919 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2023-01-24T14:43:32.922 INFO:tasks.ceph.osd.1:Started 2023-01-24T14:43:32.922 INFO:tasks.ceph.osd.2:Restarting daemon 2023-01-24T14:43:32.922 DEBUG:teuthology.orchestra.run.smithi035:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2023-01-24T14:43:32.924 INFO:tasks.ceph.osd.2:Started 2023-01-24T14:43:32.925 INFO:tasks.ceph.osd.3:Restarting daemon 2023-01-24T14:43:32.925 DEBUG:teuthology.orchestra.run.smithi035:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2023-01-24T14:43:32.927 INFO:tasks.ceph.osd.3:Started 2023-01-24T14:43:32.928 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-01-24T14:43:33.092 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:33.090+0000 7f1fe6025700 1 Processor -- start 2023-01-24T14:43:33.092 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:33.090+0000 7f1fe6025700 1 -- start start 2023-01-24T14:43:33.094 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:33.091+0000 7f1fe6025700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f1fe01329f0 0x7f1fe009ff40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T14:43:33.094 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:33.091+0000 7f1fe6025700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fe009a400 con 0x7f1fe01325e0 2023-01-24T14:43:33.094 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:33.091+0000 7f1fe6025700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1fe00a0ce0 con 0x7f1fe01329f0 2023-01-24T14:43:33.095 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:33.091+0000 7f1fdeffd700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f1fe01329f0 0x7f1fe009ff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T14:43:33.095 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:33.091+0000 7f1fdf7fe700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f1fe01325e0 0x7f1fe009f830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:52050/0 (socket says 172.21.15.27:52050) 2023-01-24T14:43:33.095 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:33.091+0000 7f1fdf7fe700 1 -- 172.21.15.27:0/2701598486 learned_addr learned my addr 172.21.15.27:0/2701598486 (peer_addr_for_me v1:172.21.15.27:0/0) 2023-01-24T14:43:33.095 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:33.091+0000 7f1fdeffd700 1 -- 172.21.15.27:0/2701598486 >> v1:172.21.15.27:6789/0 conn(0x7f1fe01325e0 legacy=0x7f1fe009f830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T14:43:33.095 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:33.091+0000 7f1fde7fc700 1 -- 172.21.15.27:0/2701598486 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 819350612 0 0) 0x7f1fe009a400 con 0x7f1fe01325e0 2023-01-24T14:43:33.096 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:33.091+0000 7f1fdeffd700 1 -- 172.21.15.27:0/2701598486 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fe0099f70 con 0x7f1fe01329f0 2023-01-24T14:43:33.096 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:33.091+0000 7f1fdeffd700 1 --2- 172.21.15.27:0/2701598486 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1fe01329f0 0x7f1fe009ff40 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de080e4dcda980c9 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T14:43:33.096 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:33.092+0000 7f1fde7fc700 1 -- 172.21.15.27:0/2701598486 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f1fd4009cd0 con 0x7f1fe01329f0 2023-01-24T14:43:33.096 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:33.092+0000 7f1fde7fc700 1 -- 172.21.15.27:0/2701598486 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f1fd4009e30 con 0x7f1fe01329f0 2023-01-24T14:43:33.096 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:33.092+0000 7f1fde7fc700 1 -- 172.21.15.27:0/2701598486 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f1fd400e360 con 0x7f1fe01329f0 2023-01-24T14:43:33.097 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:33.092+0000 7f1fe6025700 1 -- 172.21.15.27:0/2701598486 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1fe01329f0 msgr2=0x7f1fe009ff40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T14:43:33.097 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:33.092+0000 7f1fe6025700 1 --2- 172.21.15.27:0/2701598486 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1fe01329f0 0x7f1fe009ff40 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T14:43:33.097 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:33.093+0000 7f1fe6025700 1 -- 172.21.15.27:0/2701598486 shutdown_connections 2023-01-24T14:43:33.097 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:33.093+0000 7f1fe6025700 1 --2- 172.21.15.27:0/2701598486 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1fe01329f0 0x7f1fe009ff40 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T14:43:33.097 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:33.093+0000 7f1fe6025700 1 -- 172.21.15.27:0/2701598486 >> 172.21.15.27:0/2701598486 conn(0x7f1fe00a4bf0 msgr2=0x7f1fe00a5000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T14:43:33.097 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:33.093+0000 7f1fe6025700 1 -- 172.21.15.27:0/2701598486 shutdown_connections 2023-01-24T14:43:33.098 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:33.093+0000 7f1fe6025700 1 -- 172.21.15.27:0/2701598486 wait complete. 2023-01-24T14:43:33.099 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:33.094+0000 7f1fe6025700 1 Processor -- start 2023-01-24T14:43:33.100 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:33.094+0000 7f1fe6025700 1 -- start start 2023-01-24T14:43:33.100 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:33.095+0000 7f1fe6025700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1fe01325e0 0x7f1fe011e210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T14:43:33.100 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:33.095+0000 7f1fe6025700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fe011d780 con 0x7f1fe01325e0 2023-01-24T14:43:33.100 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:33.095+0000 7f1fdf7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1fe01325e0 0x7f1fe011e210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T14:43:33.101 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:33.095+0000 7f1fdf7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1fe01325e0 0x7f1fe011e210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56942/0 (socket says 172.21.15.27:56942) 2023-01-24T14:43:33.101 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:33.095+0000 7f1fdf7fe700 1 -- 172.21.15.27:0/309206334 learned_addr learned my addr 172.21.15.27:0/309206334 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-01-24T14:43:33.101 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:33.096+0000 7f1fdf7fe700 1 -- 172.21.15.27:0/309206334 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fe0099440 con 0x7f1fe01325e0 2023-01-24T14:43:33.101 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:33.096+0000 7f1fdf7fe700 1 --2- 172.21.15.27:0/309206334 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1fe01325e0 0x7f1fe011e210 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T14:43:33.101 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:33.096+0000 7f1fc7fff700 1 -- 172.21.15.27:0/309206334 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f1fd0004b80 con 0x7f1fe01325e0 2023-01-24T14:43:33.102 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:33.096+0000 7f1fc7fff700 1 -- 172.21.15.27:0/309206334 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f1fd0004ce0 con 0x7f1fe01325e0 2023-01-24T14:43:33.102 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:33.096+0000 7f1fc7fff700 1 -- 172.21.15.27:0/309206334 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f1fd000c550 con 0x7f1fe01325e0 2023-01-24T14:43:33.102 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:33.096+0000 7f1fe6025700 1 -- 172.21.15.27:0/309206334 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fe00983c0 con 0x7f1fe01325e0 2023-01-24T14:43:33.102 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:33.096+0000 7f1fe6025700 1 -- 172.21.15.27:0/309206334 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fe0132e00 con 0x7f1fe01325e0 2023-01-24T14:43:33.102 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:33.097+0000 7f1fc7fff700 1 -- 172.21.15.27:0/309206334 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 1) v1 ==== 656+0+0 (crc 0 0 0) 0x7f1fd000c9a0 con 0x7f1fe01325e0 2023-01-24T14:43:33.103 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:33.097+0000 7f1fc7fff700 1 -- 172.21.15.27:0/309206334 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (crc 0 0 0) 0x7f1fd000d150 con 0x7f1fe01325e0 2023-01-24T14:43:33.103 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:33.098+0000 7f1fe6025700 1 -- 172.21.15.27:0/309206334 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1fe011dd50 con 0x7f1fe01325e0 2023-01-24T14:43:33.103 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:33.100+0000 7f1fc7fff700 1 -- 172.21.15.27:0/309206334 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71752 (crc 0 0 0) 0x7f1fd000e1c0 con 0x7f1fe01325e0 2023-01-24T14:43:33.182 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:33.181+0000 7f1fe6025700 1 -- 172.21.15.27:0/309206334 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1fe0074980 con 0x7f1fe01325e0 2023-01-24T14:43:33.183 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:33.182+0000 7f1fc7fff700 1 -- 172.21.15.27:0/309206334 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v6) v1 ==== 74+0+4130 (crc 0 0 0) 0x7f1fd000e360 con 0x7f1fe01325e0 2023-01-24T14:43:33.183 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-24T14:43:33.184 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":6,"fsid":"5e79a879-b90d-445c-b439-aec3d80a8b8a","created":"2023-01-24T14:43:31.588819+0000","modified":"2023-01-24T14:43:32.895676+0000","last_up_change":"0.000000","last_in_change":"2023-01-24T14:43:32.895676+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","pools":[],"osds":[{"osd":0,"uuid":"ed53e235-e041-46ca-b000-fecf0c6ce916","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bbbf53aa-af7f-499d-a2d5-3ac39dcc810b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"14ce6465-900d-4d15-95d8-115fcb049310","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cf19c994-afc8-4b3f-b6a3-7a4df2680f89","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-01-24T14:43:33.185 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:33.183+0000 7f1fe6025700 1 -- 172.21.15.27:0/309206334 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1fe01325e0 msgr2=0x7f1fe011e210 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T14:43:33.185 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:33.183+0000 7f1fe6025700 1 --2- 172.21.15.27:0/309206334 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1fe01325e0 0x7f1fe011e210 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T14:43:33.185 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:33.183+0000 7f1fe6025700 1 -- 172.21.15.27:0/309206334 shutdown_connections 2023-01-24T14:43:33.186 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:33.183+0000 7f1fe6025700 1 --2- 172.21.15.27:0/309206334 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1fe01325e0 0x7f1fe011e210 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T14:43:33.186 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:33.183+0000 7f1fe6025700 1 -- 172.21.15.27:0/309206334 >> 172.21.15.27:0/309206334 conn(0x7f1fe00a4bf0 msgr2=0x7f1fe01ba070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T14:43:33.186 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:33.183+0000 7f1fe6025700 1 -- 172.21.15.27:0/309206334 shutdown_connections 2023-01-24T14:43:33.186 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:33.183+0000 7f1fe6025700 1 -- 172.21.15.27:0/309206334 wait complete. 2023-01-24T14:43:33.202 INFO:tasks.ceph.ceph_manager.ceph:[] 2023-01-24T14:43:33.203 INFO:tasks.ceph:Waiting for OSDs to come up 2023-01-24T14:43:33.606 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-01-24T14:43:33.742 INFO:teuthology.misc.health.smithi027.stderr:2023-01-24T14:43:33.740+0000 7f900815a700 1 Processor -- start 2023-01-24T14:43:33.742 INFO:teuthology.misc.health.smithi027.stderr:2023-01-24T14:43:33.740+0000 7f900815a700 1 -- start start 2023-01-24T14:43:33.744 INFO:teuthology.misc.health.smithi027.stderr:2023-01-24T14:43:33.742+0000 7f900815a700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f90000a02a0 0x7f9000133430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T14:43:33.744 INFO:teuthology.misc.health.smithi027.stderr:2023-01-24T14:43:33.742+0000 7f900815a700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90000720d0 con 0x7f900009f780 2023-01-24T14:43:33.744 INFO:teuthology.misc.health.smithi027.stderr:2023-01-24T14:43:33.742+0000 7f900815a700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f900009da80 con 0x7f90000a02a0 2023-01-24T14:43:33.744 INFO:teuthology.misc.health.smithi027.stderr:2023-01-24T14:43:33.742+0000 7f90056f5700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f90000a02a0 0x7f9000133430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T14:43:33.745 INFO:teuthology.misc.health.smithi027.stderr:2023-01-24T14:43:33.742+0000 7f9005ef6700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f900009f780 0x7f900009fb90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:52060/0 (socket says 172.21.15.27:52060) 2023-01-24T14:43:33.745 INFO:teuthology.misc.health.smithi027.stderr:2023-01-24T14:43:33.742+0000 7f9005ef6700 1 -- 172.21.15.27:0/4248200751 learned_addr learned my addr 172.21.15.27:0/4248200751 (peer_addr_for_me v1:172.21.15.27:0/0) 2023-01-24T14:43:33.745 INFO:teuthology.misc.health.smithi027.stderr:2023-01-24T14:43:33.742+0000 7f90056f5700 1 -- 172.21.15.27:0/4248200751 >> v1:172.21.15.27:6789/0 conn(0x7f900009f780 legacy=0x7f900009fb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T14:43:33.745 INFO:teuthology.misc.health.smithi027.stderr:2023-01-24T14:43:33.742+0000 7f90056f5700 1 -- 172.21.15.27:0/4248200751 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f900009dd20 con 0x7f90000a02a0 2023-01-24T14:43:33.745 INFO:teuthology.misc.health.smithi027.stderr:2023-01-24T14:43:33.742+0000 7f9004ef4700 1 -- 172.21.15.27:0/4248200751 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4120202789 0 0) 0x7f90000720d0 con 0x7f900009f780 2023-01-24T14:43:33.746 INFO:teuthology.misc.health.smithi027.stderr:2023-01-24T14:43:33.743+0000 7f90056f5700 1 --2- 172.21.15.27:0/4248200751 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f90000a02a0 0x7f9000133430 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=409635da88fba982 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T14:43:33.746 INFO:teuthology.misc.health.smithi027.stderr:2023-01-24T14:43:33.743+0000 7f9004ef4700 1 -- 172.21.15.27:0/4248200751 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f8ff0009cd0 con 0x7f90000a02a0 2023-01-24T14:43:33.746 INFO:teuthology.misc.health.smithi027.stderr:2023-01-24T14:43:33.743+0000 7f9004ef4700 1 -- 172.21.15.27:0/4248200751 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f8ff0009e30 con 0x7f90000a02a0 2023-01-24T14:43:33.746 INFO:teuthology.misc.health.smithi027.stderr:2023-01-24T14:43:33.743+0000 7f9004ef4700 1 -- 172.21.15.27:0/4248200751 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f8ff000e360 con 0x7f90000a02a0 2023-01-24T14:43:33.747 INFO:teuthology.misc.health.smithi027.stderr:2023-01-24T14:43:33.744+0000 7f900815a700 1 -- 172.21.15.27:0/4248200751 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f90000a02a0 msgr2=0x7f9000133430 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T14:43:33.747 INFO:teuthology.misc.health.smithi027.stderr:2023-01-24T14:43:33.744+0000 7f900815a700 1 --2- 172.21.15.27:0/4248200751 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f90000a02a0 0x7f9000133430 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T14:43:33.748 INFO:teuthology.misc.health.smithi027.stderr:2023-01-24T14:43:33.744+0000 7f900815a700 1 -- 172.21.15.27:0/4248200751 shutdown_connections 2023-01-24T14:43:33.748 INFO:teuthology.misc.health.smithi027.stderr:2023-01-24T14:43:33.744+0000 7f900815a700 1 --2- 172.21.15.27:0/4248200751 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f90000a02a0 0x7f9000133430 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T14:43:33.748 INFO:teuthology.misc.health.smithi027.stderr:2023-01-24T14:43:33.744+0000 7f900815a700 1 -- 172.21.15.27:0/4248200751 >> 172.21.15.27:0/4248200751 conn(0x7f90000a14d0 msgr2=0x7f900009d370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T14:43:33.748 INFO:teuthology.misc.health.smithi027.stderr:2023-01-24T14:43:33.744+0000 7f900815a700 1 -- 172.21.15.27:0/4248200751 shutdown_connections 2023-01-24T14:43:33.748 INFO:teuthology.misc.health.smithi027.stderr:2023-01-24T14:43:33.744+0000 7f900815a700 1 -- 172.21.15.27:0/4248200751 wait complete. 2023-01-24T14:43:33.749 INFO:teuthology.misc.health.smithi027.stderr:2023-01-24T14:43:33.746+0000 7f900815a700 1 Processor -- start 2023-01-24T14:43:33.749 INFO:teuthology.misc.health.smithi027.stderr:2023-01-24T14:43:33.746+0000 7f900815a700 1 -- start start 2023-01-24T14:43:33.751 INFO:teuthology.misc.health.smithi027.stderr:2023-01-24T14:43:33.747+0000 7f900815a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f900009f780 0x7f9000121bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T14:43:33.751 INFO:teuthology.misc.health.smithi027.stderr:2023-01-24T14:43:33.747+0000 7f900815a700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90001238b0 con 0x7f900009f780 2023-01-24T14:43:33.751 INFO:teuthology.misc.health.smithi027.stderr:2023-01-24T14:43:33.747+0000 7f9005ef6700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f900009f780 0x7f9000121bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T14:43:33.751 INFO:teuthology.misc.health.smithi027.stderr:2023-01-24T14:43:33.747+0000 7f9005ef6700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f900009f780 0x7f9000121bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56954/0 (socket says 172.21.15.27:56954) 2023-01-24T14:43:33.751 INFO:teuthology.misc.health.smithi027.stderr:2023-01-24T14:43:33.747+0000 7f9005ef6700 1 -- 172.21.15.27:0/1974181441 learned_addr learned my addr 172.21.15.27:0/1974181441 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-01-24T14:43:33.752 INFO:teuthology.misc.health.smithi027.stderr:2023-01-24T14:43:33.748+0000 7f9005ef6700 1 -- 172.21.15.27:0/1974181441 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9000099f50 con 0x7f900009f780 2023-01-24T14:43:33.752 INFO:teuthology.misc.health.smithi027.stderr:2023-01-24T14:43:33.748+0000 7f9005ef6700 1 --2- 172.21.15.27:0/1974181441 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f900009f780 0x7f9000121bb0 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T14:43:33.752 INFO:teuthology.misc.health.smithi027.stderr:2023-01-24T14:43:33.748+0000 7f8ff67fc700 1 -- 172.21.15.27:0/1974181441 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f8ffc00d890 con 0x7f900009f780 2023-01-24T14:43:33.752 INFO:teuthology.misc.health.smithi027.stderr:2023-01-24T14:43:33.748+0000 7f8ff67fc700 1 -- 172.21.15.27:0/1974181441 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f8ffc00d9f0 con 0x7f900009f780 2023-01-24T14:43:33.752 INFO:teuthology.misc.health.smithi027.stderr:2023-01-24T14:43:33.748+0000 7f900815a700 1 -- 172.21.15.27:0/1974181441 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f900009a2f0 con 0x7f900009f780 2023-01-24T14:43:33.753 INFO:teuthology.misc.health.smithi027.stderr:2023-01-24T14:43:33.748+0000 7f8ff67fc700 1 -- 172.21.15.27:0/1974181441 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f8ffc00dc90 con 0x7f900009f780 2023-01-24T14:43:33.753 INFO:teuthology.misc.health.smithi027.stderr:2023-01-24T14:43:33.748+0000 7f900815a700 1 -- 172.21.15.27:0/1974181441 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f900009a600 con 0x7f900009f780 2023-01-24T14:43:33.753 INFO:teuthology.misc.health.smithi027.stderr:2023-01-24T14:43:33.749+0000 7f8ff67fc700 1 -- 172.21.15.27:0/1974181441 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 1) v1 ==== 656+0+0 (crc 0 0 0) 0x7f8ffc010650 con 0x7f900009f780 2023-01-24T14:43:33.753 INFO:teuthology.misc.health.smithi027.stderr:2023-01-24T14:43:33.749+0000 7f8ff67fc700 1 -- 172.21.15.27:0/1974181441 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (crc 0 0 0) 0x7f8ffc00c040 con 0x7f900009f780 2023-01-24T14:43:33.753 INFO:teuthology.misc.health.smithi027.stderr:2023-01-24T14:43:33.749+0000 7f900815a700 1 -- 172.21.15.27:0/1974181441 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9000061e90 con 0x7f900009f780 2023-01-24T14:43:33.754 INFO:teuthology.misc.health.smithi027.stderr:2023-01-24T14:43:33.752+0000 7f8ff67fc700 1 -- 172.21.15.27:0/1974181441 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71752 (crc 0 0 0) 0x7f8ffc00a040 con 0x7f900009f780 2023-01-24T14:43:33.819 INFO:tasks.ceph.osd.1.smithi027.stderr:WARNING: debug mode. Not for benchmarking or production 2023-01-24T14:43:33.820 INFO:tasks.ceph.osd.1.smithi027.stderr:INFO 2023-01-24 14:43:33,820 seastar - Reactor backend: epoll 2023-01-24T14:43:33.822 INFO:tasks.ceph.osd.0.smithi027.stderr:WARNING: debug mode. Not for benchmarking or production 2023-01-24T14:43:33.824 INFO:tasks.ceph.osd.0.smithi027.stderr:INFO 2023-01-24 14:43:33,822 seastar - Reactor backend: epoll 2023-01-24T14:43:33.825 INFO:teuthology.misc.health.smithi027.stderr:2023-01-24T14:43:33.823+0000 7f900815a700 1 -- 172.21.15.27:0/1974181441 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9000067090 con 0x7f900009f780 2023-01-24T14:43:33.825 INFO:teuthology.misc.health.smithi027.stderr:2023-01-24T14:43:33.823+0000 7f8ff67fc700 1 -- 172.21.15.27:0/1974181441 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v6) v1 ==== 74+0+4130 (crc 0 0 0) 0x7f8ffc00de40 con 0x7f900009f780 2023-01-24T14:43:33.825 INFO:teuthology.misc.health.smithi027.stdout: 2023-01-24T14:43:33.826 INFO:teuthology.misc.health.smithi027.stdout:{"epoch":6,"fsid":"5e79a879-b90d-445c-b439-aec3d80a8b8a","created":"2023-01-24T14:43:31.588819+0000","modified":"2023-01-24T14:43:32.895676+0000","last_up_change":"0.000000","last_in_change":"2023-01-24T14:43:32.895676+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","pools":[],"osds":[{"osd":0,"uuid":"ed53e235-e041-46ca-b000-fecf0c6ce916","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bbbf53aa-af7f-499d-a2d5-3ac39dcc810b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"14ce6465-900d-4d15-95d8-115fcb049310","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cf19c994-afc8-4b3f-b6a3-7a4df2680f89","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-01-24T14:43:33.827 INFO:teuthology.misc.health.smithi027.stderr:2023-01-24T14:43:33.824+0000 7f900815a700 1 -- 172.21.15.27:0/1974181441 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f900009f780 msgr2=0x7f9000121bb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T14:43:33.827 INFO:teuthology.misc.health.smithi027.stderr:2023-01-24T14:43:33.824+0000 7f900815a700 1 --2- 172.21.15.27:0/1974181441 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f900009f780 0x7f9000121bb0 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T14:43:33.827 INFO:teuthology.misc.health.smithi027.stderr:2023-01-24T14:43:33.824+0000 7f900815a700 1 -- 172.21.15.27:0/1974181441 shutdown_connections 2023-01-24T14:43:33.827 INFO:teuthology.misc.health.smithi027.stderr:2023-01-24T14:43:33.824+0000 7f900815a700 1 --2- 172.21.15.27:0/1974181441 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f900009f780 0x7f9000121bb0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T14:43:33.828 INFO:teuthology.misc.health.smithi027.stderr:2023-01-24T14:43:33.824+0000 7f900815a700 1 -- 172.21.15.27:0/1974181441 >> 172.21.15.27:0/1974181441 conn(0x7f90000a14d0 msgr2=0x7f9000116ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T14:43:33.828 INFO:teuthology.misc.health.smithi027.stderr:2023-01-24T14:43:33.824+0000 7f900815a700 1 -- 172.21.15.27:0/1974181441 shutdown_connections 2023-01-24T14:43:33.828 INFO:teuthology.misc.health.smithi027.stderr:2023-01-24T14:43:33.824+0000 7f900815a700 1 -- 172.21.15.27:0/1974181441 wait complete. 2023-01-24T14:43:33.833 INFO:tasks.ceph.mgr.x.smithi035.stderr:2023-01-24T14:43:33.831+0000 7f6a691e91c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-01-24T14:43:33.842 DEBUG:teuthology.misc:0 of 4 OSDs are up 2023-01-24T14:43:33.867 INFO:tasks.ceph.osd.1.smithi027.stderr:INFO 2023-01-24 14:43:33,866 [shard 0] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-01-24T14:43:33.867 INFO:tasks.ceph.osd.1.smithi027.stderr:INFO 2023-01-24 14:43:33,866 [shard 0] seastar - IO queue uses 0.75ms latency goal for device 0 2023-01-24T14:43:33.867 INFO:tasks.ceph.osd.1.smithi027.stderr:INFO 2023-01-24 14:43:33,866 [shard 0] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-01-24T14:43:33.867 INFO:tasks.ceph.osd.1.smithi027.stderr:INFO 2023-01-24 14:43:33,866 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-01-24T14:43:33.869 INFO:tasks.ceph.osd.1.smithi027.stderr:==102699==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-01-24T14:43:33.875 INFO:tasks.ceph.osd.1.smithi027.stderr:==102699==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7ffe4cae4000; bottom 0x7f059c133000; size: 0x00f8b09b1000 (1068114841600) 2023-01-24T14:43:33.876 INFO:tasks.ceph.osd.1.smithi027.stderr:False positive error reports may follow 2023-01-24T14:43:33.876 INFO:tasks.ceph.osd.1.smithi027.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-01-24T14:43:33.880 INFO:tasks.ceph.osd.3.smithi035.stderr:WARNING: debug mode. Not for benchmarking or production 2023-01-24T14:43:33.880 INFO:tasks.ceph.osd.3.smithi035.stderr:INFO 2023-01-24 14:43:33,880 seastar - Reactor backend: epoll 2023-01-24T14:43:33.883 INFO:tasks.ceph.osd.0.smithi027.stderr:INFO 2023-01-24 14:43:33,882 [shard 0] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-01-24T14:43:33.883 INFO:tasks.ceph.osd.0.smithi027.stderr:INFO 2023-01-24 14:43:33,882 [shard 0] seastar - IO queue uses 0.75ms latency goal for device 0 2023-01-24T14:43:33.883 INFO:tasks.ceph.osd.0.smithi027.stderr:INFO 2023-01-24 14:43:33,882 [shard 0] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-01-24T14:43:33.883 INFO:tasks.ceph.osd.0.smithi027.stderr:INFO 2023-01-24 14:43:33,882 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-01-24T14:43:33.885 INFO:tasks.ceph.osd.0.smithi027.stderr:==102700==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-01-24T14:43:33.889 INFO:tasks.ceph.osd.2.smithi035.stderr:WARNING: debug mode. Not for benchmarking or production 2023-01-24T14:43:33.889 INFO:tasks.ceph.osd.2.smithi035.stderr:INFO 2023-01-24 14:43:33,889 seastar - Reactor backend: epoll 2023-01-24T14:43:33.891 INFO:tasks.ceph.osd.0.smithi027.stderr:==102700==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7fff497f3000; bottom 0x7f65264e7000; size: 0x009a2330c000 (662015361024) 2023-01-24T14:43:33.892 INFO:tasks.ceph.osd.0.smithi027.stderr:False positive error reports may follow 2023-01-24T14:43:33.892 INFO:tasks.ceph.osd.0.smithi027.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-01-24T14:43:33.918 INFO:tasks.ceph.osd.2.smithi035.stderr:INFO 2023-01-24 14:43:33,918 [shard 0] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-01-24T14:43:33.919 INFO:tasks.ceph.osd.2.smithi035.stderr:INFO 2023-01-24 14:43:33,918 [shard 0] seastar - IO queue uses 0.75ms latency goal for device 0 2023-01-24T14:43:33.919 INFO:tasks.ceph.osd.2.smithi035.stderr:INFO 2023-01-24 14:43:33,918 [shard 0] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-01-24T14:43:33.919 INFO:tasks.ceph.osd.2.smithi035.stderr:INFO 2023-01-24 14:43:33,918 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-01-24T14:43:33.920 INFO:tasks.ceph.osd.2.smithi035.stderr:==102234==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-01-24T14:43:33.922 INFO:tasks.ceph.osd.3.smithi035.stderr:INFO 2023-01-24 14:43:33,921 [shard 0] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-01-24T14:43:33.922 INFO:tasks.ceph.osd.3.smithi035.stderr:INFO 2023-01-24 14:43:33,921 [shard 0] seastar - IO queue uses 0.75ms latency goal for device 0 2023-01-24T14:43:33.923 INFO:tasks.ceph.osd.3.smithi035.stderr:INFO 2023-01-24 14:43:33,921 [shard 0] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-01-24T14:43:33.923 INFO:tasks.ceph.osd.3.smithi035.stderr:INFO 2023-01-24 14:43:33,921 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-01-24T14:43:33.924 INFO:tasks.ceph.osd.3.smithi035.stderr:==102233==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-01-24T14:43:33.931 INFO:tasks.ceph.osd.3.smithi035.stderr:==102233==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7fffaea9c000; bottom 0x7f7469931000; size: 0x008b4516b000 (598159568896) 2023-01-24T14:43:33.931 INFO:tasks.ceph.osd.3.smithi035.stderr:False positive error reports may follow 2023-01-24T14:43:33.931 INFO:tasks.ceph.osd.3.smithi035.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-01-24T14:43:33.946 INFO:tasks.ceph.osd.1.smithi027.stderr:INFO 2023-01-24 14:43:33,945 [shard 0] osd - populating config from monitor 2023-01-24T14:43:33.946 INFO:tasks.ceph.osd.0.smithi027.stderr:INFO 2023-01-24 14:43:33,946 [shard 0] osd - populating config from monitor 2023-01-24T14:43:33.948 INFO:tasks.ceph.osd.0.smithi027.stderr:INFO 2023-01-24 14:43:33,947 [shard 0] auth - AuthRegistry(0x7f65264ab6b0) adding auth protocol: cephx 2023-01-24T14:43:33.948 INFO:tasks.ceph.osd.0.smithi027.stderr:INFO 2023-01-24 14:43:33,947 [shard 0] auth - AuthRegistry(0x7f65264ab6b0) adding auth protocol: cephx 2023-01-24T14:43:33.948 INFO:tasks.ceph.osd.0.smithi027.stderr:INFO 2023-01-24 14:43:33,947 [shard 0] auth - AuthRegistry(0x7f65264ab6b0) adding auth protocol: cephx 2023-01-24T14:43:33.948 INFO:tasks.ceph.osd.0.smithi027.stderr:INFO 2023-01-24 14:43:33,947 [shard 0] auth - AuthRegistry(0x7f65264ab6b0) adding con mode: crc 2023-01-24T14:43:33.948 INFO:tasks.ceph.osd.0.smithi027.stderr:INFO 2023-01-24 14:43:33,947 [shard 0] auth - AuthRegistry(0x7f65264ab6b0) adding con mode: crc 2023-01-24T14:43:33.949 INFO:tasks.ceph.osd.0.smithi027.stderr:INFO 2023-01-24 14:43:33,947 [shard 0] auth - AuthRegistry(0x7f65264ab6b0) adding con mode: crc 2023-01-24T14:43:33.949 INFO:tasks.ceph.osd.0.smithi027.stderr:INFO 2023-01-24 14:43:33,947 [shard 0] auth - AuthRegistry(0x7f65264ab6b0) adding con mode: crc 2023-01-24T14:43:33.949 INFO:tasks.ceph.osd.0.smithi027.stderr:INFO 2023-01-24 14:43:33,947 [shard 0] auth - AuthRegistry(0x7f65264ab6b0) adding con mode: crc 2023-01-24T14:43:33.949 INFO:tasks.ceph.osd.0.smithi027.stderr:INFO 2023-01-24 14:43:33,947 [shard 0] auth - AuthRegistry(0x7f65264ab6b0) adding con mode: crc 2023-01-24T14:43:33.951 INFO:tasks.ceph.osd.1.smithi027.stderr:INFO 2023-01-24 14:43:33,947 [shard 0] auth - AuthRegistry(0x7f059c0f76b0) adding auth protocol: cephx 2023-01-24T14:43:33.951 INFO:tasks.ceph.osd.1.smithi027.stderr:INFO 2023-01-24 14:43:33,947 [shard 0] auth - AuthRegistry(0x7f059c0f76b0) adding auth protocol: cephx 2023-01-24T14:43:33.951 INFO:tasks.ceph.osd.1.smithi027.stderr:INFO 2023-01-24 14:43:33,947 [shard 0] auth - AuthRegistry(0x7f059c0f76b0) adding auth protocol: cephx 2023-01-24T14:43:33.951 INFO:tasks.ceph.osd.1.smithi027.stderr:INFO 2023-01-24 14:43:33,947 [shard 0] auth - AuthRegistry(0x7f059c0f76b0) adding con mode: crc 2023-01-24T14:43:33.952 INFO:tasks.ceph.osd.1.smithi027.stderr:INFO 2023-01-24 14:43:33,947 [shard 0] auth - AuthRegistry(0x7f059c0f76b0) adding con mode: crc 2023-01-24T14:43:33.952 INFO:tasks.ceph.osd.1.smithi027.stderr:INFO 2023-01-24 14:43:33,947 [shard 0] auth - AuthRegistry(0x7f059c0f76b0) adding con mode: crc 2023-01-24T14:43:33.952 INFO:tasks.ceph.osd.1.smithi027.stderr:INFO 2023-01-24 14:43:33,947 [shard 0] auth - AuthRegistry(0x7f059c0f76b0) adding con mode: crc 2023-01-24T14:43:33.952 INFO:tasks.ceph.osd.1.smithi027.stderr:INFO 2023-01-24 14:43:33,947 [shard 0] auth - AuthRegistry(0x7f059c0f76b0) adding con mode: crc 2023-01-24T14:43:33.952 INFO:tasks.ceph.osd.1.smithi027.stderr:INFO 2023-01-24 14:43:33,947 [shard 0] auth - AuthRegistry(0x7f059c0f76b0) adding con mode: crc 2023-01-24T14:43:33.953 INFO:tasks.ceph.osd.1.smithi027.stderr:INFO 2023-01-24 14:43:33,947 [shard 0] auth - auth: KeyRing::load: loaded key file /var/lib/ceph/osd/ceph-1/keyring 2023-01-24T14:43:33.953 INFO:tasks.ceph.osd.0.smithi027.stderr:INFO 2023-01-24 14:43:33,948 [shard 0] auth - auth: KeyRing::load: loaded key file /var/lib/ceph/osd/ceph-0/keyring 2023-01-24T14:43:33.953 INFO:tasks.ceph.osd.2.smithi035.stderr:==102234==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7ffe312d1000; bottom 0x7fcc0e112000; size: 0x0032231bf000 (215337398272) 2023-01-24T14:43:33.954 INFO:tasks.ceph.osd.2.smithi035.stderr:False positive error reports may follow 2023-01-24T14:43:33.954 INFO:tasks.ceph.osd.2.smithi035.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-01-24T14:43:33.962 INFO:tasks.ceph.osd.0.smithi027.stderr:DEBUG 2023-01-24 14:43:33,961 [shard 0] monc - build_monmap: for_mkfs=false 2023-01-24T14:43:33.963 INFO:tasks.ceph.osd.0.smithi027.stderr:INFO 2023-01-24 14:43:33,962 [shard 0] monc - reopen_session to mon.-1 2023-01-24T14:43:33.963 INFO:tasks.ceph.osd.0.smithi027.stderr:INFO 2023-01-24 14:43:33,962 [shard 0] monc - connecting to mon.1 2023-01-24T14:43:33.964 INFO:tasks.ceph.osd.0.smithi027.stderr:INFO 2023-01-24 14:43:33,962 [shard 0] ms - [0x614000008040 client.?(temp_mon_client) - >> mon.? v2:172.21.15.27:3300/0] ProtocolV2::start_connect(): peer_addr=v2:172.21.15.27:3300/0, peer_name=mon.?, cc=14939722321823623968 policy(lossy=false, server=false, standby=false, resetcheck=true) 2023-01-24T14:43:33.964 INFO:tasks.ceph.osd.0.smithi027.stderr:INFO 2023-01-24 14:43:33,962 [shard 0] ms - [0x614000008040 client.?(temp_mon_client) - >> mon.? v2:172.21.15.27:3300/0] write_event: delay ... 2023-01-24T14:43:33.964 INFO:tasks.ceph.osd.0.smithi027.stderr:WARN 2023-01-24 14:43:33,962 [shard 0] monc - mon.0 does not have an addr compatible with me 2023-01-24T14:43:33.965 INFO:tasks.ceph.osd.0.smithi027.stderr:INFO 2023-01-24 14:43:33,963 [shard 0] ms - [0x614000008040 client.?(temp_mon_client) -@58789 >> mon.? v2:172.21.15.27:3300/0] peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58789/0 (socket says 172.21.15.27:58789) 2023-01-24T14:43:33.965 INFO:tasks.ceph.osd.0.smithi027.stderr:INFO 2023-01-24 14:43:33,963 [shard 0] ms - [0x614000008040 client.?(temp_mon_client) 172.21.15.27:0/102700@58789 >> mon.? v2:172.21.15.27:3300/0] learned myaddr=172.21.15.27:0/102700 (unbound) 2023-01-24T14:43:33.966 INFO:tasks.ceph.osd.0.smithi027.stderr:INFO 2023-01-24 14:43:33,963 [shard 0] monc - get_auth_request(conn=[0x614000008040 client.?(temp_mon_client) 172.21.15.27:0/102700@58789 >> mon.? v2:172.21.15.27:3300/0], auth_method=0) 2023-01-24T14:43:33.966 INFO:tasks.ceph.osd.0.smithi027.stderr:INFO 2023-01-24 14:43:33,963 [shard 0] monc - method 2 preferred_modes [1] 2023-01-24T14:43:33.967 INFO:tasks.ceph.osd.1.smithi027.stderr:DEBUG 2023-01-24 14:43:33,964 [shard 0] monc - build_monmap: for_mkfs=false 2023-01-24T14:43:33.967 INFO:tasks.ceph.osd.1.smithi027.stderr:INFO 2023-01-24 14:43:33,964 [shard 0] monc - reopen_session to mon.-1 2023-01-24T14:43:33.967 INFO:tasks.ceph.osd.1.smithi027.stderr:INFO 2023-01-24 14:43:33,964 [shard 0] monc - connecting to mon.1 2023-01-24T14:43:33.968 INFO:tasks.ceph.osd.1.smithi027.stderr:INFO 2023-01-24 14:43:33,964 [shard 0] ms - [0x614000008040 client.?(temp_mon_client) - >> mon.? v2:172.21.15.27:3300/0] ProtocolV2::start_connect(): peer_addr=v2:172.21.15.27:3300/0, peer_name=mon.?, cc=5327286721629681638 policy(lossy=false, server=false, standby=false, resetcheck=true) 2023-01-24T14:43:33.968 INFO:tasks.ceph.osd.1.smithi027.stderr:INFO 2023-01-24 14:43:33,964 [shard 0] ms - [0x614000008040 client.?(temp_mon_client) - >> mon.? v2:172.21.15.27:3300/0] write_event: delay ... 2023-01-24T14:43:33.968 INFO:tasks.ceph.osd.1.smithi027.stderr:WARN 2023-01-24 14:43:33,964 [shard 0] monc - mon.0 does not have an addr compatible with me 2023-01-24T14:43:33.968 INFO:tasks.ceph.osd.1.smithi027.stderr:INFO 2023-01-24 14:43:33,965 [shard 0] ms - [0x614000008040 client.?(temp_mon_client) -@59624 >> mon.? v2:172.21.15.27:3300/0] peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59624/0 (socket says 172.21.15.27:59624) 2023-01-24T14:43:33.969 INFO:tasks.ceph.osd.1.smithi027.stderr:INFO 2023-01-24 14:43:33,965 [shard 0] ms - [0x614000008040 client.?(temp_mon_client) 172.21.15.27:0/102699@59624 >> mon.? v2:172.21.15.27:3300/0] learned myaddr=172.21.15.27:0/102699 (unbound) 2023-01-24T14:43:33.969 INFO:tasks.ceph.osd.1.smithi027.stderr:INFO 2023-01-24 14:43:33,965 [shard 0] monc - get_auth_request(conn=[0x614000008040 client.?(temp_mon_client) 172.21.15.27:0/102699@59624 >> mon.? v2:172.21.15.27:3300/0], auth_method=0) 2023-01-24T14:43:33.969 INFO:tasks.ceph.osd.1.smithi027.stderr:INFO 2023-01-24 14:43:33,965 [shard 0] monc - method 2 preferred_modes [1] 2023-01-24T14:43:33.977 INFO:tasks.ceph.osd.3.smithi035.stderr:Reactor stalled for 52 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x3a1b13e0 0x3a1b1b3f 0x3a15ebef 0x3a15f66e 0x39e380bf 0x39ee31ba 0x3c084996 0x3c08beea 0x3c08fd37 0x3c0923ba 0x39b308ec 0x39b3b879 0x39b3c67e 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x3640fffa 0x36412508 0x361d17ad 0x3ad84 0x35e7229d 2023-01-24T14:43:33.978 INFO:tasks.ceph.osd.3.smithi035.stderr:kernel callstack: 2023-01-24T14:43:33.982 INFO:tasks.ceph.mgr.x.smithi035.stderr:2023-01-24T14:43:33.980+0000 7f6a691e91c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-01-24T14:43:33.988 INFO:tasks.ceph.osd.3.smithi035.stderr:INFO 2023-01-24 14:43:33,987 [shard 0] osd - populating config from monitor 2023-01-24T14:43:33.990 INFO:tasks.ceph.osd.3.smithi035.stderr:INFO 2023-01-24 14:43:33,988 [shard 0] auth - AuthRegistry(0x7f74698f56b0) adding auth protocol: cephx 2023-01-24T14:43:33.990 INFO:tasks.ceph.osd.3.smithi035.stderr:INFO 2023-01-24 14:43:33,988 [shard 0] auth - AuthRegistry(0x7f74698f56b0) adding auth protocol: cephx 2023-01-24T14:43:33.990 INFO:tasks.ceph.osd.3.smithi035.stderr:INFO 2023-01-24 14:43:33,988 [shard 0] auth - AuthRegistry(0x7f74698f56b0) adding auth protocol: cephx 2023-01-24T14:43:33.990 INFO:tasks.ceph.osd.3.smithi035.stderr:INFO 2023-01-24 14:43:33,988 [shard 0] auth - AuthRegistry(0x7f74698f56b0) adding con mode: crc 2023-01-24T14:43:33.991 INFO:tasks.ceph.osd.3.smithi035.stderr:INFO 2023-01-24 14:43:33,988 [shard 0] auth - AuthRegistry(0x7f74698f56b0) adding con mode: crc 2023-01-24T14:43:33.991 INFO:tasks.ceph.osd.3.smithi035.stderr:INFO 2023-01-24 14:43:33,989 [shard 0] auth - AuthRegistry(0x7f74698f56b0) adding con mode: crc 2023-01-24T14:43:33.991 INFO:tasks.ceph.osd.3.smithi035.stderr:INFO 2023-01-24 14:43:33,989 [shard 0] auth - AuthRegistry(0x7f74698f56b0) adding con mode: crc 2023-01-24T14:43:33.991 INFO:tasks.ceph.osd.3.smithi035.stderr:INFO 2023-01-24 14:43:33,989 [shard 0] auth - AuthRegistry(0x7f74698f56b0) adding con mode: crc 2023-01-24T14:43:33.991 INFO:tasks.ceph.osd.3.smithi035.stderr:INFO 2023-01-24 14:43:33,989 [shard 0] auth - AuthRegistry(0x7f74698f56b0) adding con mode: crc 2023-01-24T14:43:33.992 INFO:tasks.ceph.osd.3.smithi035.stderr:INFO 2023-01-24 14:43:33,989 [shard 0] auth - auth: KeyRing::load: loaded key file /var/lib/ceph/osd/ceph-3/keyring 2023-01-24T14:43:33.994 INFO:tasks.ceph.osd.3.smithi035.stderr:DEBUG 2023-01-24 14:43:33,992 [shard 0] monc - build_monmap: for_mkfs=false 2023-01-24T14:43:33.994 INFO:tasks.ceph.osd.3.smithi035.stderr:INFO 2023-01-24 14:43:33,993 [shard 0] monc - reopen_session to mon.-1 2023-01-24T14:43:33.994 INFO:tasks.ceph.osd.3.smithi035.stderr:WARN 2023-01-24 14:43:33,993 [shard 0] monc - mon.0 does not have an addr compatible with me 2023-01-24T14:43:33.994 INFO:tasks.ceph.osd.3.smithi035.stderr:INFO 2023-01-24 14:43:33,993 [shard 0] monc - connecting to mon.1 2023-01-24T14:43:33.994 INFO:tasks.ceph.osd.3.smithi035.stderr:INFO 2023-01-24 14:43:33,993 [shard 0] ms - [0x614000008040 client.?(temp_mon_client) - >> mon.? v2:172.21.15.27:3300/0] ProtocolV2::start_connect(): peer_addr=v2:172.21.15.27:3300/0, peer_name=mon.?, cc=8889623517455072657 policy(lossy=false, server=false, standby=false, resetcheck=true) 2023-01-24T14:43:33.995 INFO:tasks.ceph.osd.3.smithi035.stderr:INFO 2023-01-24 14:43:33,993 [shard 0] ms - [0x614000008040 client.?(temp_mon_client) - >> mon.? v2:172.21.15.27:3300/0] write_event: delay ... 2023-01-24T14:43:33.996 INFO:tasks.ceph.osd.3.smithi035.stderr:INFO 2023-01-24 14:43:33,995 [shard 0] ms - [0x614000008040 client.?(temp_mon_client) -@49467 >> mon.? v2:172.21.15.27:3300/0] peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.35:49467/0 (socket says 172.21.15.35:49467) 2023-01-24T14:43:33.996 INFO:tasks.ceph.osd.3.smithi035.stderr:INFO 2023-01-24 14:43:33,996 [shard 0] ms - [0x614000008040 client.?(temp_mon_client) 172.21.15.35:0/102233@49467 >> mon.? v2:172.21.15.27:3300/0] learned myaddr=172.21.15.35:0/102233 (unbound) 2023-01-24T14:43:33.997 INFO:tasks.ceph.osd.3.smithi035.stderr:INFO 2023-01-24 14:43:33,996 [shard 0] monc - get_auth_request(conn=[0x614000008040 client.?(temp_mon_client) 172.21.15.35:0/102233@49467 >> mon.? v2:172.21.15.27:3300/0], auth_method=0) 2023-01-24T14:43:33.997 INFO:tasks.ceph.osd.3.smithi035.stderr:INFO 2023-01-24 14:43:33,996 [shard 0] monc - method 2 preferred_modes [1] 2023-01-24T14:43:34.004 INFO:tasks.ceph.osd.0.smithi027.stderr:INFO 2023-01-24 14:43:34,004 [shard 0] monc - responding with 36 bytes 2023-01-24T14:43:34.005 INFO:tasks.ceph.osd.0.smithi027.stderr:INFO 2023-01-24 14:43:34,005 [shard 0] monc - found mon.noname-a 2023-01-24T14:43:34.006 INFO:tasks.ceph.osd.0.smithi027.stderr:DEBUG 2023-01-24 14:43:34,005 [shard 0] monc - reopen_session mon connection attempts complete 2023-01-24T14:43:34.006 INFO:tasks.ceph.osd.0.smithi027.stderr:INFO 2023-01-24 14:43:34,005 [shard 0] monc - renew_rotating_keyring renewing rotating keys (they expired before 1674571384.0051515) 2023-01-24T14:43:34.006 INFO:tasks.ceph.osd.0.smithi027.stderr:INFO 2023-01-24 14:43:34,005 [shard 0] monc - sending auth(proto 2 2 bytes epoch 0) v1 2023-01-24T14:43:34.007 INFO:tasks.ceph.osd.0.smithi027.stderr:INFO 2023-01-24 14:43:34,005 [shard 0] monc - waiting 2023-01-24T14:43:34.008 INFO:tasks.ceph.osd.1.smithi027.stderr:INFO 2023-01-24 14:43:34,006 [shard 0] monc - responding with 36 bytes 2023-01-24T14:43:34.009 INFO:tasks.ceph.osd.1.smithi027.stderr:INFO 2023-01-24 14:43:34,006 [shard 0] monc - found mon.noname-a 2023-01-24T14:43:34.009 INFO:tasks.ceph.osd.1.smithi027.stderr:DEBUG 2023-01-24 14:43:34,006 [shard 0] monc - reopen_session mon connection attempts complete 2023-01-24T14:43:34.009 INFO:tasks.ceph.osd.1.smithi027.stderr:INFO 2023-01-24 14:43:34,006 [shard 0] monc - renew_rotating_keyring renewing rotating keys (they expired before 1674571384.0067918) 2023-01-24T14:43:34.009 INFO:tasks.ceph.osd.1.smithi027.stderr:INFO 2023-01-24 14:43:34,006 [shard 0] monc - sending auth(proto 2 2 bytes epoch 0) v1 2023-01-24T14:43:34.009 INFO:tasks.ceph.osd.1.smithi027.stderr:INFO 2023-01-24 14:43:34,006 [shard 0] monc - waiting 2023-01-24T14:43:34.010 INFO:tasks.ceph.osd.2.smithi035.stderr:INFO 2023-01-24 14:43:34,005 [shard 0] osd - populating config from monitor 2023-01-24T14:43:34.010 INFO:tasks.ceph.osd.2.smithi035.stderr:INFO 2023-01-24 14:43:34,005 [shard 0] auth - AuthRegistry(0x7fcc0e0d66b0) adding auth protocol: cephx 2023-01-24T14:43:34.010 INFO:tasks.ceph.osd.2.smithi035.stderr:INFO 2023-01-24 14:43:34,005 [shard 0] auth - AuthRegistry(0x7fcc0e0d66b0) adding auth protocol: cephx 2023-01-24T14:43:34.011 INFO:tasks.ceph.osd.2.smithi035.stderr:INFO 2023-01-24 14:43:34,005 [shard 0] auth - AuthRegistry(0x7fcc0e0d66b0) adding auth protocol: cephx 2023-01-24T14:43:34.011 INFO:tasks.ceph.osd.2.smithi035.stderr:INFO 2023-01-24 14:43:34,005 [shard 0] auth - AuthRegistry(0x7fcc0e0d66b0) adding con mode: crc 2023-01-24T14:43:34.011 INFO:tasks.ceph.osd.2.smithi035.stderr:INFO 2023-01-24 14:43:34,006 [shard 0] auth - AuthRegistry(0x7fcc0e0d66b0) adding con mode: crc 2023-01-24T14:43:34.011 INFO:tasks.ceph.osd.2.smithi035.stderr:INFO 2023-01-24 14:43:34,006 [shard 0] auth - AuthRegistry(0x7fcc0e0d66b0) adding con mode: crc 2023-01-24T14:43:34.012 INFO:tasks.ceph.osd.2.smithi035.stderr:INFO 2023-01-24 14:43:34,006 [shard 0] auth - AuthRegistry(0x7fcc0e0d66b0) adding con mode: crc 2023-01-24T14:43:34.012 INFO:tasks.ceph.osd.2.smithi035.stderr:INFO 2023-01-24 14:43:34,006 [shard 0] auth - AuthRegistry(0x7fcc0e0d66b0) adding con mode: crc 2023-01-24T14:43:34.012 INFO:tasks.ceph.osd.2.smithi035.stderr:INFO 2023-01-24 14:43:34,006 [shard 0] auth - AuthRegistry(0x7fcc0e0d66b0) adding con mode: crc 2023-01-24T14:43:34.012 INFO:tasks.ceph.osd.2.smithi035.stderr:INFO 2023-01-24 14:43:34,006 [shard 0] auth - auth: KeyRing::load: loaded key file /var/lib/ceph/osd/ceph-2/keyring 2023-01-24T14:43:34.019 INFO:tasks.ceph.osd.2.smithi035.stderr:DEBUG 2023-01-24 14:43:34,018 [shard 0] monc - build_monmap: for_mkfs=false 2023-01-24T14:43:34.019 INFO:tasks.ceph.osd.2.smithi035.stderr:INFO 2023-01-24 14:43:34,018 [shard 0] monc - reopen_session to mon.-1 2023-01-24T14:43:34.019 INFO:tasks.ceph.osd.2.smithi035.stderr:INFO 2023-01-24 14:43:34,018 [shard 0] monc - connecting to mon.1 2023-01-24T14:43:34.019 INFO:tasks.ceph.osd.2.smithi035.stderr:INFO 2023-01-24 14:43:34,018 [shard 0] ms - [0x614000008040 client.?(temp_mon_client) - >> mon.? v2:172.21.15.27:3300/0] ProtocolV2::start_connect(): peer_addr=v2:172.21.15.27:3300/0, peer_name=mon.?, cc=2168829120539482652 policy(lossy=false, server=false, standby=false, resetcheck=true) 2023-01-24T14:43:34.019 INFO:tasks.ceph.osd.2.smithi035.stderr:INFO 2023-01-24 14:43:34,018 [shard 0] ms - [0x614000008040 client.?(temp_mon_client) - >> mon.? v2:172.21.15.27:3300/0] write_event: delay ... 2023-01-24T14:43:34.020 INFO:tasks.ceph.osd.2.smithi035.stderr:WARN 2023-01-24 14:43:34,018 [shard 0] monc - mon.0 does not have an addr compatible with me 2023-01-24T14:43:34.020 INFO:tasks.ceph.osd.2.smithi035.stderr:INFO 2023-01-24 14:43:34,019 [shard 0] ms - [0x614000008040 client.?(temp_mon_client) -@64353 >> mon.? v2:172.21.15.27:3300/0] peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.35:64353/0 (socket says 172.21.15.35:64353) 2023-01-24T14:43:34.021 INFO:tasks.ceph.osd.2.smithi035.stderr:INFO 2023-01-24 14:43:34,019 [shard 0] ms - [0x614000008040 client.?(temp_mon_client) 172.21.15.35:0/102234@64353 >> mon.? v2:172.21.15.27:3300/0] learned myaddr=172.21.15.35:0/102234 (unbound) 2023-01-24T14:43:34.021 INFO:tasks.ceph.osd.2.smithi035.stderr:INFO 2023-01-24 14:43:34,019 [shard 0] monc - get_auth_request(conn=[0x614000008040 client.?(temp_mon_client) 172.21.15.35:0/102234@64353 >> mon.? v2:172.21.15.27:3300/0], auth_method=0) 2023-01-24T14:43:34.021 INFO:tasks.ceph.osd.2.smithi035.stderr:INFO 2023-01-24 14:43:34,019 [shard 0] monc - method 2 preferred_modes [1] 2023-01-24T14:43:34.036 INFO:tasks.ceph.osd.3.smithi035.stderr:INFO 2023-01-24 14:43:34,036 [shard 0] monc - responding with 36 bytes 2023-01-24T14:43:34.037 INFO:tasks.ceph.osd.3.smithi035.stderr:INFO 2023-01-24 14:43:34,037 [shard 0] monc - found mon.noname-a 2023-01-24T14:43:34.037 INFO:tasks.ceph.osd.3.smithi035.stderr:DEBUG 2023-01-24 14:43:34,037 [shard 0] monc - reopen_session mon connection attempts complete 2023-01-24T14:43:34.038 INFO:tasks.ceph.osd.3.smithi035.stderr:INFO 2023-01-24 14:43:34,037 [shard 0] monc - renew_rotating_keyring renewing rotating keys (they expired before 1674571384.037233) 2023-01-24T14:43:34.038 INFO:tasks.ceph.osd.3.smithi035.stderr:INFO 2023-01-24 14:43:34,037 [shard 0] monc - sending auth(proto 2 2 bytes epoch 0) v1 2023-01-24T14:43:34.038 INFO:tasks.ceph.osd.3.smithi035.stderr:INFO 2023-01-24 14:43:34,037 [shard 0] monc - waiting 2023-01-24T14:43:34.046 INFO:tasks.ceph.osd.0.smithi027.stderr:WARN 2023-01-24 14:43:34,046 [shard 0] ms - [0x614000008040 client.?(temp_mon_client) 172.21.15.27:0/102700@58789 >> mon.0 v2:172.21.15.27:3300/0] UPDATE Policy(lossy=true) from server flags 2023-01-24T14:43:34.046 INFO:tasks.ceph.osd.0.smithi027.stderr:INFO 2023-01-24 14:43:34,046 [shard 0] ms - [0x614000008040 client.?(temp_mon_client) 172.21.15.27:0/102700@58789 >> mon.0 v2:172.21.15.27:3300/0] connected: gs=1, pgs=25, cs=0, client_cookie=14939722321823623968, server_cookie=0, in_seq=0, out_seq=0, out_q=2 2023-01-24T14:43:34.049 INFO:tasks.ceph.osd.0.smithi027.stderr:INFO 2023-01-24 14:43:34,047 [shard 0] monc - got monmap 1, mon.a, is now rank 0 2023-01-24T14:43:34.049 INFO:tasks.ceph.osd.0.smithi027.stderr:INFO 2023-01-24 14:43:34,047 [shard 0] monc - handle_monmap: renewing tickets 2023-01-24T14:43:34.049 INFO:tasks.ceph.osd.0.smithi027.stderr:INFO 2023-01-24 14:43:34,047 [shard 0] monc - renew_rotating_keyring renewing rotating keys (they expired before 1674571384.047018) 2023-01-24T14:43:34.050 INFO:tasks.ceph.osd.0.smithi027.stderr:INFO 2023-01-24 14:43:34,047 [shard 0] monc - renew_rotating_keyring called too often (last: 1674571414.0051515) 2023-01-24T14:43:34.050 INFO:tasks.ceph.osd.0.smithi027.stderr:DEBUG 2023-01-24 14:43:34,047 [shard 0] monc - renew_tickets: don't need new tickets 2023-01-24T14:43:34.050 INFO:tasks.ceph.osd.0.smithi027.stderr:INFO 2023-01-24 14:43:34,047 [shard 0] monc - handle_mon_map: renewed tickets 2023-01-24T14:43:34.050 INFO:tasks.ceph.osd.0.smithi027.stderr:INFO 2023-01-24 14:43:34,047 [shard 0] monc - handle_auth_reply [0x614000008040 client.?(temp_mon_client) 172.21.15.27:0/102700@58789 >> mon.0 v2:172.21.15.27:3300/0] returns auth_reply(proto 2 0 (0) Success) v1: 0 2023-01-24T14:43:34.050 INFO:tasks.ceph.osd.0.smithi027.stderr:INFO 2023-01-24 14:43:34,047 [shard 0] monc - handle_auth_reply 2023-01-24T14:43:34.050 INFO:tasks.ceph.osd.0.smithi027.stderr:DEBUG 2023-01-24 14:43:34,047 [shard 0] monc - renew_tickets: don't need new tickets 2023-01-24T14:43:34.051 INFO:tasks.ceph.osd.0.smithi027.stderr:INFO 2023-01-24 14:43:34,047 [shard 0] monc - renew_rotating_keyring renewing rotating keys (they expired before 1674571384.0474987) 2023-01-24T14:43:34.051 INFO:tasks.ceph.osd.0.smithi027.stderr:INFO 2023-01-24 14:43:34,047 [shard 0] monc - renew_rotating_keyring called too often (last: 1674571414.0051515) 2023-01-24T14:43:34.051 INFO:tasks.ceph.osd.0.smithi027.stderr:INFO 2023-01-24 14:43:34,047 [shard 0] monc - do_auth_single: [0x614000008040 client.?(temp_mon_client) 172.21.15.27:0/102700@58789 >> mon.0 v2:172.21.15.27:3300/0] returns auth_reply(proto 2 0 (0) Success) v1: 0 2023-01-24T14:43:34.052 INFO:tasks.ceph.osd.0.smithi027.stderr:INFO 2023-01-24 14:43:34,048 [shard 0] monc - set_mon_vals no callback set 2023-01-24T14:43:34.053 INFO:tasks.ceph.osd.0.smithi027.stderr:DEBUG 2023-01-24 14:43:34,051 [shard 0] osd - operator(): got config from monitor, fsid 5e79a879-b90d-445c-b439-aec3d80a8b8a 2023-01-24T14:43:34.053 INFO:tasks.ceph.osd.0.smithi027.stderr:INFO 2023-01-24 14:43:34,051 [shard 0] monc - stop 2023-01-24T14:43:34.055 INFO:tasks.ceph.osd.0.smithi027.stderr:INFO 2023-01-24 14:43:34,051 [shard 0] monc - close 2023-01-24T14:43:34.055 INFO:tasks.ceph.osd.0.smithi027.stderr:INFO 2023-01-24 14:43:34,051 [shard 0] ms - [0x614000008040 client.?(temp_mon_client) 172.21.15.27:0/102700@58789 >> mon.0 v2:172.21.15.27:3300/0] closing: reset no, replace no 2023-01-24T14:43:34.055 INFO:tasks.ceph.osd.0.smithi027.stderr:INFO 2023-01-24 14:43:34,051 [shard 0] ms - [0x614000008040 client.?(temp_mon_client) 172.21.15.27:0/102700@58789 >> mon.0 v2:172.21.15.27:3300/0] execute_ready(): protocol aborted at CLOSING -- std::system_error (error crimson::net:4, read eof) 2023-01-24T14:43:34.055 INFO:tasks.ceph.osd.0.smithi027.stderr:INFO 2023-01-24 14:43:34,051 [shard 0] osd - get_early_config: set --smp 1 2023-01-24T14:43:34.058 INFO:tasks.ceph.osd.1.smithi027.stderr:WARN 2023-01-24 14:43:34,053 [shard 0] ms - [0x614000008040 client.?(temp_mon_client) 172.21.15.27:0/102699@59624 >> mon.0 v2:172.21.15.27:3300/0] UPDATE Policy(lossy=true) from server flags 2023-01-24T14:43:34.058 INFO:tasks.ceph.osd.1.smithi027.stderr:INFO 2023-01-24 14:43:34,054 [shard 0] ms - [0x614000008040 client.?(temp_mon_client) 172.21.15.27:0/102699@59624 >> mon.0 v2:172.21.15.27:3300/0] connected: gs=1, pgs=26, cs=0, client_cookie=5327286721629681638, server_cookie=0, in_seq=0, out_seq=0, out_q=2 2023-01-24T14:43:34.058 INFO:tasks.ceph.osd.1.smithi027.stderr:INFO 2023-01-24 14:43:34,054 [shard 0] monc - got monmap 1, mon.a, is now rank 0 2023-01-24T14:43:34.059 INFO:tasks.ceph.osd.1.smithi027.stderr:INFO 2023-01-24 14:43:34,054 [shard 0] monc - handle_monmap: renewing tickets 2023-01-24T14:43:34.059 INFO:tasks.ceph.osd.1.smithi027.stderr:INFO 2023-01-24 14:43:34,054 [shard 0] monc - renew_rotating_keyring renewing rotating keys (they expired before 1674571384.054508) 2023-01-24T14:43:34.059 INFO:tasks.ceph.osd.1.smithi027.stderr:INFO 2023-01-24 14:43:34,054 [shard 0] monc - renew_rotating_keyring called too often (last: 1674571414.0067918) 2023-01-24T14:43:34.059 INFO:tasks.ceph.osd.1.smithi027.stderr:DEBUG 2023-01-24 14:43:34,054 [shard 0] monc - renew_tickets: don't need new tickets 2023-01-24T14:43:34.059 INFO:tasks.ceph.osd.1.smithi027.stderr:INFO 2023-01-24 14:43:34,054 [shard 0] monc - handle_mon_map: renewed tickets 2023-01-24T14:43:34.060 INFO:tasks.ceph.osd.1.smithi027.stderr:INFO 2023-01-24 14:43:34,054 [shard 0] monc - handle_auth_reply [0x614000008040 client.?(temp_mon_client) 172.21.15.27:0/102699@59624 >> mon.0 v2:172.21.15.27:3300/0] returns auth_reply(proto 2 0 (0) Success) v1: 0 2023-01-24T14:43:34.060 INFO:tasks.ceph.osd.1.smithi027.stderr:INFO 2023-01-24 14:43:34,054 [shard 0] monc - handle_auth_reply 2023-01-24T14:43:34.060 INFO:tasks.ceph.osd.1.smithi027.stderr:DEBUG 2023-01-24 14:43:34,054 [shard 0] monc - renew_tickets: don't need new tickets 2023-01-24T14:43:34.060 INFO:tasks.ceph.osd.1.smithi027.stderr:INFO 2023-01-24 14:43:34,054 [shard 0] monc - renew_rotating_keyring renewing rotating keys (they expired before 1674571384.0549645) 2023-01-24T14:43:34.060 INFO:tasks.ceph.osd.1.smithi027.stderr:INFO 2023-01-24 14:43:34,054 [shard 0] monc - renew_rotating_keyring called too often (last: 1674571414.0067918) 2023-01-24T14:43:34.061 INFO:tasks.ceph.osd.1.smithi027.stderr:INFO 2023-01-24 14:43:34,055 [shard 0] monc - do_auth_single: [0x614000008040 client.?(temp_mon_client) 172.21.15.27:0/102699@59624 >> mon.0 v2:172.21.15.27:3300/0] returns auth_reply(proto 2 0 (0) Success) v1: 0 2023-01-24T14:43:34.061 INFO:tasks.ceph.osd.1.smithi027.stderr:INFO 2023-01-24 14:43:34,055 [shard 0] monc - set_mon_vals no callback set 2023-01-24T14:43:34.062 INFO:tasks.ceph.osd.1.smithi027.stderr:DEBUG 2023-01-24 14:43:34,058 [shard 0] osd - operator(): got config from monitor, fsid 5e79a879-b90d-445c-b439-aec3d80a8b8a 2023-01-24T14:43:34.062 INFO:tasks.ceph.osd.1.smithi027.stderr:INFO 2023-01-24 14:43:34,058 [shard 0] monc - stop 2023-01-24T14:43:34.062 INFO:tasks.ceph.osd.1.smithi027.stderr:INFO 2023-01-24 14:43:34,058 [shard 0] monc - close 2023-01-24T14:43:34.063 INFO:tasks.ceph.osd.1.smithi027.stderr:INFO 2023-01-24 14:43:34,058 [shard 0] ms - [0x614000008040 client.?(temp_mon_client) 172.21.15.27:0/102699@59624 >> mon.0 v2:172.21.15.27:3300/0] closing: reset no, replace no 2023-01-24T14:43:34.063 INFO:tasks.ceph.osd.1.smithi027.stderr:INFO 2023-01-24 14:43:34,058 [shard 0] ms - [0x614000008040 client.?(temp_mon_client) 172.21.15.27:0/102699@59624 >> mon.0 v2:172.21.15.27:3300/0] execute_ready(): protocol aborted at CLOSING -- std::system_error (error crimson::net:4, read eof) 2023-01-24T14:43:34.063 INFO:tasks.ceph.osd.1.smithi027.stderr:INFO 2023-01-24 14:43:34,059 [shard 0] osd - get_early_config: set --smp 1 2023-01-24T14:43:34.063 INFO:tasks.ceph.osd.2.smithi035.stderr:INFO 2023-01-24 14:43:34,060 [shard 0] monc - responding with 36 bytes 2023-01-24T14:43:34.063 INFO:tasks.ceph.osd.2.smithi035.stderr:INFO 2023-01-24 14:43:34,060 [shard 0] monc - found mon.noname-a 2023-01-24T14:43:34.064 INFO:tasks.ceph.osd.2.smithi035.stderr:DEBUG 2023-01-24 14:43:34,060 [shard 0] monc - reopen_session mon connection attempts complete 2023-01-24T14:43:34.064 INFO:tasks.ceph.osd.2.smithi035.stderr:INFO 2023-01-24 14:43:34,061 [shard 0] monc - renew_rotating_keyring renewing rotating keys (they expired before 1674571384.0610762) 2023-01-24T14:43:34.064 INFO:tasks.ceph.osd.2.smithi035.stderr:INFO 2023-01-24 14:43:34,061 [shard 0] monc - sending auth(proto 2 2 bytes epoch 0) v1 2023-01-24T14:43:34.064 INFO:tasks.ceph.osd.2.smithi035.stderr:INFO 2023-01-24 14:43:34,061 [shard 0] monc - waiting 2023-01-24T14:43:34.078 INFO:tasks.ceph.osd.3.smithi035.stderr:WARN 2023-01-24 14:43:34,078 [shard 0] ms - [0x614000008040 client.?(temp_mon_client) 172.21.15.35:0/102233@49467 >> mon.0 v2:172.21.15.27:3300/0] UPDATE Policy(lossy=true) from server flags 2023-01-24T14:43:34.079 INFO:tasks.ceph.osd.3.smithi035.stderr:INFO 2023-01-24 14:43:34,078 [shard 0] ms - [0x614000008040 client.?(temp_mon_client) 172.21.15.35:0/102233@49467 >> mon.0 v2:172.21.15.27:3300/0] connected: gs=1, pgs=27, cs=0, client_cookie=8889623517455072657, server_cookie=0, in_seq=0, out_seq=0, out_q=2 2023-01-24T14:43:34.080 INFO:tasks.ceph.osd.3.smithi035.stderr:INFO 2023-01-24 14:43:34,078 [shard 0] monc - got monmap 1, mon.a, is now rank 0 2023-01-24T14:43:34.081 INFO:tasks.ceph.osd.3.smithi035.stderr:INFO 2023-01-24 14:43:34,078 [shard 0] monc - handle_monmap: renewing tickets 2023-01-24T14:43:34.081 INFO:tasks.ceph.osd.3.smithi035.stderr:INFO 2023-01-24 14:43:34,078 [shard 0] monc - renew_rotating_keyring renewing rotating keys (they expired before 1674571384.078834) 2023-01-24T14:43:34.081 INFO:tasks.ceph.osd.3.smithi035.stderr:INFO 2023-01-24 14:43:34,078 [shard 0] monc - renew_rotating_keyring called too often (last: 1674571414.037233) 2023-01-24T14:43:34.081 INFO:tasks.ceph.osd.3.smithi035.stderr:DEBUG 2023-01-24 14:43:34,079 [shard 0] monc - renew_tickets: don't need new tickets 2023-01-24T14:43:34.081 INFO:tasks.ceph.osd.3.smithi035.stderr:INFO 2023-01-24 14:43:34,079 [shard 0] monc - handle_mon_map: renewed tickets 2023-01-24T14:43:34.082 INFO:tasks.ceph.osd.3.smithi035.stderr:INFO 2023-01-24 14:43:34,079 [shard 0] monc - handle_auth_reply [0x614000008040 client.?(temp_mon_client) 172.21.15.35:0/102233@49467 >> mon.0 v2:172.21.15.27:3300/0] returns auth_reply(proto 2 0 (0) Success) v1: 0 2023-01-24T14:43:34.082 INFO:tasks.ceph.osd.3.smithi035.stderr:INFO 2023-01-24 14:43:34,079 [shard 0] monc - handle_auth_reply 2023-01-24T14:43:34.082 INFO:tasks.ceph.osd.3.smithi035.stderr:DEBUG 2023-01-24 14:43:34,079 [shard 0] monc - renew_tickets: don't need new tickets 2023-01-24T14:43:34.082 INFO:tasks.ceph.osd.3.smithi035.stderr:INFO 2023-01-24 14:43:34,079 [shard 0] monc - renew_rotating_keyring renewing rotating keys (they expired before 1674571384.0793393) 2023-01-24T14:43:34.083 INFO:tasks.ceph.osd.3.smithi035.stderr:INFO 2023-01-24 14:43:34,079 [shard 0] monc - renew_rotating_keyring called too often (last: 1674571414.037233) 2023-01-24T14:43:34.083 INFO:tasks.ceph.osd.3.smithi035.stderr:INFO 2023-01-24 14:43:34,079 [shard 0] monc - do_auth_single: [0x614000008040 client.?(temp_mon_client) 172.21.15.35:0/102233@49467 >> mon.0 v2:172.21.15.27:3300/0] returns auth_reply(proto 2 0 (0) Success) v1: 0 2023-01-24T14:43:34.083 INFO:tasks.ceph.osd.3.smithi035.stderr:INFO 2023-01-24 14:43:34,080 [shard 0] monc - set_mon_vals no callback set 2023-01-24T14:43:34.084 INFO:tasks.ceph.osd.3.smithi035.stderr:DEBUG 2023-01-24 14:43:34,083 [shard 0] osd - operator(): got config from monitor, fsid 5e79a879-b90d-445c-b439-aec3d80a8b8a 2023-01-24T14:43:34.084 INFO:tasks.ceph.osd.3.smithi035.stderr:INFO 2023-01-24 14:43:34,083 [shard 0] monc - stop 2023-01-24T14:43:34.084 INFO:tasks.ceph.osd.3.smithi035.stderr:INFO 2023-01-24 14:43:34,083 [shard 0] monc - close 2023-01-24T14:43:34.085 INFO:tasks.ceph.osd.3.smithi035.stderr:INFO 2023-01-24 14:43:34,083 [shard 0] ms - [0x614000008040 client.?(temp_mon_client) 172.21.15.35:0/102233@49467 >> mon.0 v2:172.21.15.27:3300/0] closing: reset no, replace no 2023-01-24T14:43:34.085 INFO:tasks.ceph.osd.3.smithi035.stderr:INFO 2023-01-24 14:43:34,083 [shard 0] ms - [0x614000008040 client.?(temp_mon_client) 172.21.15.35:0/102233@49467 >> mon.0 v2:172.21.15.27:3300/0] execute_ready(): protocol aborted at CLOSING -- std::system_error (error crimson::net:4, read eof) 2023-01-24T14:43:34.085 INFO:tasks.ceph.osd.3.smithi035.stderr:INFO 2023-01-24 14:43:34,083 [shard 0] osd - get_early_config: set --smp 1 2023-01-24T14:43:34.102 INFO:tasks.ceph.osd.2.smithi035.stderr:WARN 2023-01-24 14:43:34,102 [shard 0] ms - [0x614000008040 client.?(temp_mon_client) 172.21.15.35:0/102234@64353 >> mon.0 v2:172.21.15.27:3300/0] UPDATE Policy(lossy=true) from server flags 2023-01-24T14:43:34.103 INFO:tasks.ceph.osd.2.smithi035.stderr:INFO 2023-01-24 14:43:34,102 [shard 0] ms - [0x614000008040 client.?(temp_mon_client) 172.21.15.35:0/102234@64353 >> mon.0 v2:172.21.15.27:3300/0] connected: gs=1, pgs=28, cs=0, client_cookie=2168829120539482652, server_cookie=0, in_seq=0, out_seq=0, out_q=2 2023-01-24T14:43:34.104 INFO:tasks.ceph.osd.2.smithi035.stderr:INFO 2023-01-24 14:43:34,103 [shard 0] monc - got monmap 1, mon.a, is now rank 0 2023-01-24T14:43:34.104 INFO:tasks.ceph.osd.2.smithi035.stderr:INFO 2023-01-24 14:43:34,103 [shard 0] monc - handle_monmap: renewing tickets 2023-01-24T14:43:34.105 INFO:tasks.ceph.osd.2.smithi035.stderr:INFO 2023-01-24 14:43:34,103 [shard 0] monc - renew_rotating_keyring renewing rotating keys (they expired before 1674571384.1029341) 2023-01-24T14:43:34.105 INFO:tasks.ceph.osd.2.smithi035.stderr:INFO 2023-01-24 14:43:34,103 [shard 0] monc - renew_rotating_keyring called too often (last: 1674571414.0610762) 2023-01-24T14:43:34.105 INFO:tasks.ceph.osd.2.smithi035.stderr:DEBUG 2023-01-24 14:43:34,103 [shard 0] monc - renew_tickets: don't need new tickets 2023-01-24T14:43:34.105 INFO:tasks.ceph.osd.2.smithi035.stderr:INFO 2023-01-24 14:43:34,103 [shard 0] monc - handle_mon_map: renewed tickets 2023-01-24T14:43:34.105 INFO:tasks.ceph.osd.2.smithi035.stderr:INFO 2023-01-24 14:43:34,103 [shard 0] monc - handle_auth_reply [0x614000008040 client.?(temp_mon_client) 172.21.15.35:0/102234@64353 >> mon.0 v2:172.21.15.27:3300/0] returns auth_reply(proto 2 0 (0) Success) v1: 0 2023-01-24T14:43:34.106 INFO:tasks.ceph.osd.2.smithi035.stderr:INFO 2023-01-24 14:43:34,103 [shard 0] monc - handle_auth_reply 2023-01-24T14:43:34.106 INFO:tasks.ceph.osd.2.smithi035.stderr:DEBUG 2023-01-24 14:43:34,103 [shard 0] monc - renew_tickets: don't need new tickets 2023-01-24T14:43:34.106 INFO:tasks.ceph.osd.2.smithi035.stderr:INFO 2023-01-24 14:43:34,103 [shard 0] monc - renew_rotating_keyring renewing rotating keys (they expired before 1674571384.1034467) 2023-01-24T14:43:34.106 INFO:tasks.ceph.osd.2.smithi035.stderr:INFO 2023-01-24 14:43:34,103 [shard 0] monc - renew_rotating_keyring called too often (last: 1674571414.0610762) 2023-01-24T14:43:34.106 INFO:tasks.ceph.osd.2.smithi035.stderr:INFO 2023-01-24 14:43:34,103 [shard 0] monc - do_auth_single: [0x614000008040 client.?(temp_mon_client) 172.21.15.35:0/102234@64353 >> mon.0 v2:172.21.15.27:3300/0] returns auth_reply(proto 2 0 (0) Success) v1: 0 2023-01-24T14:43:34.111 INFO:tasks.ceph.osd.2.smithi035.stderr:INFO 2023-01-24 14:43:34,110 [shard 0] monc - set_mon_vals no callback set 2023-01-24T14:43:34.119 INFO:tasks.ceph.osd.2.smithi035.stderr:DEBUG 2023-01-24 14:43:34,117 [shard 0] osd - operator(): got config from monitor, fsid 5e79a879-b90d-445c-b439-aec3d80a8b8a 2023-01-24T14:43:34.119 INFO:tasks.ceph.osd.2.smithi035.stderr:INFO 2023-01-24 14:43:34,117 [shard 0] monc - stop 2023-01-24T14:43:34.119 INFO:tasks.ceph.osd.2.smithi035.stderr:INFO 2023-01-24 14:43:34,117 [shard 0] monc - close 2023-01-24T14:43:34.119 INFO:tasks.ceph.osd.2.smithi035.stderr:INFO 2023-01-24 14:43:34,117 [shard 0] ms - [0x614000008040 client.?(temp_mon_client) 172.21.15.35:0/102234@64353 >> mon.0 v2:172.21.15.27:3300/0] closing: reset no, replace no 2023-01-24T14:43:34.120 INFO:tasks.ceph.osd.2.smithi035.stderr:INFO 2023-01-24 14:43:34,118 [shard 0] ms - [0x614000008040 client.?(temp_mon_client) 172.21.15.35:0/102234@64353 >> mon.0 v2:172.21.15.27:3300/0] execute_ready(): protocol aborted at CLOSING -- std::system_error (error crimson::net:4, read eof) 2023-01-24T14:43:34.120 INFO:tasks.ceph.osd.2.smithi035.stderr:INFO 2023-01-24 14:43:34,118 [shard 0] osd - get_early_config: set --smp 1 2023-01-24T14:43:34.238 INFO:tasks.ceph.mgr.x.smithi035.stderr:2023-01-24T14:43:34.236+0000 7f6a691e91c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-01-24T14:43:34.321 INFO:tasks.ceph.mgr.x.smithi035.stderr:2023-01-24T14:43:34.319+0000 7f6a691e91c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-01-24T14:43:34.476 INFO:tasks.ceph.mgr.x.smithi035.stderr:2023-01-24T14:43:34.474+0000 7f6a691e91c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-01-24T14:43:34.772 INFO:tasks.ceph.mgr.x.smithi035.stderr:2023-01-24T14:43:34.770+0000 7f6a691e91c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-01-24T14:43:34.988 INFO:tasks.ceph.mgr.x.smithi035.stderr:2023-01-24T14:43:34.986+0000 7f6a691e91c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-01-24T14:43:35.029 INFO:tasks.ceph.osd.1.smithi027.stderr:WARNING: debug mode. Not for benchmarking or production 2023-01-24T14:43:35.029 INFO:tasks.ceph.osd.1.smithi027.stderr:INFO 2023-01-24 14:43:35,029 seastar - Reactor backend: epoll 2023-01-24T14:43:35.036 INFO:tasks.ceph.osd.0.smithi027.stderr:WARNING: debug mode. Not for benchmarking or production 2023-01-24T14:43:35.037 INFO:tasks.ceph.osd.0.smithi027.stderr:INFO 2023-01-24 14:43:35,037 seastar - Reactor backend: epoll 2023-01-24T14:43:35.042 INFO:tasks.ceph.osd.3.smithi035.stderr:WARNING: debug mode. Not for benchmarking or production 2023-01-24T14:43:35.043 INFO:tasks.ceph.osd.3.smithi035.stderr:INFO 2023-01-24 14:43:35,043 seastar - Reactor backend: epoll 2023-01-24T14:43:35.045 INFO:tasks.ceph.osd.3.smithi035.stderr:INFO 2023-01-24 14:43:35,044 [shard 0] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-01-24T14:43:35.045 INFO:tasks.ceph.osd.3.smithi035.stderr:INFO 2023-01-24 14:43:35,044 [shard 0] seastar - IO queue uses 0.75ms latency goal for device 0 2023-01-24T14:43:35.045 INFO:tasks.ceph.osd.3.smithi035.stderr:INFO 2023-01-24 14:43:35,044 [shard 0] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-01-24T14:43:35.045 INFO:tasks.ceph.osd.3.smithi035.stderr:INFO 2023-01-24 14:43:35,044 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-01-24T14:43:35.046 INFO:tasks.ceph.osd.3.smithi035.stderr:==102222==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-01-24T14:43:35.053 INFO:tasks.ceph.osd.3.smithi035.stderr:==102222==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7fffaea9c000; bottom 0x7f746992a000; size: 0x008b45172000 (598159597568) 2023-01-24T14:43:35.053 INFO:tasks.ceph.osd.3.smithi035.stderr:False positive error reports may follow 2023-01-24T14:43:35.053 INFO:tasks.ceph.osd.3.smithi035.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-01-24T14:43:35.058 INFO:tasks.ceph.osd.1.smithi027.stderr:INFO 2023-01-24 14:43:35,057 [shard 0] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-01-24T14:43:35.058 INFO:tasks.ceph.osd.1.smithi027.stderr:INFO 2023-01-24 14:43:35,057 [shard 0] seastar - IO queue uses 0.75ms latency goal for device 0 2023-01-24T14:43:35.059 INFO:tasks.ceph.osd.1.smithi027.stderr:INFO 2023-01-24 14:43:35,057 [shard 0] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-01-24T14:43:35.059 INFO:tasks.ceph.osd.1.smithi027.stderr:INFO 2023-01-24 14:43:35,057 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-01-24T14:43:35.059 INFO:tasks.ceph.osd.1.smithi027.stderr:==102631==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-01-24T14:43:35.065 INFO:tasks.ceph.osd.1.smithi027.stderr:==102631==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7ffe4cae4000; bottom 0x7f059c12c000; size: 0x00f8b09b8000 (1068114870272) 2023-01-24T14:43:35.066 INFO:tasks.ceph.osd.1.smithi027.stderr:False positive error reports may follow 2023-01-24T14:43:35.066 INFO:tasks.ceph.osd.1.smithi027.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-01-24T14:43:35.075 INFO:tasks.ceph.osd.2.smithi035.stderr:WARNING: debug mode. Not for benchmarking or production 2023-01-24T14:43:35.076 INFO:tasks.ceph.osd.2.smithi035.stderr:INFO 2023-01-24 14:43:35,076 seastar - Reactor backend: epoll 2023-01-24T14:43:35.077 INFO:tasks.ceph.osd.0.smithi027.stderr:INFO 2023-01-24 14:43:35,076 [shard 0] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-01-24T14:43:35.077 INFO:tasks.ceph.osd.0.smithi027.stderr:INFO 2023-01-24 14:43:35,076 [shard 0] seastar - IO queue uses 0.75ms latency goal for device 0 2023-01-24T14:43:35.077 INFO:tasks.ceph.osd.0.smithi027.stderr:INFO 2023-01-24 14:43:35,076 [shard 0] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-01-24T14:43:35.078 INFO:tasks.ceph.osd.0.smithi027.stderr:INFO 2023-01-24 14:43:35,076 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-01-24T14:43:35.078 INFO:tasks.ceph.osd.0.smithi027.stderr:==102630==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-01-24T14:43:35.084 INFO:tasks.ceph.osd.0.smithi027.stderr:==102630==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7fff497f3000; bottom 0x7f65264e0000; size: 0x009a23313000 (662015389696) 2023-01-24T14:43:35.085 INFO:tasks.ceph.osd.0.smithi027.stderr:False positive error reports may follow 2023-01-24T14:43:35.085 INFO:tasks.ceph.osd.0.smithi027.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-01-24T14:43:35.094 INFO:tasks.ceph.osd.2.smithi035.stderr:INFO 2023-01-24 14:43:35,094 [shard 0] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-01-24T14:43:35.094 INFO:tasks.ceph.osd.2.smithi035.stderr:INFO 2023-01-24 14:43:35,094 [shard 0] seastar - IO queue uses 0.75ms latency goal for device 0 2023-01-24T14:43:35.095 INFO:tasks.ceph.osd.2.smithi035.stderr:INFO 2023-01-24 14:43:35,094 [shard 0] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-01-24T14:43:35.095 INFO:tasks.ceph.osd.2.smithi035.stderr:INFO 2023-01-24 14:43:35,094 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-01-24T14:43:35.096 INFO:tasks.ceph.osd.2.smithi035.stderr:==102221==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-01-24T14:43:35.102 INFO:tasks.ceph.osd.2.smithi035.stderr:==102221==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7ffe312d1000; bottom 0x7fcc0e10b000; size: 0x0032231c6000 (215337426944) 2023-01-24T14:43:35.103 INFO:tasks.ceph.osd.2.smithi035.stderr:False positive error reports may follow 2023-01-24T14:43:35.103 INFO:tasks.ceph.osd.2.smithi035.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-01-24T14:43:35.140 INFO:tasks.ceph.mgr.x.smithi035.stderr:2023-01-24T14:43:35.137+0000 7f6a691e91c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-01-24T14:43:35.184 INFO:tasks.ceph.osd.2.smithi035.stderr:Reactor stalled for 38 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x36027155 0x3602b6aa 0x3602ef36 0x3602f68b 0x4518f242 0x4518ac16 0x3cfb299a 0x3cfba991 0x3cb7407e 0x36420738 0x364238e3 0x445f3ec8 2023-01-24T14:43:35.185 INFO:tasks.ceph.osd.2.smithi035.stderr:kernel callstack: 2023-01-24T14:43:35.264 INFO:tasks.ceph.mgr.x.smithi035.stderr:2023-01-24T14:43:35.261+0000 7f6a691e91c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-01-24T14:43:35.421 INFO:tasks.ceph.mgr.x.smithi035.stderr:2023-01-24T14:43:35.419+0000 7f6a691e91c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-01-24T14:43:35.505 INFO:tasks.ceph.mgr.x.smithi035.stderr:2023-01-24T14:43:35.503+0000 7f6a691e91c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-01-24T14:43:35.740 INFO:tasks.ceph.mgr.x.smithi035.stderr:2023-01-24T14:43:35.738+0000 7f6a691e91c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-01-24T14:43:35.823 INFO:tasks.ceph.mgr.x.smithi035.stderr:2023-01-24T14:43:35.820+0000 7f6a691e91c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-01-24T14:43:36.470 INFO:tasks.ceph.mgr.x.smithi035.stderr:2023-01-24T14:43:36.468+0000 7f6a691e91c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-01-24T14:43:36.734 INFO:tasks.ceph.mgr.x.smithi035.stderr:2023-01-24T14:43:36.731+0000 7f6a691e91c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-01-24T14:43:36.814 INFO:tasks.ceph.mgr.x.smithi035.stderr:2023-01-24T14:43:36.812+0000 7f6a691e91c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-01-24T14:43:36.900 INFO:tasks.ceph.mgr.x.smithi035.stderr:2023-01-24T14:43:36.898+0000 7f6a691e91c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-01-24T14:43:37.092 INFO:tasks.ceph.mgr.x.smithi035.stderr:2023-01-24T14:43:37.090+0000 7f6a691e91c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-01-24T14:43:37.168 INFO:tasks.ceph.mgr.x.smithi035.stderr:2023-01-24T14:43:37.166+0000 7f6a691e91c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-01-24T14:43:37.326 INFO:tasks.ceph.mgr.x.smithi035.stderr:2023-01-24T14:43:37.324+0000 7f6a691e91c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-01-24T14:43:38.348 INFO:tasks.ceph.mgr.x.smithi035.stderr:2023-01-24T14:43:38.346+0000 7f6a25868700 -1 mgr.server handle_report got status from non-daemon mon.a 2023-01-24T14:43:40.246 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-01-24T14:43:40.384 INFO:teuthology.misc.health.smithi027.stderr:2023-01-24T14:43:40.382+0000 7fdcf175f700 1 Processor -- start 2023-01-24T14:43:40.384 INFO:teuthology.misc.health.smithi027.stderr:2023-01-24T14:43:40.382+0000 7fdcf175f700 1 -- start start 2023-01-24T14:43:40.386 INFO:teuthology.misc.health.smithi027.stderr:2023-01-24T14:43:40.383+0000 7fdcf175f700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fdcec0a3410 0x7fdcec1373b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T14:43:40.386 INFO:teuthology.misc.health.smithi027.stderr:2023-01-24T14:43:40.383+0000 7fdcf175f700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcec098290 con 0x7fdcec0a28f0 2023-01-24T14:43:40.386 INFO:teuthology.misc.health.smithi027.stderr:2023-01-24T14:43:40.383+0000 7fdcf175f700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdcec0a3dd0 con 0x7fdcec0a3410 2023-01-24T14:43:40.386 INFO:teuthology.misc.health.smithi027.stderr:2023-01-24T14:43:40.383+0000 7fdcea7fc700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fdcec0a3410 0x7fdcec1373b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T14:43:40.387 INFO:teuthology.misc.health.smithi027.stderr:2023-01-24T14:43:40.383+0000 7fdcea7fc700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fdcec0a3410 0x7fdcec1373b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48784/0 (socket says 172.21.15.27:48784) 2023-01-24T14:43:40.387 INFO:teuthology.misc.health.smithi027.stderr:2023-01-24T14:43:40.383+0000 7fdcea7fc700 1 -- 172.21.15.27:0/1223060905 learned_addr learned my addr 172.21.15.27:0/1223060905 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-01-24T14:43:40.387 INFO:teuthology.misc.health.smithi027.stderr:2023-01-24T14:43:40.383+0000 7fdceaffd700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fdcec0a28f0 0x7fdcec0a2d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:44908/0 (socket says 172.21.15.27:44908) 2023-01-24T14:43:40.387 INFO:teuthology.misc.health.smithi027.stderr:2023-01-24T14:43:40.383+0000 7fdcea7fc700 1 -- 172.21.15.27:0/1223060905 >> v1:172.21.15.27:6789/0 conn(0x7fdcec0a28f0 legacy=0x7fdcec0a2d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T14:43:40.387 INFO:teuthology.misc.health.smithi027.stderr:2023-01-24T14:43:40.383+0000 7fdcea7fc700 1 -- 172.21.15.27:0/1223060905 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcec098be0 con 0x7fdcec0a3410 2023-01-24T14:43:40.388 INFO:teuthology.misc.health.smithi027.stderr:2023-01-24T14:43:40.384+0000 7fdcea7fc700 1 --2- 172.21.15.27:0/1223060905 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdcec0a3410 0x7fdcec1373b0 crc :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52c5fdc3be180229 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T14:43:40.388 INFO:teuthology.misc.health.smithi027.stderr:2023-01-24T14:43:40.384+0000 7fdce9ffb700 1 -- 172.21.15.27:0/1223060905 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fdcd4009d90 con 0x7fdcec0a3410 2023-01-24T14:43:40.388 INFO:teuthology.misc.health.smithi027.stderr:2023-01-24T14:43:40.384+0000 7fdce9ffb700 1 -- 172.21.15.27:0/1223060905 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7fdcd400e230 con 0x7fdcec0a3410 2023-01-24T14:43:40.388 INFO:teuthology.misc.health.smithi027.stderr:2023-01-24T14:43:40.384+0000 7fdce9ffb700 1 -- 172.21.15.27:0/1223060905 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fdcd400e4d0 con 0x7fdcec0a3410 2023-01-24T14:43:40.388 INFO:teuthology.misc.health.smithi027.stderr:2023-01-24T14:43:40.384+0000 7fdcf175f700 1 -- 172.21.15.27:0/1223060905 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdcec0a3410 msgr2=0x7fdcec1373b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T14:43:40.389 INFO:teuthology.misc.health.smithi027.stderr:2023-01-24T14:43:40.384+0000 7fdcf175f700 1 --2- 172.21.15.27:0/1223060905 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdcec0a3410 0x7fdcec1373b0 crc :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T14:43:40.389 INFO:teuthology.misc.health.smithi027.stderr:2023-01-24T14:43:40.384+0000 7fdcf175f700 1 -- 172.21.15.27:0/1223060905 shutdown_connections 2023-01-24T14:43:40.389 INFO:teuthology.misc.health.smithi027.stderr:2023-01-24T14:43:40.384+0000 7fdcf175f700 1 --2- 172.21.15.27:0/1223060905 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdcec0a3410 0x7fdcec1373b0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T14:43:40.389 INFO:teuthology.misc.health.smithi027.stderr:2023-01-24T14:43:40.384+0000 7fdcf175f700 1 -- 172.21.15.27:0/1223060905 >> 172.21.15.27:0/1223060905 conn(0x7fdcec0a4960 msgr2=0x7fdcec0a4d70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T14:43:40.389 INFO:teuthology.misc.health.smithi027.stderr:2023-01-24T14:43:40.385+0000 7fdcf175f700 1 -- 172.21.15.27:0/1223060905 shutdown_connections 2023-01-24T14:43:40.390 INFO:teuthology.misc.health.smithi027.stderr:2023-01-24T14:43:40.385+0000 7fdcf175f700 1 -- 172.21.15.27:0/1223060905 wait complete. 2023-01-24T14:43:40.392 INFO:teuthology.misc.health.smithi027.stderr:2023-01-24T14:43:40.386+0000 7fdcf175f700 1 Processor -- start 2023-01-24T14:43:40.392 INFO:teuthology.misc.health.smithi027.stderr:2023-01-24T14:43:40.386+0000 7fdcf175f700 1 -- start start 2023-01-24T14:43:40.392 INFO:teuthology.misc.health.smithi027.stderr:2023-01-24T14:43:40.387+0000 7fdcf175f700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdcec0a28f0 0x7fdcec11f180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T14:43:40.393 INFO:teuthology.misc.health.smithi027.stderr:2023-01-24T14:43:40.387+0000 7fdcf175f700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcec1203d0 con 0x7fdcec0a28f0 2023-01-24T14:43:40.393 INFO:teuthology.misc.health.smithi027.stderr:2023-01-24T14:43:40.387+0000 7fdceaffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdcec0a28f0 0x7fdcec11f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T14:43:40.393 INFO:teuthology.misc.health.smithi027.stderr:2023-01-24T14:43:40.387+0000 7fdceaffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdcec0a28f0 0x7fdcec11f180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48790/0 (socket says 172.21.15.27:48790) 2023-01-24T14:43:40.393 INFO:teuthology.misc.health.smithi027.stderr:2023-01-24T14:43:40.387+0000 7fdceaffd700 1 -- 172.21.15.27:0/2976897728 learned_addr learned my addr 172.21.15.27:0/2976897728 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-01-24T14:43:40.393 INFO:teuthology.misc.health.smithi027.stderr:2023-01-24T14:43:40.387+0000 7fdceaffd700 1 -- 172.21.15.27:0/2976897728 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcec098920 con 0x7fdcec0a28f0 2023-01-24T14:43:40.394 INFO:teuthology.misc.health.smithi027.stderr:2023-01-24T14:43:40.387+0000 7fdceaffd700 1 --2- 172.21.15.27:0/2976897728 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdcec0a28f0 0x7fdcec11f180 crc :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T14:43:40.394 INFO:teuthology.misc.health.smithi027.stderr:2023-01-24T14:43:40.387+0000 7fdce37fe700 1 -- 172.21.15.27:0/2976897728 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fdce40059a0 con 0x7fdcec0a28f0 2023-01-24T14:43:40.394 INFO:teuthology.misc.health.smithi027.stderr:2023-01-24T14:43:40.387+0000 7fdce37fe700 1 -- 172.21.15.27:0/2976897728 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7fdce4005b00 con 0x7fdcec0a28f0 2023-01-24T14:43:40.394 INFO:teuthology.misc.health.smithi027.stderr:2023-01-24T14:43:40.387+0000 7fdcf175f700 1 -- 172.21.15.27:0/2976897728 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdcec099240 con 0x7fdcec0a28f0 2023-01-24T14:43:40.394 INFO:teuthology.misc.health.smithi027.stderr:2023-01-24T14:43:40.387+0000 7fdce37fe700 1 -- 172.21.15.27:0/2976897728 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fdce4005da0 con 0x7fdcec0a28f0 2023-01-24T14:43:40.395 INFO:teuthology.misc.health.smithi027.stderr:2023-01-24T14:43:40.387+0000 7fdcf175f700 1 -- 172.21.15.27:0/2976897728 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdcec098f30 con 0x7fdcec0a28f0 2023-01-24T14:43:40.395 INFO:teuthology.misc.health.smithi027.stderr:2023-01-24T14:43:40.388+0000 7fdce37fe700 1 -- 172.21.15.27:0/2976897728 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 4) v1 ==== 42729+0+0 (crc 0 0 0) 0x7fdce40070b0 con 0x7fdcec0a28f0 2023-01-24T14:43:40.395 INFO:teuthology.misc.health.smithi027.stderr:2023-01-24T14:43:40.388+0000 7fdcf175f700 1 -- 172.21.15.27:0/2976897728 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdcec0a1b50 con 0x7fdcec0a28f0 2023-01-24T14:43:40.395 INFO:teuthology.misc.health.smithi027.stderr:2023-01-24T14:43:40.388+0000 7fdce37fe700 1 --2- 172.21.15.27:0/2976897728 >> [v2:172.21.15.35:6808/101897,v1:172.21.15.35:6809/101897] conn(0x7fdccc035790 0x7fdccc037ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T14:43:40.395 INFO:teuthology.misc.health.smithi027.stderr:2023-01-24T14:43:40.388+0000 7fdce37fe700 1 -- 172.21.15.27:0/2976897728 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 2286+0+0 (crc 0 0 0) 0x7fdce4043ff0 con 0x7fdcec0a28f0 2023-01-24T14:43:40.396 INFO:teuthology.misc.health.smithi027.stderr:2023-01-24T14:43:40.389+0000 7fdcea7fc700 1 --2- 172.21.15.27:0/2976897728 >> [v2:172.21.15.35:6808/101897,v1:172.21.15.35:6809/101897] conn(0x7fdccc035790 0x7fdccc037ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T14:43:40.396 INFO:teuthology.misc.health.smithi027.stderr:2023-01-24T14:43:40.389+0000 7fdcea7fc700 1 --2- 172.21.15.27:0/2976897728 >> [v2:172.21.15.35:6808/101897,v1:172.21.15.35:6809/101897] conn(0x7fdccc035790 0x7fdccc037ca0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T14:43:40.396 INFO:teuthology.misc.health.smithi027.stderr:2023-01-24T14:43:40.393+0000 7fdce37fe700 1 -- 172.21.15.27:0/2976897728 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172632 (crc 0 0 0) 0x7fdce400a4a0 con 0x7fdcec0a28f0 2023-01-24T14:43:40.586 INFO:teuthology.misc.health.smithi027.stderr:2023-01-24T14:43:40.585+0000 7fdcf175f700 1 -- 172.21.15.27:0/2976897728 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdcec11e6f0 con 0x7fdcec0a28f0 2023-01-24T14:43:40.587 INFO:teuthology.misc.health.smithi027.stderr:2023-01-24T14:43:40.585+0000 7fdce37fe700 1 -- 172.21.15.27:0/2976897728 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+6509 (crc 0 0 0) 0x7fdce400a4a0 con 0x7fdcec0a28f0 2023-01-24T14:43:40.587 INFO:teuthology.misc.health.smithi027.stdout: 2023-01-24T14:43:40.587 INFO:teuthology.misc.health.smithi027.stdout:{"epoch":10,"fsid":"5e79a879-b90d-445c-b439-aec3d80a8b8a","created":"2023-01-24T14:43:31.588819+0000","modified":"2023-01-24T14:43:39.783371+0000","last_up_change":"2023-01-24T14:43:37.777525+0000","last_in_change":"2023-01-24T14:43:32.895676+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-01-24T14:43:39.392444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"ed53e235-e041-46ca-b000-fecf0c6ce916","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6803","nonce":102630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":102630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6805","nonce":102630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":102630}]},"public_addr":"172.21.15.27:6803/102630","cluster_addr":"172.21.15.27:6802/102630","heartbeat_back_addr":"172.21.15.27:6805/102630","heartbeat_front_addr":"172.21.15.27:6804/102630","state":["exists","up"]},{"osd":1,"uuid":"bbbf53aa-af7f-499d-a2d5-3ac39dcc810b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6801","nonce":102631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":102631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":102631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6807","nonce":102631}]},"public_addr":"172.21.15.27:6801/102631","cluster_addr":"172.21.15.27:6800/102631","heartbeat_back_addr":"172.21.15.27:6806/102631","heartbeat_front_addr":"172.21.15.27:6807/102631","state":["exists","up"]},{"osd":2,"uuid":"14ce6465-900d-4d15-95d8-115fcb049310","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.35:6802","nonce":102221}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.35:6803","nonce":102221}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.35:6807","nonce":102221}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.35:6806","nonce":102221}]},"public_addr":"172.21.15.35:6802/102221","cluster_addr":"172.21.15.35:6803/102221","heartbeat_back_addr":"172.21.15.35:6807/102221","heartbeat_front_addr":"172.21.15.35:6806/102221","state":["exists","up"]},{"osd":3,"uuid":"cf19c994-afc8-4b3f-b6a3-7a4df2680f89","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.35:6800","nonce":102222}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.35:6801","nonce":102222}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.35:6805","nonce":102222}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.35:6804","nonce":102222}]},"public_addr":"172.21.15.35:6800/102222","cluster_addr":"172.21.15.35:6801/102222","heartbeat_back_addr":"172.21.15.35:6805/102222","heartbeat_front_addr":"172.21.15.35:6804/102222","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-01-24T14:43:40.589 INFO:teuthology.misc.health.smithi027.stderr:2023-01-24T14:43:40.587+0000 7fdcf175f700 1 -- 172.21.15.27:0/2976897728 >> [v2:172.21.15.35:6808/101897,v1:172.21.15.35:6809/101897] conn(0x7fdccc035790 msgr2=0x7fdccc037ca0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T14:43:40.589 INFO:teuthology.misc.health.smithi027.stderr:2023-01-24T14:43:40.587+0000 7fdcf175f700 1 --2- 172.21.15.27:0/2976897728 >> [v2:172.21.15.35:6808/101897,v1:172.21.15.35:6809/101897] conn(0x7fdccc035790 0x7fdccc037ca0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T14:43:40.589 INFO:teuthology.misc.health.smithi027.stderr:2023-01-24T14:43:40.587+0000 7fdcf175f700 1 -- 172.21.15.27:0/2976897728 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdcec0a28f0 msgr2=0x7fdcec11f180 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T14:43:40.590 INFO:teuthology.misc.health.smithi027.stderr:2023-01-24T14:43:40.587+0000 7fdcf175f700 1 --2- 172.21.15.27:0/2976897728 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdcec0a28f0 0x7fdcec11f180 crc :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T14:43:40.590 INFO:teuthology.misc.health.smithi027.stderr:2023-01-24T14:43:40.587+0000 7fdcf175f700 1 -- 172.21.15.27:0/2976897728 shutdown_connections 2023-01-24T14:43:40.590 INFO:teuthology.misc.health.smithi027.stderr:2023-01-24T14:43:40.587+0000 7fdcf175f700 1 --2- 172.21.15.27:0/2976897728 >> [v2:172.21.15.35:6808/101897,v1:172.21.15.35:6809/101897] conn(0x7fdccc035790 0x7fdccc037ca0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T14:43:40.590 INFO:teuthology.misc.health.smithi027.stderr:2023-01-24T14:43:40.587+0000 7fdcf175f700 1 --2- 172.21.15.27:0/2976897728 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdcec0a28f0 0x7fdcec11f180 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T14:43:40.591 INFO:teuthology.misc.health.smithi027.stderr:2023-01-24T14:43:40.587+0000 7fdcf175f700 1 -- 172.21.15.27:0/2976897728 >> 172.21.15.27:0/2976897728 conn(0x7fdcec0a4960 msgr2=0x7fdcec1ba020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T14:43:40.591 INFO:teuthology.misc.health.smithi027.stderr:2023-01-24T14:43:40.587+0000 7fdcf175f700 1 -- 172.21.15.27:0/2976897728 shutdown_connections 2023-01-24T14:43:40.591 INFO:teuthology.misc.health.smithi027.stderr:2023-01-24T14:43:40.587+0000 7fdcf175f700 1 -- 172.21.15.27:0/2976897728 wait complete. 2023-01-24T14:43:40.600 DEBUG:teuthology.misc:4 of 4 OSDs are up 2023-01-24T14:43:40.600 INFO:tasks.ceph:Creating RBD pool 2023-01-24T14:43:40.600 DEBUG:teuthology.orchestra.run.smithi027:> sudo ceph --cluster ceph osd pool create rbd 8 2023-01-24T14:43:40.733 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:40.730+0000 7ff86b5b3700 1 Processor -- start 2023-01-24T14:43:40.733 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:40.730+0000 7ff86b5b3700 1 -- start start 2023-01-24T14:43:40.734 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:40.731+0000 7ff86b5b3700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ff8641308f0 0x7ff864135080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T14:43:40.734 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:40.731+0000 7ff86b5b3700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff86409b040 con 0x7ff86412fdd0 2023-01-24T14:43:40.734 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:40.731+0000 7ff86b5b3700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8640713e0 con 0x7ff8641308f0 2023-01-24T14:43:40.734 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:40.731+0000 7ff868b4e700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ff8641308f0 0x7ff864135080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T14:43:40.734 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:40.731+0000 7ff86934f700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7ff86412fdd0 0x7ff8641301e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:44910/0 (socket says 172.21.15.27:44910) 2023-01-24T14:43:40.735 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:40.731+0000 7ff86934f700 1 -- 172.21.15.27:0/2422375069 learned_addr learned my addr 172.21.15.27:0/2422375069 (peer_addr_for_me v1:172.21.15.27:0/0) 2023-01-24T14:43:40.735 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:40.732+0000 7ff868b4e700 1 -- 172.21.15.27:0/2422375069 >> v1:172.21.15.27:6789/0 conn(0x7ff86412fdd0 legacy=0x7ff8641301e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T14:43:40.735 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:40.732+0000 7ff868b4e700 1 -- 172.21.15.27:0/2422375069 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff864099ac0 con 0x7ff8641308f0 2023-01-24T14:43:40.735 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:40.732+0000 7ff868b4e700 1 --2- 172.21.15.27:0/2422375069 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff8641308f0 0x7ff864135080 crc :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be745f089a47e427 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T14:43:40.735 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:40.732+0000 7ff857fff700 1 -- 172.21.15.27:0/2422375069 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7ff858009cd0 con 0x7ff8641308f0 2023-01-24T14:43:40.736 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:40.732+0000 7ff857fff700 1 -- 172.21.15.27:0/2422375069 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7ff858009e30 con 0x7ff8641308f0 2023-01-24T14:43:40.736 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:40.732+0000 7ff857fff700 1 -- 172.21.15.27:0/2422375069 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7ff85800e360 con 0x7ff8641308f0 2023-01-24T14:43:40.738 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:40.732+0000 7ff86b5b3700 1 -- 172.21.15.27:0/2422375069 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff8641308f0 msgr2=0x7ff864135080 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T14:43:40.739 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:40.732+0000 7ff86b5b3700 1 --2- 172.21.15.27:0/2422375069 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff8641308f0 0x7ff864135080 crc :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T14:43:40.739 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:40.732+0000 7ff86b5b3700 1 -- 172.21.15.27:0/2422375069 shutdown_connections 2023-01-24T14:43:40.739 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:40.732+0000 7ff86b5b3700 1 --2- 172.21.15.27:0/2422375069 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff8641308f0 0x7ff864135080 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T14:43:40.739 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:40.733+0000 7ff86b5b3700 1 -- 172.21.15.27:0/2422375069 >> 172.21.15.27:0/2422375069 conn(0x7ff8640a3270 msgr2=0x7ff86406f970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T14:43:40.739 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:40.733+0000 7ff86b5b3700 1 -- 172.21.15.27:0/2422375069 shutdown_connections 2023-01-24T14:43:40.740 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:40.733+0000 7ff86b5b3700 1 -- 172.21.15.27:0/2422375069 wait complete. 2023-01-24T14:43:40.740 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:40.734+0000 7ff86b5b3700 1 Processor -- start 2023-01-24T14:43:40.740 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:40.734+0000 7ff86b5b3700 1 -- start start 2023-01-24T14:43:40.740 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:40.734+0000 7ff86b5b3700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff86412fdd0 0x7ff8641c3780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T14:43:40.740 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:40.734+0000 7ff86b5b3700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff86406c8d0 con 0x7ff86412fdd0 2023-01-24T14:43:40.741 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:40.735+0000 7ff86934f700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff86412fdd0 0x7ff8641c3780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T14:43:40.741 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:40.735+0000 7ff86934f700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff86412fdd0 0x7ff8641c3780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48808/0 (socket says 172.21.15.27:48808) 2023-01-24T14:43:40.741 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:40.735+0000 7ff86934f700 1 -- 172.21.15.27:0/477052500 learned_addr learned my addr 172.21.15.27:0/477052500 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-01-24T14:43:40.741 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:40.735+0000 7ff86934f700 1 -- 172.21.15.27:0/477052500 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8640990f0 con 0x7ff86412fdd0 2023-01-24T14:43:40.741 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:40.735+0000 7ff86934f700 1 --2- 172.21.15.27:0/477052500 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff86412fdd0 0x7ff8641c3780 crc :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T14:43:40.742 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:40.735+0000 7ff855ffb700 1 -- 172.21.15.27:0/477052500 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7ff860005a60 con 0x7ff86412fdd0 2023-01-24T14:43:40.742 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:40.735+0000 7ff855ffb700 1 -- 172.21.15.27:0/477052500 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7ff860005bc0 con 0x7ff86412fdd0 2023-01-24T14:43:40.742 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:40.735+0000 7ff86b5b3700 1 -- 172.21.15.27:0/477052500 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff864071680 con 0x7ff86412fdd0 2023-01-24T14:43:40.742 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:40.735+0000 7ff855ffb700 1 -- 172.21.15.27:0/477052500 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7ff860005e60 con 0x7ff86412fdd0 2023-01-24T14:43:40.742 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:40.735+0000 7ff86b5b3700 1 -- 172.21.15.27:0/477052500 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff86409a1e0 con 0x7ff86412fdd0 2023-01-24T14:43:40.742 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:40.736+0000 7ff855ffb700 1 -- 172.21.15.27:0/477052500 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 4) v1 ==== 42729+0+0 (crc 0 0 0) 0x7ff86000c7b0 con 0x7ff86412fdd0 2023-01-24T14:43:40.743 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:40.736+0000 7ff855ffb700 1 --2- 172.21.15.27:0/477052500 >> [v2:172.21.15.35:6808/101897,v1:172.21.15.35:6809/101897] conn(0x7ff848035790 0x7ff848037ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T14:43:40.743 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:40.736+0000 7ff855ffb700 1 -- 172.21.15.27:0/477052500 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 2286+0+0 (crc 0 0 0) 0x7ff860044040 con 0x7ff86412fdd0 2023-01-24T14:43:40.743 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:40.736+0000 7ff86b5b3700 1 -- 172.21.15.27:0/477052500 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff86406d020 con 0x7ff86412fdd0 2023-01-24T14:43:40.743 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:40.737+0000 7ff868b4e700 1 --2- 172.21.15.27:0/477052500 >> [v2:172.21.15.35:6808/101897,v1:172.21.15.35:6809/101897] conn(0x7ff848035790 0x7ff848037ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T14:43:40.743 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:40.737+0000 7ff868b4e700 1 --2- 172.21.15.27:0/477052500 >> [v2:172.21.15.35:6808/101897,v1:172.21.15.35:6809/101897] conn(0x7ff848035790 0x7ff848037ca0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T14:43:40.744 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:40.741+0000 7ff855ffb700 1 -- 172.21.15.27:0/477052500 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172632 (crc 0 0 0) 0x7ff86000a670 con 0x7ff86412fdd0 2023-01-24T14:43:40.936 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:40.935+0000 7ff86b5b3700 1 -- 172.21.15.27:0/477052500 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7ff86406f660 con 0x7ff86412fdd0 2023-01-24T14:43:41.801 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:41.800+0000 7ff855ffb700 1 -- 172.21.15.27:0/477052500 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]=0 pool 'rbd' created v12) v1 ==== 109+0+0 (crc 0 0 0) 0x7ff860006040 con 0x7ff86412fdd0 2023-01-24T14:43:41.804 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:41.802+0000 7ff86b5b3700 1 -- 172.21.15.27:0/477052500 >> [v2:172.21.15.35:6808/101897,v1:172.21.15.35:6809/101897] conn(0x7ff848035790 msgr2=0x7ff848037ca0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T14:43:41.804 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:41.802+0000 7ff86b5b3700 1 --2- 172.21.15.27:0/477052500 >> [v2:172.21.15.35:6808/101897,v1:172.21.15.35:6809/101897] conn(0x7ff848035790 0x7ff848037ca0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T14:43:41.804 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:41.802+0000 7ff86b5b3700 1 -- 172.21.15.27:0/477052500 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff86412fdd0 msgr2=0x7ff8641c3780 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T14:43:41.805 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:41.802+0000 7ff86b5b3700 1 --2- 172.21.15.27:0/477052500 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff86412fdd0 0x7ff8641c3780 crc :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T14:43:41.805 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:41.802+0000 7ff86b5b3700 1 -- 172.21.15.27:0/477052500 shutdown_connections 2023-01-24T14:43:41.805 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:41.802+0000 7ff86b5b3700 1 --2- 172.21.15.27:0/477052500 >> [v2:172.21.15.35:6808/101897,v1:172.21.15.35:6809/101897] conn(0x7ff848035790 0x7ff848037ca0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T14:43:41.805 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:41.802+0000 7ff86b5b3700 1 --2- 172.21.15.27:0/477052500 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff86412fdd0 0x7ff8641c3780 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T14:43:41.805 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:41.802+0000 7ff86b5b3700 1 -- 172.21.15.27:0/477052500 >> 172.21.15.27:0/477052500 conn(0x7ff8640a3270 msgr2=0x7ff8641b7cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T14:43:41.806 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:41.803+0000 7ff86b5b3700 1 -- 172.21.15.27:0/477052500 shutdown_connections 2023-01-24T14:43:41.806 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:41.803+0000 7ff86b5b3700 1 -- 172.21.15.27:0/477052500 wait complete. 2023-01-24T14:43:41.806 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'rbd' created 2023-01-24T14:43:41.817 DEBUG:teuthology.orchestra.run.smithi027:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2023-01-24T14:43:41.958 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:41.955+0000 7f6ef30c9700 1 Processor -- start 2023-01-24T14:43:41.958 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:41.955+0000 7f6ef30c9700 1 -- start start 2023-01-24T14:43:41.959 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:41.956+0000 7f6ef30c9700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f6eec1300e0 0x7f6eec1304f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T14:43:41.959 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:41.956+0000 7f6ef30c9700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6eec09b7e0 con 0x7f6eec130a30 2023-01-24T14:43:41.959 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:41.956+0000 7f6ef30c9700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6eec06a6c0 con 0x7f6eec1300e0 2023-01-24T14:43:41.959 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:41.956+0000 7f6ef0e65700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f6eec1300e0 0x7f6eec1304f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T14:43:41.959 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:41.956+0000 7f6eebfff700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f6eec130a30 0x7f6eec09cd90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:44926/0 (socket says 172.21.15.27:44926) 2023-01-24T14:43:41.960 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:41.956+0000 7f6eebfff700 1 -- 172.21.15.27:0/371985940 learned_addr learned my addr 172.21.15.27:0/371985940 (peer_addr_for_me v1:172.21.15.27:0/0) 2023-01-24T14:43:41.960 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:41.956+0000 7f6ef0e65700 1 -- 172.21.15.27:0/371985940 >> v1:172.21.15.27:6789/0 conn(0x7f6eec130a30 legacy=0x7f6eec09cd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T14:43:41.960 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:41.956+0000 7f6ef0e65700 1 -- 172.21.15.27:0/371985940 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6eec06cf60 con 0x7f6eec1300e0 2023-01-24T14:43:41.960 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:41.957+0000 7f6ef0e65700 1 --2- 172.21.15.27:0/371985940 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6eec1300e0 0x7f6eec1304f0 crc :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24fa820ce0f6a657 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T14:43:41.960 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:41.957+0000 7f6eeb7fe700 1 -- 172.21.15.27:0/371985940 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f6ee0009cd0 con 0x7f6eec1300e0 2023-01-24T14:43:41.961 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:41.957+0000 7f6eeb7fe700 1 -- 172.21.15.27:0/371985940 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f6ee0009e30 con 0x7f6eec1300e0 2023-01-24T14:43:41.961 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:41.957+0000 7f6eeb7fe700 1 -- 172.21.15.27:0/371985940 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f6ee000e360 con 0x7f6eec1300e0 2023-01-24T14:43:41.963 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:41.958+0000 7f6ef30c9700 1 -- 172.21.15.27:0/371985940 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6eec1300e0 msgr2=0x7f6eec1304f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T14:43:41.964 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:41.958+0000 7f6ef30c9700 1 --2- 172.21.15.27:0/371985940 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6eec1300e0 0x7f6eec1304f0 crc :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T14:43:41.964 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:41.958+0000 7f6ef30c9700 1 -- 172.21.15.27:0/371985940 shutdown_connections 2023-01-24T14:43:41.964 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:41.958+0000 7f6ef30c9700 1 --2- 172.21.15.27:0/371985940 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6eec1300e0 0x7f6eec1304f0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T14:43:41.964 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:41.958+0000 7f6ef30c9700 1 -- 172.21.15.27:0/371985940 >> 172.21.15.27:0/371985940 conn(0x7f6eec0a13c0 msgr2=0x7f6eec0a17d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T14:43:41.965 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:41.958+0000 7f6ef30c9700 1 -- 172.21.15.27:0/371985940 shutdown_connections 2023-01-24T14:43:41.965 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:41.958+0000 7f6ef30c9700 1 -- 172.21.15.27:0/371985940 wait complete. 2023-01-24T14:43:41.965 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:41.959+0000 7f6ef30c9700 1 Processor -- start 2023-01-24T14:43:41.966 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:41.959+0000 7f6ef30c9700 1 -- start start 2023-01-24T14:43:41.966 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:41.960+0000 7f6ef30c9700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6eec1300e0 0x7f6eec119c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T14:43:41.966 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:41.960+0000 7f6ef30c9700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6eec11d410 con 0x7f6eec1300e0 2023-01-24T14:43:41.966 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:41.960+0000 7f6ef0e65700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6eec1300e0 0x7f6eec119c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T14:43:41.966 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:41.960+0000 7f6ef0e65700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6eec1300e0 0x7f6eec119c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48834/0 (socket says 172.21.15.27:48834) 2023-01-24T14:43:41.967 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:41.960+0000 7f6ef0e65700 1 -- 172.21.15.27:0/2674206410 learned_addr learned my addr 172.21.15.27:0/2674206410 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-01-24T14:43:41.967 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:41.960+0000 7f6ef0e65700 1 -- 172.21.15.27:0/2674206410 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6eec06a8e0 con 0x7f6eec1300e0 2023-01-24T14:43:41.967 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:41.961+0000 7f6ef0e65700 1 --2- 172.21.15.27:0/2674206410 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6eec1300e0 0x7f6eec119c20 crc :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T14:43:41.967 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:41.961+0000 7f6ee97fa700 1 -- 172.21.15.27:0/2674206410 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f6ee0013740 con 0x7f6eec1300e0 2023-01-24T14:43:41.967 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:41.961+0000 7f6ef30c9700 1 -- 172.21.15.27:0/2674206410 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6eec06c1f0 con 0x7f6eec1300e0 2023-01-24T14:43:41.967 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:41.961+0000 7f6ee97fa700 1 -- 172.21.15.27:0/2674206410 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f6ee00138a0 con 0x7f6eec1300e0 2023-01-24T14:43:41.968 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:41.961+0000 7f6ee97fa700 1 -- 172.21.15.27:0/2674206410 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f6ee0013b40 con 0x7f6eec1300e0 2023-01-24T14:43:41.968 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:41.961+0000 7f6ef30c9700 1 -- 172.21.15.27:0/2674206410 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6eec06e940 con 0x7f6eec1300e0 2023-01-24T14:43:41.968 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:41.962+0000 7f6ee97fa700 1 -- 172.21.15.27:0/2674206410 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 4) v1 ==== 42729+0+0 (crc 0 0 0) 0x7f6ee0013d40 con 0x7f6eec1300e0 2023-01-24T14:43:41.968 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:41.962+0000 7f6ee97fa700 1 --2- 172.21.15.27:0/2674206410 >> [v2:172.21.15.35:6808/101897,v1:172.21.15.35:6809/101897] conn(0x7f6ed0035740 0x7f6ed0037c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T14:43:41.968 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:41.962+0000 7f6ee97fa700 1 -- 172.21.15.27:0/2674206410 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(12..12 src has 1..12) v4 ==== 2639+0+0 (crc 0 0 0) 0x7f6ee0018040 con 0x7f6eec1300e0 2023-01-24T14:43:41.969 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:41.962+0000 7f6eebfff700 1 --2- 172.21.15.27:0/2674206410 >> [v2:172.21.15.35:6808/101897,v1:172.21.15.35:6809/101897] conn(0x7f6ed0035740 0x7f6ed0037c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T14:43:41.969 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:41.962+0000 7f6ef30c9700 1 -- 172.21.15.27:0/2674206410 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6eec072260 con 0x7f6eec1300e0 2023-01-24T14:43:41.969 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:41.962+0000 7f6eebfff700 1 --2- 172.21.15.27:0/2674206410 >> [v2:172.21.15.35:6808/101897,v1:172.21.15.35:6809/101897] conn(0x7f6ed0035740 0x7f6ed0037c50 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T14:43:41.969 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:41.967+0000 7f6ee97fa700 1 -- 172.21.15.27:0/2674206410 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172632 (crc 0 0 0) 0x7f6ee0011380 con 0x7f6eec1300e0 2023-01-24T14:43:42.191 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:42.190+0000 7f6ef30c9700 1 -- 172.21.15.27:0/2674206410 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f6eec074980 con 0x7f6eec1300e0 2023-01-24T14:43:42.844 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:42.843+0000 7f6ee97fa700 1 -- 172.21.15.27:0/2674206410 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}]=0 enabled application 'rbd' on pool 'rbd' v13) v1 ==== 173+0+0 (crc 0 0 0) 0x7f6ee000f2a0 con 0x7f6eec1300e0 2023-01-24T14:43:42.847 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:42.845+0000 7f6ef30c9700 1 -- 172.21.15.27:0/2674206410 >> [v2:172.21.15.35:6808/101897,v1:172.21.15.35:6809/101897] conn(0x7f6ed0035740 msgr2=0x7f6ed0037c50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T14:43:42.847 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:42.845+0000 7f6ef30c9700 1 --2- 172.21.15.27:0/2674206410 >> [v2:172.21.15.35:6808/101897,v1:172.21.15.35:6809/101897] conn(0x7f6ed0035740 0x7f6ed0037c50 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T14:43:42.847 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:42.845+0000 7f6ef30c9700 1 -- 172.21.15.27:0/2674206410 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6eec1300e0 msgr2=0x7f6eec119c20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T14:43:42.847 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:42.845+0000 7f6ef30c9700 1 --2- 172.21.15.27:0/2674206410 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6eec1300e0 0x7f6eec119c20 crc :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T14:43:42.848 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:42.845+0000 7f6ef30c9700 1 -- 172.21.15.27:0/2674206410 shutdown_connections 2023-01-24T14:43:42.848 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:42.845+0000 7f6ef30c9700 1 --2- 172.21.15.27:0/2674206410 >> [v2:172.21.15.35:6808/101897,v1:172.21.15.35:6809/101897] conn(0x7f6ed0035740 0x7f6ed0037c50 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T14:43:42.848 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:42.845+0000 7f6ef30c9700 1 --2- 172.21.15.27:0/2674206410 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6eec1300e0 0x7f6eec119c20 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T14:43:42.848 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:42.845+0000 7f6ef30c9700 1 -- 172.21.15.27:0/2674206410 >> 172.21.15.27:0/2674206410 conn(0x7f6eec0a13c0 msgr2=0x7f6eec1b7df0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T14:43:42.848 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:42.845+0000 7f6ef30c9700 1 -- 172.21.15.27:0/2674206410 shutdown_connections 2023-01-24T14:43:42.849 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:42.845+0000 7f6ef30c9700 1 -- 172.21.15.27:0/2674206410 wait complete. 2023-01-24T14:43:42.849 INFO:teuthology.orchestra.run.smithi027.stderr:enabled application 'rbd' on pool 'rbd' 2023-01-24T14:43:42.864 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2023-01-24T14:43:42.864 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2023-01-24T14:43:42.864 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2023-01-24T14:43:42.865 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2023-01-24T14:43:42.865 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-01-24T14:43:43.004 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:43.002+0000 7ff706054700 1 Processor -- start 2023-01-24T14:43:43.004 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:43.002+0000 7ff706054700 1 -- start start 2023-01-24T14:43:43.006 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:43.003+0000 7ff706054700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ff70006e000 0x7ff70012c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T14:43:43.006 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:43.003+0000 7ff706054700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff70006dbd0 con 0x7ff70012cd00 2023-01-24T14:43:43.006 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:43.003+0000 7ff706054700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff70012c520 con 0x7ff70006e000 2023-01-24T14:43:43.006 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:43.003+0000 7ff6ff7fe700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ff70006e000 0x7ff70012c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T14:43:43.006 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:43.003+0000 7ff6feffd700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7ff70012cd00 0x7ff70009c1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:44938/0 (socket says 172.21.15.27:44938) 2023-01-24T14:43:43.007 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:43.003+0000 7ff6feffd700 1 -- 172.21.15.27:0/182020066 learned_addr learned my addr 172.21.15.27:0/182020066 (peer_addr_for_me v1:172.21.15.27:0/0) 2023-01-24T14:43:43.007 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:43.004+0000 7ff6ff7fe700 1 -- 172.21.15.27:0/182020066 >> v1:172.21.15.27:6789/0 conn(0x7ff70012cd00 legacy=0x7ff70009c1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T14:43:43.007 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:43.004+0000 7ff6ff7fe700 1 -- 172.21.15.27:0/182020066 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff70006eea0 con 0x7ff70006e000 2023-01-24T14:43:43.007 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:43.004+0000 7ff6fe7fc700 1 -- 172.21.15.27:0/182020066 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1167997174 0 0) 0x7ff70006dbd0 con 0x7ff70012cd00 2023-01-24T14:43:43.007 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:43.004+0000 7ff6ff7fe700 1 --2- 172.21.15.27:0/182020066 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff70006e000 0x7ff70012c7c0 crc :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d674a206aca3d77e server_cookie=0 in_seq=0 out_seq=0 2023-01-24T14:43:43.008 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:43.004+0000 7ff6fe7fc700 1 -- 172.21.15.27:0/182020066 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7ff6f000e040 con 0x7ff70006e000 2023-01-24T14:43:43.008 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:43.004+0000 7ff6fe7fc700 1 -- 172.21.15.27:0/182020066 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7ff6f0007670 con 0x7ff70006e000 2023-01-24T14:43:43.008 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:43.004+0000 7ff6fe7fc700 1 -- 172.21.15.27:0/182020066 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7ff6f0007910 con 0x7ff70006e000 2023-01-24T14:43:43.011 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:43.005+0000 7ff706054700 1 -- 172.21.15.27:0/182020066 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff70006e000 msgr2=0x7ff70012c7c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T14:43:43.011 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:43.005+0000 7ff706054700 1 --2- 172.21.15.27:0/182020066 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff70006e000 0x7ff70012c7c0 crc :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T14:43:43.011 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:43.005+0000 7ff706054700 1 -- 172.21.15.27:0/182020066 shutdown_connections 2023-01-24T14:43:43.011 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:43.005+0000 7ff706054700 1 --2- 172.21.15.27:0/182020066 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff70006e000 0x7ff70012c7c0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T14:43:43.012 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:43.005+0000 7ff706054700 1 -- 172.21.15.27:0/182020066 >> 172.21.15.27:0/182020066 conn(0x7ff7000a13d0 msgr2=0x7ff70009d2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T14:43:43.012 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:43.005+0000 7ff706054700 1 -- 172.21.15.27:0/182020066 shutdown_connections 2023-01-24T14:43:43.012 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:43.005+0000 7ff706054700 1 -- 172.21.15.27:0/182020066 wait complete. 2023-01-24T14:43:43.012 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:43.006+0000 7ff706054700 1 Processor -- start 2023-01-24T14:43:43.012 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:43.006+0000 7ff706054700 1 -- start start 2023-01-24T14:43:43.013 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:43.007+0000 7ff706054700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff70006e000 0x7ff7001c40f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T14:43:43.013 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:43.007+0000 7ff706054700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff700129dd0 con 0x7ff70006e000 2023-01-24T14:43:43.013 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:43.007+0000 7ff6ff7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff70006e000 0x7ff7001c40f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T14:43:43.013 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:43.007+0000 7ff6ff7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff70006e000 0x7ff7001c40f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48854/0 (socket says 172.21.15.27:48854) 2023-01-24T14:43:43.013 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:43.007+0000 7ff6ff7fe700 1 -- 172.21.15.27:0/2260365116 learned_addr learned my addr 172.21.15.27:0/2260365116 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-01-24T14:43:43.014 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:43.008+0000 7ff6ff7fe700 1 -- 172.21.15.27:0/2260365116 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff70012d110 con 0x7ff70006e000 2023-01-24T14:43:43.014 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:43.008+0000 7ff6ff7fe700 1 --2- 172.21.15.27:0/2260365116 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff70006e000 0x7ff7001c40f0 crc :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T14:43:43.014 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:43.008+0000 7ff6e7fff700 1 -- 172.21.15.27:0/2260365116 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7ff6f000e040 con 0x7ff70006e000 2023-01-24T14:43:43.014 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:43.008+0000 7ff6e7fff700 1 -- 172.21.15.27:0/2260365116 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7ff6f0013920 con 0x7ff70006e000 2023-01-24T14:43:43.014 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:43.008+0000 7ff706054700 1 -- 172.21.15.27:0/2260365116 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff70009e040 con 0x7ff70006e000 2023-01-24T14:43:43.014 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:43.008+0000 7ff6e7fff700 1 -- 172.21.15.27:0/2260365116 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7ff6f0013bc0 con 0x7ff70006e000 2023-01-24T14:43:43.015 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:43.008+0000 7ff706054700 1 -- 172.21.15.27:0/2260365116 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff70012d2a0 con 0x7ff70006e000 2023-01-24T14:43:43.015 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:43.009+0000 7ff6e7fff700 1 -- 172.21.15.27:0/2260365116 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 4) v1 ==== 42729+0+0 (crc 0 0 0) 0x7ff6f0016400 con 0x7ff70006e000 2023-01-24T14:43:43.015 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:43.009+0000 7ff6e7fff700 1 --2- 172.21.15.27:0/2260365116 >> [v2:172.21.15.35:6808/101897,v1:172.21.15.35:6809/101897] conn(0x7ff6ec035740 0x7ff6ec037c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T14:43:43.015 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:43.009+0000 7ff6e7fff700 1 -- 172.21.15.27:0/2260365116 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(13..13 src has 1..13) v4 ==== 2650+0+0 (crc 0 0 0) 0x7ff6f0045c80 con 0x7ff70006e000 2023-01-24T14:43:43.015 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:43.009+0000 7ff706054700 1 -- 172.21.15.27:0/2260365116 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff700129b10 con 0x7ff70006e000 2023-01-24T14:43:43.016 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:43.009+0000 7ff6feffd700 1 --2- 172.21.15.27:0/2260365116 >> [v2:172.21.15.35:6808/101897,v1:172.21.15.35:6809/101897] conn(0x7ff6ec035740 0x7ff6ec037c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T14:43:43.016 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:43.010+0000 7ff6feffd700 1 --2- 172.21.15.27:0/2260365116 >> [v2:172.21.15.35:6808/101897,v1:172.21.15.35:6809/101897] conn(0x7ff6ec035740 0x7ff6ec037c50 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T14:43:43.016 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:43.014+0000 7ff6e7fff700 1 -- 172.21.15.27:0/2260365116 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172632 (crc 0 0 0) 0x7ff6f0010e10 con 0x7ff70006e000 2023-01-24T14:43:43.263 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:43.262+0000 7ff706054700 1 -- 172.21.15.27:0/2260365116 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7ff700067a10 con 0x7ff70006e000 2023-01-24T14:43:43.265 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:43.264+0000 7ff6e7fff700 1 -- 172.21.15.27:0/2260365116 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v4) v1 ==== 74+0+82533 (crc 0 0 0) 0x7ff6f000b0b0 con 0x7ff70006e000 2023-01-24T14:43:43.266 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-24T14:43:43.268 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:43.266+0000 7ff706054700 1 -- 172.21.15.27:0/2260365116 >> [v2:172.21.15.35:6808/101897,v1:172.21.15.35:6809/101897] conn(0x7ff6ec035740 msgr2=0x7ff6ec037c50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T14:43:43.268 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:43.266+0000 7ff706054700 1 --2- 172.21.15.27:0/2260365116 >> [v2:172.21.15.35:6808/101897,v1:172.21.15.35:6809/101897] conn(0x7ff6ec035740 0x7ff6ec037c50 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T14:43:43.268 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:43.266+0000 7ff706054700 1 -- 172.21.15.27:0/2260365116 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff70006e000 msgr2=0x7ff7001c40f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T14:43:43.268 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:43.266+0000 7ff706054700 1 --2- 172.21.15.27:0/2260365116 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff70006e000 0x7ff7001c40f0 crc :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T14:43:43.269 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:43.266+0000 7ff706054700 1 -- 172.21.15.27:0/2260365116 shutdown_connections 2023-01-24T14:43:43.269 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:43.266+0000 7ff706054700 1 --2- 172.21.15.27:0/2260365116 >> [v2:172.21.15.35:6808/101897,v1:172.21.15.35:6809/101897] conn(0x7ff6ec035740 0x7ff6ec037c50 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T14:43:43.269 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:43.266+0000 7ff706054700 1 --2- 172.21.15.27:0/2260365116 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff70006e000 0x7ff7001c40f0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T14:43:43.269 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:43.266+0000 7ff706054700 1 -- 172.21.15.27:0/2260365116 >> 172.21.15.27:0/2260365116 conn(0x7ff7000a13d0 msgr2=0x7ff70009bec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T14:43:43.269 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:43.266+0000 7ff706054700 1 -- 172.21.15.27:0/2260365116 shutdown_connections 2023-01-24T14:43:43.270 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:43.266+0000 7ff706054700 1 -- 172.21.15.27:0/2260365116 wait complete. 2023-01-24T14:43:43.281 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":4,"active_gid":4098,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.35:6808","nonce":101897},{"type":"v1","addr":"172.21.15.35:6809","nonce":101897}]},"active_addr":"172.21.15.35:6809/101897","active_change":"2023-01-24T14:43:37.339870+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":0,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.35:0","nonce":3032921841}]},{"addrvec":[{"type":"v2","addr":"172.21.15.35:0","nonce":3085129129}]},{"addrvec":[{"type":"v2","addr":"172.21.15.35:0","nonce":313645418}]}]}} 2023-01-24T14:43:43.285 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2023-01-24T14:43:43.286 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2023-01-24T14:43:43.286 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-01-24T14:43:43.420 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:43.418+0000 7f2378eb5700 1 Processor -- start 2023-01-24T14:43:43.420 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:43.418+0000 7f2378eb5700 1 -- start start 2023-01-24T14:43:43.421 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:43.419+0000 7f2378eb5700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2374098650 0x7f2374133da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T14:43:43.422 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:43.419+0000 7f2378eb5700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2374069650 con 0x7f2374097b30 2023-01-24T14:43:43.422 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:43.419+0000 7f2378eb5700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2374099510 con 0x7f2374098650 2023-01-24T14:43:43.422 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:43.420+0000 7f237259c700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f2374097b30 0x7f2374097f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:44942/0 (socket says 172.21.15.27:44942) 2023-01-24T14:43:43.422 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:43.420+0000 7f2369d9b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2374098650 0x7f2374133da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T14:43:43.423 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:43.420+0000 7f237259c700 1 -- 172.21.15.27:0/2583524469 learned_addr learned my addr 172.21.15.27:0/2583524469 (peer_addr_for_me v1:172.21.15.27:0/0) 2023-01-24T14:43:43.423 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:43.420+0000 7f2369d9b700 1 -- 172.21.15.27:0/2583524469 >> v1:172.21.15.27:6789/0 conn(0x7f2374097b30 legacy=0x7f2374097f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T14:43:43.423 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:43.420+0000 7f2369d9b700 1 -- 172.21.15.27:0/2583524469 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2374099f00 con 0x7f2374098650 2023-01-24T14:43:43.423 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:43.420+0000 7f2369d9b700 1 --2- 172.21.15.27:0/2583524469 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2374098650 0x7f2374133da0 crc :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5cfe1b48696b6271 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T14:43:43.423 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:43.420+0000 7f2371d9b700 1 -- 172.21.15.27:0/2583524469 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f235c009cd0 con 0x7f2374098650 2023-01-24T14:43:43.424 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:43.420+0000 7f2371d9b700 1 -- 172.21.15.27:0/2583524469 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f235c009e30 con 0x7f2374098650 2023-01-24T14:43:43.424 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:43.420+0000 7f2371d9b700 1 -- 172.21.15.27:0/2583524469 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f235c00e360 con 0x7f2374098650 2023-01-24T14:43:43.426 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:43.421+0000 7f2378eb5700 1 -- 172.21.15.27:0/2583524469 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2374098650 msgr2=0x7f2374133da0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T14:43:43.427 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:43.421+0000 7f2378eb5700 1 --2- 172.21.15.27:0/2583524469 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2374098650 0x7f2374133da0 crc :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T14:43:43.427 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:43.421+0000 7f2378eb5700 1 -- 172.21.15.27:0/2583524469 shutdown_connections 2023-01-24T14:43:43.427 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:43.421+0000 7f2378eb5700 1 --2- 172.21.15.27:0/2583524469 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2374098650 0x7f2374133da0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T14:43:43.427 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:43.421+0000 7f2378eb5700 1 -- 172.21.15.27:0/2583524469 >> 172.21.15.27:0/2583524469 conn(0x7f237409da80 msgr2=0x7f237409dec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T14:43:43.428 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:43.421+0000 7f2378eb5700 1 -- 172.21.15.27:0/2583524469 shutdown_connections 2023-01-24T14:43:43.428 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:43.421+0000 7f2378eb5700 1 -- 172.21.15.27:0/2583524469 wait complete. 2023-01-24T14:43:43.428 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:43.422+0000 7f2378eb5700 1 Processor -- start 2023-01-24T14:43:43.428 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:43.422+0000 7f2378eb5700 1 -- start start 2023-01-24T14:43:43.428 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:43.423+0000 7f2378eb5700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2374097b30 0x7f23741c4830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T14:43:43.429 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:43.423+0000 7f2378eb5700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2374118850 con 0x7f2374097b30 2023-01-24T14:43:43.429 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:43.423+0000 7f237259c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2374097b30 0x7f23741c4830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T14:43:43.429 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:43.423+0000 7f237259c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2374097b30 0x7f23741c4830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48870/0 (socket says 172.21.15.27:48870) 2023-01-24T14:43:43.429 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:43.423+0000 7f237259c700 1 -- 172.21.15.27:0/3811946361 learned_addr learned my addr 172.21.15.27:0/3811946361 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-01-24T14:43:43.429 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:43.423+0000 7f237259c700 1 -- 172.21.15.27:0/3811946361 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2374099360 con 0x7f2374097b30 2023-01-24T14:43:43.430 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:43.423+0000 7f237259c700 1 --2- 172.21.15.27:0/3811946361 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2374097b30 0x7f23741c4830 crc :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T14:43:43.430 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:43.424+0000 7f236b7fe700 1 -- 172.21.15.27:0/3811946361 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f2364005a60 con 0x7f2374097b30 2023-01-24T14:43:43.430 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:43.424+0000 7f236b7fe700 1 -- 172.21.15.27:0/3811946361 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f2364005bc0 con 0x7f2374097b30 2023-01-24T14:43:43.430 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:43.424+0000 7f236b7fe700 1 -- 172.21.15.27:0/3811946361 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f2364005e60 con 0x7f2374097b30 2023-01-24T14:43:43.430 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:43.424+0000 7f2378eb5700 1 -- 172.21.15.27:0/3811946361 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23740991d0 con 0x7f2374097b30 2023-01-24T14:43:43.431 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:43.424+0000 7f2378eb5700 1 -- 172.21.15.27:0/3811946361 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2374069260 con 0x7f2374097b30 2023-01-24T14:43:43.431 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:43.425+0000 7f236b7fe700 1 -- 172.21.15.27:0/3811946361 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 4) v1 ==== 42729+0+0 (crc 0 0 0) 0x7f236400c7b0 con 0x7f2374097b30 2023-01-24T14:43:43.431 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:43.425+0000 7f2378eb5700 1 -- 172.21.15.27:0/3811946361 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2374118650 con 0x7f2374097b30 2023-01-24T14:43:43.431 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:43.425+0000 7f236b7fe700 1 --2- 172.21.15.27:0/3811946361 >> [v2:172.21.15.35:6808/101897,v1:172.21.15.35:6809/101897] conn(0x7f2354035790 0x7f2354037ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T14:43:43.431 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:43.425+0000 7f236b7fe700 1 -- 172.21.15.27:0/3811946361 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(13..13 src has 1..13) v4 ==== 2650+0+0 (crc 0 0 0) 0x7f2364044150 con 0x7f2374097b30 2023-01-24T14:43:43.431 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:43.425+0000 7f2369d9b700 1 --2- 172.21.15.27:0/3811946361 >> [v2:172.21.15.35:6808/101897,v1:172.21.15.35:6809/101897] conn(0x7f2354035790 0x7f2354037ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T14:43:43.432 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:43.425+0000 7f2369d9b700 1 --2- 172.21.15.27:0/3811946361 >> [v2:172.21.15.35:6808/101897,v1:172.21.15.35:6809/101897] conn(0x7f2354035790 0x7f2354037ca0 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T14:43:43.432 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:43.430+0000 7f236b7fe700 1 -- 172.21.15.27:0/3811946361 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172632 (crc 0 0 0) 0x7f2364006040 con 0x7f2374097b30 2023-01-24T14:43:43.620 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:43.619+0000 7f2378eb5700 1 -- 172.21.15.27:0/3811946361 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2374074980 con 0x7f2374097b30 2023-01-24T14:43:43.621 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:43.620+0000 7f236b7fe700 1 -- 172.21.15.27:0/3811946361 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v13) v1 ==== 74+0+8036 (crc 0 0 0) 0x7f2364006040 con 0x7f2374097b30 2023-01-24T14:43:43.621 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-24T14:43:43.621 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":13,"fsid":"5e79a879-b90d-445c-b439-aec3d80a8b8a","created":"2023-01-24T14:43:31.588819+0000","modified":"2023-01-24T14:43:42.840838+0000","last_up_change":"2023-01-24T14:43:37.777525+0000","last_in_change":"2023-01-24T14:43:32.895676+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-01-24T14:43:39.392444+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-01-24T14:43:40.969666+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"ed53e235-e041-46ca-b000-fecf0c6ce916","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6803","nonce":102630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":102630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6805","nonce":102630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":102630}]},"public_addr":"172.21.15.27:6803/102630","cluster_addr":"172.21.15.27:6802/102630","heartbeat_back_addr":"172.21.15.27:6805/102630","heartbeat_front_addr":"172.21.15.27:6804/102630","state":["exists","up"]},{"osd":1,"uuid":"bbbf53aa-af7f-499d-a2d5-3ac39dcc810b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6801","nonce":102631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":102631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":102631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6807","nonce":102631}]},"public_addr":"172.21.15.27:6801/102631","cluster_addr":"172.21.15.27:6800/102631","heartbeat_back_addr":"172.21.15.27:6806/102631","heartbeat_front_addr":"172.21.15.27:6807/102631","state":["exists","up"]},{"osd":2,"uuid":"14ce6465-900d-4d15-95d8-115fcb049310","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.35:6802","nonce":102221}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.35:6803","nonce":102221}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.35:6807","nonce":102221}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.35:6806","nonce":102221}]},"public_addr":"172.21.15.35:6802/102221","cluster_addr":"172.21.15.35:6803/102221","heartbeat_back_addr":"172.21.15.35:6807/102221","heartbeat_front_addr":"172.21.15.35:6806/102221","state":["exists","up"]},{"osd":3,"uuid":"cf19c994-afc8-4b3f-b6a3-7a4df2680f89","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.35:6800","nonce":102222}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.35:6801","nonce":102222}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.35:6805","nonce":102222}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.35:6804","nonce":102222}]},"public_addr":"172.21.15.35:6800/102222","cluster_addr":"172.21.15.35:6801/102222","heartbeat_back_addr":"172.21.15.35:6805/102222","heartbeat_front_addr":"172.21.15.35:6804/102222","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-01-24T14:43:43.623 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:43.621+0000 7f2378eb5700 1 -- 172.21.15.27:0/3811946361 >> [v2:172.21.15.35:6808/101897,v1:172.21.15.35:6809/101897] conn(0x7f2354035790 msgr2=0x7f2354037ca0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T14:43:43.623 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:43.621+0000 7f2378eb5700 1 --2- 172.21.15.27:0/3811946361 >> [v2:172.21.15.35:6808/101897,v1:172.21.15.35:6809/101897] conn(0x7f2354035790 0x7f2354037ca0 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T14:43:43.623 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:43.621+0000 7f2378eb5700 1 -- 172.21.15.27:0/3811946361 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2374097b30 msgr2=0x7f23741c4830 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T14:43:43.624 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:43.621+0000 7f2378eb5700 1 --2- 172.21.15.27:0/3811946361 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2374097b30 0x7f23741c4830 crc :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T14:43:43.624 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:43.621+0000 7f2378eb5700 1 -- 172.21.15.27:0/3811946361 shutdown_connections 2023-01-24T14:43:43.624 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:43.621+0000 7f2378eb5700 1 --2- 172.21.15.27:0/3811946361 >> [v2:172.21.15.35:6808/101897,v1:172.21.15.35:6809/101897] conn(0x7f2354035790 0x7f2354037ca0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T14:43:43.624 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:43.621+0000 7f2378eb5700 1 --2- 172.21.15.27:0/3811946361 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2374097b30 0x7f23741c4830 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T14:43:43.624 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:43.621+0000 7f2378eb5700 1 -- 172.21.15.27:0/3811946361 >> 172.21.15.27:0/3811946361 conn(0x7f237409da80 msgr2=0x7f23741b6aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T14:43:43.625 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:43.622+0000 7f2378eb5700 1 -- 172.21.15.27:0/3811946361 shutdown_connections 2023-01-24T14:43:43.625 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:43.622+0000 7f2378eb5700 1 -- 172.21.15.27:0/3811946361 wait complete. 2023-01-24T14:43:43.635 INFO:tasks.ceph.ceph_manager.ceph:all up! 2023-01-24T14:43:43.636 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-01-24T14:43:43.767 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:43.766+0000 7fb5b2092700 1 Processor -- start 2023-01-24T14:43:43.768 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:43.766+0000 7fb5b2092700 1 -- start start 2023-01-24T14:43:43.770 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:43.766+0000 7fb5b2092700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb5ac0983d0 0x7fb5ac1321c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T14:43:43.770 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:43.766+0000 7fb5b2092700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5ac131ef0 con 0x7fb5ac132700 2023-01-24T14:43:43.770 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:43.766+0000 7fb5b2092700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5ac132b10 con 0x7fb5ac0983d0 2023-01-24T14:43:43.770 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:43.767+0000 7fb5ab7fe700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb5ac0983d0 0x7fb5ac1321c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T14:43:43.771 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:43.767+0000 7fb5aaffd700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fb5ac132700 0x7fb5ac09b9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:44950/0 (socket says 172.21.15.27:44950) 2023-01-24T14:43:43.771 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:43.767+0000 7fb5ab7fe700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb5ac0983d0 0x7fb5ac1321c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48876/0 (socket says 172.21.15.27:48876) 2023-01-24T14:43:43.771 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:43.767+0000 7fb5aaffd700 1 -- 172.21.15.27:0/4157422809 learned_addr learned my addr 172.21.15.27:0/4157422809 (peer_addr_for_me v1:172.21.15.27:0/0) 2023-01-24T14:43:43.771 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:43.767+0000 7fb5ab7fe700 1 -- 172.21.15.27:0/4157422809 >> v1:172.21.15.27:6789/0 conn(0x7fb5ac132700 legacy=0x7fb5ac09b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T14:43:43.771 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:43.767+0000 7fb5ab7fe700 1 -- 172.21.15.27:0/4157422809 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5ac132c50 con 0x7fb5ac0983d0 2023-01-24T14:43:43.772 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:43.767+0000 7fb5ab7fe700 1 --2- 172.21.15.27:0/4157422809 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5ac0983d0 0x7fb5ac1321c0 crc :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa47891053d38a7a server_cookie=0 in_seq=0 out_seq=0 2023-01-24T14:43:43.772 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:43.767+0000 7fb5aa7fc700 1 -- 172.21.15.27:0/4157422809 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fb59c009cd0 con 0x7fb5ac0983d0 2023-01-24T14:43:43.772 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:43.767+0000 7fb5aa7fc700 1 -- 172.21.15.27:0/4157422809 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7fb59c009e30 con 0x7fb5ac0983d0 2023-01-24T14:43:43.772 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:43.767+0000 7fb5aa7fc700 1 -- 172.21.15.27:0/4157422809 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fb59c00e360 con 0x7fb5ac0983d0 2023-01-24T14:43:43.772 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:43.768+0000 7fb5b2092700 1 -- 172.21.15.27:0/4157422809 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5ac0983d0 msgr2=0x7fb5ac1321c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T14:43:43.773 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:43.768+0000 7fb5b2092700 1 --2- 172.21.15.27:0/4157422809 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5ac0983d0 0x7fb5ac1321c0 crc :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T14:43:43.773 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:43.768+0000 7fb5b2092700 1 -- 172.21.15.27:0/4157422809 shutdown_connections 2023-01-24T14:43:43.773 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:43.768+0000 7fb5b2092700 1 --2- 172.21.15.27:0/4157422809 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5ac0983d0 0x7fb5ac1321c0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T14:43:43.773 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:43.768+0000 7fb5b2092700 1 -- 172.21.15.27:0/4157422809 >> 172.21.15.27:0/4157422809 conn(0x7fb5ac09da80 msgr2=0x7fb5ac09dec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T14:43:43.773 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:43.768+0000 7fb5b2092700 1 -- 172.21.15.27:0/4157422809 shutdown_connections 2023-01-24T14:43:43.774 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:43.768+0000 7fb5b2092700 1 -- 172.21.15.27:0/4157422809 wait complete. 2023-01-24T14:43:43.776 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:43.769+0000 7fb5b2092700 1 Processor -- start 2023-01-24T14:43:43.776 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:43.769+0000 7fb5b2092700 1 -- start start 2023-01-24T14:43:43.776 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:43.770+0000 7fb5b2092700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5ac0983d0 0x7fb5ac12e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T14:43:43.776 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:43.770+0000 7fb5b2092700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5ac071c00 con 0x7fb5ac0983d0 2023-01-24T14:43:43.777 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:43.770+0000 7fb5ab7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5ac0983d0 0x7fb5ac12e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T14:43:43.777 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:43.770+0000 7fb5ab7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5ac0983d0 0x7fb5ac12e920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48888/0 (socket says 172.21.15.27:48888) 2023-01-24T14:43:43.777 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:43.770+0000 7fb5ab7fe700 1 -- 172.21.15.27:0/2207959406 learned_addr learned my addr 172.21.15.27:0/2207959406 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-01-24T14:43:43.777 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:43.770+0000 7fb5ab7fe700 1 -- 172.21.15.27:0/2207959406 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5ac099680 con 0x7fb5ac0983d0 2023-01-24T14:43:43.777 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:43.770+0000 7fb5ab7fe700 1 --2- 172.21.15.27:0/2207959406 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5ac0983d0 0x7fb5ac12e920 crc :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T14:43:43.778 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:43.770+0000 7fb593fff700 1 -- 172.21.15.27:0/2207959406 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fb59c013530 con 0x7fb5ac0983d0 2023-01-24T14:43:43.778 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:43.770+0000 7fb593fff700 1 -- 172.21.15.27:0/2207959406 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7fb59c013690 con 0x7fb5ac0983d0 2023-01-24T14:43:43.778 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:43.771+0000 7fb5b2092700 1 -- 172.21.15.27:0/2207959406 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5ac0994f0 con 0x7fb5ac0983d0 2023-01-24T14:43:43.778 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:43.771+0000 7fb593fff700 1 -- 172.21.15.27:0/2207959406 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fb59c013930 con 0x7fb5ac0983d0 2023-01-24T14:43:43.778 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:43.771+0000 7fb5b2092700 1 -- 172.21.15.27:0/2207959406 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5ac098cd0 con 0x7fb5ac0983d0 2023-01-24T14:43:43.779 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:43.771+0000 7fb593fff700 1 -- 172.21.15.27:0/2207959406 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 4) v1 ==== 42729+0+0 (crc 0 0 0) 0x7fb59c013ae0 con 0x7fb5ac0983d0 2023-01-24T14:43:43.779 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:43.771+0000 7fb593fff700 1 --2- 172.21.15.27:0/2207959406 >> [v2:172.21.15.35:6808/101897,v1:172.21.15.35:6809/101897] conn(0x7fb598035780 0x7fb598037c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T14:43:43.779 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:43.771+0000 7fb5b2092700 1 -- 172.21.15.27:0/2207959406 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5ac074980 con 0x7fb5ac0983d0 2023-01-24T14:43:43.779 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:43.772+0000 7fb593fff700 1 -- 172.21.15.27:0/2207959406 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(13..13 src has 1..13) v4 ==== 2650+0+0 (crc 0 0 0) 0x7fb59c018040 con 0x7fb5ac0983d0 2023-01-24T14:43:43.779 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:43.772+0000 7fb5aaffd700 1 --2- 172.21.15.27:0/2207959406 >> [v2:172.21.15.35:6808/101897,v1:172.21.15.35:6809/101897] conn(0x7fb598035780 0x7fb598037c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T14:43:43.780 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:43.772+0000 7fb5aaffd700 1 --2- 172.21.15.27:0/2207959406 >> [v2:172.21.15.35:6808/101897,v1:172.21.15.35:6809/101897] conn(0x7fb598035780 0x7fb598037c90 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T14:43:43.780 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:43.776+0000 7fb593fff700 1 -- 172.21.15.27:0/2207959406 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172632 (crc 0 0 0) 0x7fb59c018e00 con 0x7fb5ac0983d0 2023-01-24T14:43:43.972 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:43.970+0000 7fb5b2092700 1 -- 172.21.15.27:0/2207959406 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb5ac11ca40 con 0x7fb5ac0983d0 2023-01-24T14:43:43.972 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:43.971+0000 7fb593fff700 1 -- 172.21.15.27:0/2207959406 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v14) v1 ==== 74+0+8023 (crc 0 0 0) 0x7fb59c014d60 con 0x7fb5ac0983d0 2023-01-24T14:43:43.973 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-24T14:43:43.973 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":14,"fsid":"5e79a879-b90d-445c-b439-aec3d80a8b8a","created":"2023-01-24T14:43:31.588819+0000","modified":"2023-01-24T14:43:43.845014+0000","last_up_change":"2023-01-24T14:43:37.777525+0000","last_in_change":"2023-01-24T14:43:32.895676+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-01-24T14:43:39.392444+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-01-24T14:43:40.969666+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"ed53e235-e041-46ca-b000-fecf0c6ce916","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6803","nonce":102630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":102630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6805","nonce":102630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":102630}]},"public_addr":"172.21.15.27:6803/102630","cluster_addr":"172.21.15.27:6802/102630","heartbeat_back_addr":"172.21.15.27:6805/102630","heartbeat_front_addr":"172.21.15.27:6804/102630","state":["exists","up"]},{"osd":1,"uuid":"bbbf53aa-af7f-499d-a2d5-3ac39dcc810b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6801","nonce":102631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":102631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":102631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6807","nonce":102631}]},"public_addr":"172.21.15.27:6801/102631","cluster_addr":"172.21.15.27:6800/102631","heartbeat_back_addr":"172.21.15.27:6806/102631","heartbeat_front_addr":"172.21.15.27:6807/102631","state":["exists","up"]},{"osd":2,"uuid":"14ce6465-900d-4d15-95d8-115fcb049310","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.35:6802","nonce":102221}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.35:6803","nonce":102221}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.35:6807","nonce":102221}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.35:6806","nonce":102221}]},"public_addr":"172.21.15.35:6802/102221","cluster_addr":"172.21.15.35:6803/102221","heartbeat_back_addr":"172.21.15.35:6807/102221","heartbeat_front_addr":"172.21.15.35:6806/102221","state":["exists","up"]},{"osd":3,"uuid":"cf19c994-afc8-4b3f-b6a3-7a4df2680f89","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.35:6800","nonce":102222}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.35:6801","nonce":102222}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.35:6805","nonce":102222}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.35:6804","nonce":102222}]},"public_addr":"172.21.15.35:6800/102222","cluster_addr":"172.21.15.35:6801/102222","heartbeat_back_addr":"172.21.15.35:6805/102222","heartbeat_front_addr":"172.21.15.35:6804/102222","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-01-24T14:43:43.975 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:43.973+0000 7fb5b2092700 1 -- 172.21.15.27:0/2207959406 >> [v2:172.21.15.35:6808/101897,v1:172.21.15.35:6809/101897] conn(0x7fb598035780 msgr2=0x7fb598037c90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T14:43:43.975 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:43.973+0000 7fb5b2092700 1 --2- 172.21.15.27:0/2207959406 >> [v2:172.21.15.35:6808/101897,v1:172.21.15.35:6809/101897] conn(0x7fb598035780 0x7fb598037c90 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T14:43:43.975 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:43.973+0000 7fb5b2092700 1 -- 172.21.15.27:0/2207959406 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5ac0983d0 msgr2=0x7fb5ac12e920 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T14:43:43.975 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:43.973+0000 7fb5b2092700 1 --2- 172.21.15.27:0/2207959406 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5ac0983d0 0x7fb5ac12e920 crc :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T14:43:43.975 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:43.973+0000 7fb5b2092700 1 -- 172.21.15.27:0/2207959406 shutdown_connections 2023-01-24T14:43:43.976 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:43.973+0000 7fb5b2092700 1 --2- 172.21.15.27:0/2207959406 >> [v2:172.21.15.35:6808/101897,v1:172.21.15.35:6809/101897] conn(0x7fb598035780 0x7fb598037c90 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T14:43:43.976 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:43.973+0000 7fb5b2092700 1 --2- 172.21.15.27:0/2207959406 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5ac0983d0 0x7fb5ac12e920 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T14:43:43.976 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:43.973+0000 7fb5b2092700 1 -- 172.21.15.27:0/2207959406 >> 172.21.15.27:0/2207959406 conn(0x7fb5ac09da80 msgr2=0x7fb5ac1b9e30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T14:43:43.977 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:43.973+0000 7fb5b2092700 1 -- 172.21.15.27:0/2207959406 shutdown_connections 2023-01-24T14:43:43.977 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:43.973+0000 7fb5b2092700 1 -- 172.21.15.27:0/2207959406 wait complete. 2023-01-24T14:43:43.987 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2023-01-24T14:43:43.988 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2023-01-24T14:43:43.988 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2023-01-24T14:43:43.989 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2023-01-24T14:43:44.133 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.132+0000 7fe3a11f1700 1 Processor -- start 2023-01-24T14:43:44.134 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.132+0000 7fe3a11f1700 1 -- start start 2023-01-24T14:43:44.134 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.132+0000 7fef6466e700 1 Processor -- start 2023-01-24T14:43:44.134 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.132+0000 7fef6466e700 1 -- start start 2023-01-24T14:43:44.135 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.133+0000 7fe3a11f1700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe39c12c010 0x7fe39c133ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T14:43:44.135 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.133+0000 7fe3a11f1700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe39c09ac30 con 0x7fe39c099fa0 2023-01-24T14:43:44.136 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.133+0000 7fe3a11f1700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe39c09a3b0 con 0x7fe39c12c010 2023-01-24T14:43:44.136 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.133+0000 7fe39a59c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe39c12c010 0x7fe39c133ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T14:43:44.136 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.133+0000 7fe39a59c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe39c12c010 0x7fe39c133ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48892/0 (socket says 172.21.15.27:48892) 2023-01-24T14:43:44.136 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.133+0000 7fe39ad9d700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fe39c099fa0 0x7fe39c12b900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:44952/0 (socket says 172.21.15.27:44952) 2023-01-24T14:43:44.136 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.133+0000 7fe39a59c700 1 -- 172.21.15.27:0/1049879984 learned_addr learned my addr 172.21.15.27:0/1049879984 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-01-24T14:43:44.137 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.134+0000 7fe39a59c700 1 -- 172.21.15.27:0/1049879984 >> v1:172.21.15.27:6789/0 conn(0x7fe39c099fa0 legacy=0x7fe39c12b900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T14:43:44.137 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.134+0000 7fe39a59c700 1 -- 172.21.15.27:0/1049879984 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe39c12b4f0 con 0x7fe39c12c010 2023-01-24T14:43:44.137 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.134+0000 7fef6466e700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fef5c097a20 0x7fef5c097e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T14:43:44.137 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.134+0000 7fef6466e700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef5c09af40 con 0x7fef5c098370 2023-01-24T14:43:44.138 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.134+0000 7fef6466e700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef5c09a380 con 0x7fef5c097a20 2023-01-24T14:43:44.138 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.134+0000 7fef6240a700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fef5c097a20 0x7fef5c097e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T14:43:44.138 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.134+0000 7fef6240a700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fef5c097a20 0x7fef5c097e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48904/0 (socket says 172.21.15.27:48904) 2023-01-24T14:43:44.138 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.134+0000 7fef6240a700 1 -- 172.21.15.27:0/3377781425 learned_addr learned my addr 172.21.15.27:0/3377781425 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-01-24T14:43:44.139 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.134+0000 7fe39a59c700 1 --2- 172.21.15.27:0/1049879984 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe39c12c010 0x7fe39c133ce0 crc :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9f7061aa98a60c0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T14:43:44.139 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.134+0000 7fef6240a700 1 -- 172.21.15.27:0/3377781425 >> v1:172.21.15.27:6789/0 conn(0x7fef5c098370 legacy=0x7fef5c12af60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T14:43:44.139 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.134+0000 7fe399d9b700 1 -- 172.21.15.27:0/1049879984 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fe390009d90 con 0x7fe39c12c010 2023-01-24T14:43:44.140 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.134+0000 7fe399d9b700 1 -- 172.21.15.27:0/1049879984 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7fe39000e230 con 0x7fe39c12c010 2023-01-24T14:43:44.140 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.134+0000 7fe399d9b700 1 -- 172.21.15.27:0/1049879984 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fe39000e4d0 con 0x7fe39c12c010 2023-01-24T14:43:44.141 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.134+0000 7fef6240a700 1 -- 172.21.15.27:0/3377781425 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef5c071f20 con 0x7fef5c097a20 2023-01-24T14:43:44.141 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.134+0000 7fef6240a700 1 --2- 172.21.15.27:0/3377781425 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fef5c097a20 0x7fef5c097e30 crc :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59e599d07a158a94 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T14:43:44.141 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.135+0000 7fef61408700 1 -- 172.21.15.27:0/3377781425 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fef5800d580 con 0x7fef5c097a20 2023-01-24T14:43:44.141 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.135+0000 7fef61408700 1 -- 172.21.15.27:0/3377781425 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7fef5800d6e0 con 0x7fef5c097a20 2023-01-24T14:43:44.142 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.135+0000 7fef61408700 1 -- 172.21.15.27:0/3377781425 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fef5800d980 con 0x7fef5c097a20 2023-01-24T14:43:44.142 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.135+0000 7fef6466e700 1 -- 172.21.15.27:0/3377781425 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fef5c097a20 msgr2=0x7fef5c097e30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T14:43:44.142 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.135+0000 7fef6466e700 1 --2- 172.21.15.27:0/3377781425 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fef5c097a20 0x7fef5c097e30 crc :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T14:43:44.142 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.135+0000 7fe3a11f1700 1 -- 172.21.15.27:0/1049879984 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe39c12c010 msgr2=0x7fe39c133ce0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T14:43:44.143 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.135+0000 7fe3a11f1700 1 --2- 172.21.15.27:0/1049879984 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe39c12c010 0x7fe39c133ce0 crc :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T14:43:44.143 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.136+0000 7fef6466e700 1 -- 172.21.15.27:0/3377781425 shutdown_connections 2023-01-24T14:43:44.143 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.136+0000 7fef6466e700 1 --2- 172.21.15.27:0/3377781425 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fef5c097a20 0x7fef5c097e30 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T14:43:44.144 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.136+0000 7fe3a11f1700 1 -- 172.21.15.27:0/1049879984 shutdown_connections 2023-01-24T14:43:44.144 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.136+0000 7fe3a11f1700 1 --2- 172.21.15.27:0/1049879984 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe39c12c010 0x7fe39c133ce0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T14:43:44.144 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.136+0000 7fef6466e700 1 -- 172.21.15.27:0/3377781425 >> 172.21.15.27:0/3377781425 conn(0x7fef5c0a13d0 msgr2=0x7fef5c09d2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T14:43:44.145 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.136+0000 7fe3a11f1700 1 -- 172.21.15.27:0/1049879984 >> 172.21.15.27:0/1049879984 conn(0x7fe39c0a13d0 msgr2=0x7fe39c09d2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T14:43:44.145 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.136+0000 7fef6466e700 1 -- 172.21.15.27:0/3377781425 shutdown_connections 2023-01-24T14:43:44.146 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.136+0000 7fe3a11f1700 1 -- 172.21.15.27:0/1049879984 shutdown_connections 2023-01-24T14:43:44.146 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.136+0000 7fef6466e700 1 -- 172.21.15.27:0/3377781425 wait complete. 2023-01-24T14:43:44.146 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.136+0000 7fe3a11f1700 1 -- 172.21.15.27:0/1049879984 wait complete. 2023-01-24T14:43:44.147 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.137+0000 7fef6466e700 1 Processor -- start 2023-01-24T14:43:44.147 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.137+0000 7fef6466e700 1 -- start start 2023-01-24T14:43:44.148 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.138+0000 7fe3a11f1700 1 Processor -- start 2023-01-24T14:43:44.148 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.138+0000 7fe3a11f1700 1 -- start start 2023-01-24T14:43:44.149 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.138+0000 7fef6466e700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fef5c097a20 0x7fef5c11edd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T14:43:44.149 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.138+0000 7fef6466e700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef5c120e80 con 0x7fef5c097a20 2023-01-24T14:43:44.149 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.138+0000 7fef6240a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fef5c097a20 0x7fef5c11edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T14:43:44.149 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.138+0000 7fef6240a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fef5c097a20 0x7fef5c11edd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48906/0 (socket says 172.21.15.27:48906) 2023-01-24T14:43:44.150 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.138+0000 7fef6240a700 1 -- 172.21.15.27:0/924089748 learned_addr learned my addr 172.21.15.27:0/924089748 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-01-24T14:43:44.150 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.139+0000 7fef6240a700 1 -- 172.21.15.27:0/924089748 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef5c09aaa0 con 0x7fef5c097a20 2023-01-24T14:43:44.150 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.139+0000 7fe3a11f1700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe39c099fa0 0x7fe39c135230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T14:43:44.151 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.139+0000 7fe3a11f1700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe39c098b30 con 0x7fe39c099fa0 2023-01-24T14:43:44.151 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.139+0000 7fe39ad9d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe39c099fa0 0x7fe39c135230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T14:43:44.151 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.139+0000 7fe39ad9d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe39c099fa0 0x7fe39c135230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48922/0 (socket says 172.21.15.27:48922) 2023-01-24T14:43:44.151 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.139+0000 7fe39ad9d700 1 -- 172.21.15.27:0/43772404 learned_addr learned my addr 172.21.15.27:0/43772404 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-01-24T14:43:44.152 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.139+0000 7fef6240a700 1 --2- 172.21.15.27:0/924089748 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fef5c097a20 0x7fef5c11edd0 crc :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T14:43:44.152 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.139+0000 7fef52ffd700 1 -- 172.21.15.27:0/924089748 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fef5800c720 con 0x7fef5c097a20 2023-01-24T14:43:44.152 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.139+0000 7fef52ffd700 1 -- 172.21.15.27:0/924089748 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7fef5800a040 con 0x7fef5c097a20 2023-01-24T14:43:44.152 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.139+0000 7fef52ffd700 1 -- 172.21.15.27:0/924089748 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fef5800c9b0 con 0x7fef5c097a20 2023-01-24T14:43:44.153 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.139+0000 7fef6466e700 1 -- 172.21.15.27:0/924089748 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef5c09a590 con 0x7fef5c097a20 2023-01-24T14:43:44.153 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.139+0000 7fe39ad9d700 1 -- 172.21.15.27:0/43772404 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe39c06dde0 con 0x7fe39c099fa0 2023-01-24T14:43:44.153 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.139+0000 7fef6466e700 1 -- 172.21.15.27:0/924089748 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef5c09e040 con 0x7fef5c097a20 2023-01-24T14:43:44.154 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.139+0000 7fe39ad9d700 1 --2- 172.21.15.27:0/43772404 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe39c099fa0 0x7fe39c135230 crc :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T14:43:44.155 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.140+0000 7fef6466e700 1 -- 172.21.15.27:0/924089748 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fef5c074980 con 0x7fef5c097a20 2023-01-24T14:43:44.155 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.140+0000 7fef52ffd700 1 -- 172.21.15.27:0/924089748 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 4) v1 ==== 42729+0+0 (crc 0 0 0) 0x7fef5800cb60 con 0x7fef5c097a20 2023-01-24T14:43:44.155 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.141+0000 7fef52ffd700 1 --2- 172.21.15.27:0/924089748 >> [v2:172.21.15.35:6808/101897,v1:172.21.15.35:6809/101897] conn(0x7fef40035780 0x7fef40037c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T14:43:44.156 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.141+0000 7fef52ffd700 1 -- 172.21.15.27:0/924089748 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 2650+0+0 (crc 0 0 0) 0x7fef58012040 con 0x7fef5c097a20 2023-01-24T14:43:44.156 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.141+0000 7fef52ffd700 1 --2- 172.21.15.27:0/924089748 >> v2:172.21.15.27:6803/102630 conn(0x7fef4003a720 0x7fef4003cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T14:43:44.156 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.141+0000 7fef52ffd700 1 -- 172.21.15.27:0/924089748 --> v2:172.21.15.27:6803/102630 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fef4003d240 con 0x7fef4003a720 2023-01-24T14:43:44.156 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.141+0000 7fef52ffd700 1 -- 172.21.15.27:0/924089748 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_get_version_reply(handle=1 version=14) v2 ==== 24+0+0 (crc 0 0 0) 0x7fef58045b60 con 0x7fef5c097a20 2023-01-24T14:43:44.156 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.141+0000 7fef61c09700 1 --2- 172.21.15.27:0/924089748 >> [v2:172.21.15.35:6808/101897,v1:172.21.15.35:6809/101897] conn(0x7fef40035780 0x7fef40037c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T14:43:44.157 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.142+0000 7fe3837fe700 1 -- 172.21.15.27:0/43772404 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fe38c0059a0 con 0x7fe39c099fa0 2023-01-24T14:43:44.157 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.142+0000 7fe3837fe700 1 -- 172.21.15.27:0/43772404 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7fe38c005b00 con 0x7fe39c099fa0 2023-01-24T14:43:44.157 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.142+0000 7fef61c09700 1 --2- 172.21.15.27:0/924089748 >> [v2:172.21.15.35:6808/101897,v1:172.21.15.35:6809/101897] conn(0x7fef40035780 0x7fef40037c90 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T14:43:44.159 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.142+0000 7fe3a11f1700 1 -- 172.21.15.27:0/43772404 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe39c09ae90 con 0x7fe39c099fa0 2023-01-24T14:43:44.159 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.142+0000 7fe3837fe700 1 -- 172.21.15.27:0/43772404 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fe38c005da0 con 0x7fe39c099fa0 2023-01-24T14:43:44.159 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.142+0000 7fe3a11f1700 1 -- 172.21.15.27:0/43772404 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe39c09df80 con 0x7fe39c099fa0 2023-01-24T14:43:44.160 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.143+0000 7fe3a11f1700 1 -- 172.21.15.27:0/43772404 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fe39c128cc0 con 0x7fe39c099fa0 2023-01-24T14:43:44.160 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.143+0000 7fe3837fe700 1 -- 172.21.15.27:0/43772404 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 4) v1 ==== 42729+0+0 (crc 0 0 0) 0x7fe38c0070b0 con 0x7fe39c099fa0 2023-01-24T14:43:44.160 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.143+0000 7fe3837fe700 1 --2- 172.21.15.27:0/43772404 >> [v2:172.21.15.35:6808/101897,v1:172.21.15.35:6809/101897] conn(0x7fe3880357e0 0x7fe388037cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T14:43:44.160 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.143+0000 7fe3837fe700 1 -- 172.21.15.27:0/43772404 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 2650+0+0 (crc 0 0 0) 0x7fe38c0440f0 con 0x7fe39c099fa0 2023-01-24T14:43:44.160 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.143+0000 7fe3837fe700 1 --2- 172.21.15.27:0/43772404 >> v2:172.21.15.35:6802/102221 conn(0x7fe38803a780 0x7fe38803cc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T14:43:44.161 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.143+0000 7fe3837fe700 1 -- 172.21.15.27:0/43772404 --> v2:172.21.15.35:6802/102221 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe38803d2a0 con 0x7fe38803a780 2023-01-24T14:43:44.161 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.143+0000 7fe3837fe700 1 -- 172.21.15.27:0/43772404 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_get_version_reply(handle=1 version=14) v2 ==== 24+0+0 (crc 0 0 0) 0x7fe38c046500 con 0x7fe39c099fa0 2023-01-24T14:43:44.161 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.143+0000 7fe39a59c700 1 --2- 172.21.15.27:0/43772404 >> [v2:172.21.15.35:6808/101897,v1:172.21.15.35:6809/101897] conn(0x7fe3880357e0 0x7fe388037cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T14:43:44.161 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.144+0000 7fe39b59e700 1 --2- 172.21.15.27:0/43772404 >> v2:172.21.15.35:6802/102221 conn(0x7fe38803a780 0x7fe38803cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-01-24T14:43:44.161 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.144+0000 7fe39a59c700 1 --2- 172.21.15.27:0/43772404 >> [v2:172.21.15.35:6808/101897,v1:172.21.15.35:6809/101897] conn(0x7fe3880357e0 0x7fe388037cf0 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T14:43:44.162 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.144+0000 7fef62c0b700 1 --2- 172.21.15.27:0/924089748 >> v2:172.21.15.27:6803/102630 conn(0x7fef4003a720 0x7fef4003cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-01-24T14:43:44.162 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.145+0000 7fdb8edb6700 1 Processor -- start 2023-01-24T14:43:44.163 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.145+0000 7fc97de4f700 1 Processor -- start 2023-01-24T14:43:44.163 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.146+0000 7fc97de4f700 1 -- start start 2023-01-24T14:43:44.163 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.145+0000 7fdb8edb6700 1 -- start start 2023-01-24T14:43:44.164 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.146+0000 7fdb8edb6700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fdb88098c00 0x7fdb8812e780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T14:43:44.164 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.146+0000 7fdb8edb6700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb8809cf30 con 0x7fdb8812ecc0 2023-01-24T14:43:44.164 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.146+0000 7fdb8edb6700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb8809cc20 con 0x7fdb88098c00 2023-01-24T14:43:44.164 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.146+0000 7fe39b59e700 1 --2- 172.21.15.27:0/43772404 >> v2:172.21.15.35:6802/102221 conn(0x7fe38803a780 0x7fe38803cc10 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T14:43:44.165 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.146+0000 7fdb8cb52700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fdb88098c00 0x7fdb8812e780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T14:43:44.165 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.146+0000 7fdb87fff700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fdb8812ecc0 0x7fdb8809bb00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:44970/0 (socket says 172.21.15.27:44970) 2023-01-24T14:43:44.166 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.146+0000 7fdb87fff700 1 -- 172.21.15.27:0/631281874 learned_addr learned my addr 172.21.15.27:0/631281874 (peer_addr_for_me v1:172.21.15.27:0/0) 2023-01-24T14:43:44.166 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.146+0000 7fdb8cb52700 1 -- 172.21.15.27:0/631281874 >> v1:172.21.15.27:6789/0 conn(0x7fdb8812ecc0 legacy=0x7fdb8809bb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T14:43:44.166 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.146+0000 7fdb8cb52700 1 -- 172.21.15.27:0/631281874 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb88099690 con 0x7fdb88098c00 2023-01-24T14:43:44.166 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.146+0000 7fdb877fe700 1 -- 172.21.15.27:0/631281874 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2253856522 0 0) 0x7fdb8809cf30 con 0x7fdb8812ecc0 2023-01-24T14:43:44.167 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.146+0000 7fc97de4f700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc97806a280 0x7fc97806a690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T14:43:44.167 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.146+0000 7fc97de4f700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc978070d30 con 0x7fc978095140 2023-01-24T14:43:44.167 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.147+0000 7fdb8cb52700 1 --2- 172.21.15.27:0/631281874 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb88098c00 0x7fdb8812e780 crc :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3dfd3a0fffe9bb82 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T14:43:44.168 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.146+0000 7fc97de4f700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc978072400 con 0x7fc97806a280 2023-01-24T14:43:44.168 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.147+0000 7fc976ffd700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc97806a280 0x7fc97806a690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T14:43:44.168 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.147+0000 7fc976ffd700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc97806a280 0x7fc97806a690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48938/0 (socket says 172.21.15.27:48938) 2023-01-24T14:43:44.168 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.147+0000 7fc976ffd700 1 -- 172.21.15.27:0/667354538 learned_addr learned my addr 172.21.15.27:0/667354538 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-01-24T14:43:44.169 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.147+0000 7fef62c0b700 1 --2- 172.21.15.27:0/924089748 >> v2:172.21.15.27:6803/102630 conn(0x7fef4003a720 0x7fef4003cbb0 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T14:43:44.169 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.147+0000 7fdb877fe700 1 -- 172.21.15.27:0/631281874 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fdb7c009cd0 con 0x7fdb88098c00 2023-01-24T14:43:44.170 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.147+0000 7fe3837fe700 1 -- 172.21.15.27:0/43772404 <== osd.2 v2:172.21.15.35:6802/102221 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+4964 (crc 0 0 0) 0x7fe38803d2a0 con 0x7fe38803a780 2023-01-24T14:43:44.170 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.147+0000 7fdb877fe700 1 -- 172.21.15.27:0/631281874 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7fdb7c009e30 con 0x7fdb88098c00 2023-01-24T14:43:44.170 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.147+0000 7fdb877fe700 1 -- 172.21.15.27:0/631281874 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fdb7c0104e0 con 0x7fdb88098c00 2023-01-24T14:43:44.171 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.147+0000 7fc976ffd700 1 -- 172.21.15.27:0/667354538 >> v1:172.21.15.27:6789/0 conn(0x7fc978095140 legacy=0x7fc978071380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T14:43:44.171 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.147+0000 7fc976ffd700 1 -- 172.21.15.27:0/667354538 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc978096880 con 0x7fc97806a280 2023-01-24T14:43:44.171 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.147+0000 7fc976ffd700 1 --2- 172.21.15.27:0/667354538 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc97806a280 0x7fc97806a690 crc :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cec578ca51e7ee9f server_cookie=0 in_seq=0 out_seq=0 2023-01-24T14:43:44.172 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.148+0000 7fc9767fc700 1 -- 172.21.15.27:0/667354538 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fc96c009d00 con 0x7fc97806a280 2023-01-24T14:43:44.172 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.148+0000 7fc9767fc700 1 -- 172.21.15.27:0/667354538 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7fc96c009e60 con 0x7fc97806a280 2023-01-24T14:43:44.172 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.148+0000 7fc9767fc700 1 -- 172.21.15.27:0/667354538 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fc96c00e360 con 0x7fc97806a280 2023-01-24T14:43:44.172 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.148+0000 7fdb8edb6700 1 -- 172.21.15.27:0/631281874 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb88098c00 msgr2=0x7fdb8812e780 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T14:43:44.173 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.148+0000 7fdb8edb6700 1 --2- 172.21.15.27:0/631281874 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb88098c00 0x7fdb8812e780 crc :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T14:43:44.173 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.148+0000 7fef52ffd700 1 -- 172.21.15.27:0/924089748 <== osd.0 v2:172.21.15.27:6803/102630 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+4964 (crc 0 0 0) 0x7fef4003d240 con 0x7fef4003a720 2023-01-24T14:43:44.174 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.148+0000 7fdb8edb6700 1 -- 172.21.15.27:0/631281874 shutdown_connections 2023-01-24T14:43:44.175 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.148+0000 7fdb8edb6700 1 --2- 172.21.15.27:0/631281874 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb88098c00 0x7fdb8812e780 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T14:43:44.175 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.148+0000 7fdb8edb6700 1 -- 172.21.15.27:0/631281874 >> 172.21.15.27:0/631281874 conn(0x7fdb8809da80 msgr2=0x7fdb8809dec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T14:43:44.175 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.148+0000 7fdb8edb6700 1 -- 172.21.15.27:0/631281874 shutdown_connections 2023-01-24T14:43:44.175 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.148+0000 7fdb8edb6700 1 -- 172.21.15.27:0/631281874 wait complete. 2023-01-24T14:43:44.175 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.149+0000 7fdb8edb6700 1 Processor -- start 2023-01-24T14:43:44.175 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.149+0000 7fdb8edb6700 1 -- start start 2023-01-24T14:43:44.176 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.151+0000 7fdb8edb6700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb88098c00 0x7fdb8809f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T14:43:44.176 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.151+0000 7fdb8edb6700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb88097a90 con 0x7fdb88098c00 2023-01-24T14:43:44.176 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.151+0000 7fdb8cb52700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb88098c00 0x7fdb8809f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T14:43:44.176 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.151+0000 7fdb8cb52700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb88098c00 0x7fdb8809f3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48952/0 (socket says 172.21.15.27:48952) 2023-01-24T14:43:44.176 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.151+0000 7fdb8cb52700 1 -- 172.21.15.27:0/2097630216 learned_addr learned my addr 172.21.15.27:0/2097630216 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-01-24T14:43:44.177 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.151+0000 7fe3a11f1700 1 -- 172.21.15.27:0/43772404 --> v2:172.21.15.35:6802/102221 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fe39c074980 con 0x7fe38803a780 2023-01-24T14:43:44.178 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.151+0000 7fdb8cb52700 1 -- 172.21.15.27:0/2097630216 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb8809a680 con 0x7fdb88098c00 2023-01-24T14:43:44.178 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.151+0000 7fdb8cb52700 1 --2- 172.21.15.27:0/2097630216 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb88098c00 0x7fdb8809f3a0 crc :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T14:43:44.178 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.152+0000 7fdb857fa700 1 -- 172.21.15.27:0/2097630216 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fdb7c011420 con 0x7fdb88098c00 2023-01-24T14:43:44.178 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.152+0000 7fdb857fa700 1 -- 172.21.15.27:0/2097630216 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7fdb7c00c040 con 0x7fdb88098c00 2023-01-24T14:43:44.179 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.152+0000 7fdb8edb6700 1 -- 172.21.15.27:0/2097630216 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb8809a4f0 con 0x7fdb88098c00 2023-01-24T14:43:44.179 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.152+0000 7fdb857fa700 1 -- 172.21.15.27:0/2097630216 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fdb7c0116b0 con 0x7fdb88098c00 2023-01-24T14:43:44.179 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.152+0000 7fc97de4f700 1 -- 172.21.15.27:0/667354538 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc97806a280 msgr2=0x7fc97806a690 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T14:43:44.179 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.152+0000 7fc97de4f700 1 --2- 172.21.15.27:0/667354538 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc97806a280 0x7fc97806a690 crc :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T14:43:44.180 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.152+0000 7fdb8edb6700 1 -- 172.21.15.27:0/2097630216 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb8812f0d0 con 0x7fdb88098c00 2023-01-24T14:43:44.180 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.152+0000 7fe3837fe700 1 -- 172.21.15.27:0/43772404 <== osd.2 v2:172.21.15.35:6802/102221 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fe39c074980 con 0x7fe38803a780 2023-01-24T14:43:44.181 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.152+0000 7fc97de4f700 1 -- 172.21.15.27:0/667354538 shutdown_connections 2023-01-24T14:43:44.181 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.152+0000 7fc97de4f700 1 --2- 172.21.15.27:0/667354538 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc97806a280 0x7fc97806a690 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T14:43:44.181 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.152+0000 7fc97de4f700 1 -- 172.21.15.27:0/667354538 >> 172.21.15.27:0/667354538 conn(0x7fc97809f630 msgr2=0x7fc97809fa40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T14:43:44.181 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.152+0000 7fc97de4f700 1 -- 172.21.15.27:0/667354538 shutdown_connections 2023-01-24T14:43:44.182 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.152+0000 7fe3a11f1700 1 -- 172.21.15.27:0/43772404 >> v2:172.21.15.35:6802/102221 conn(0x7fe38803a780 msgr2=0x7fe38803cc10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T14:43:44.182 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.152+0000 7fe3a11f1700 1 --2- 172.21.15.27:0/43772404 >> v2:172.21.15.35:6802/102221 conn(0x7fe38803a780 0x7fe38803cc10 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T14:43:44.182 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.152+0000 7fe3a11f1700 1 -- 172.21.15.27:0/43772404 >> [v2:172.21.15.35:6808/101897,v1:172.21.15.35:6809/101897] conn(0x7fe3880357e0 msgr2=0x7fe388037cf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T14:43:44.183 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.152+0000 7fe3a11f1700 1 --2- 172.21.15.27:0/43772404 >> [v2:172.21.15.35:6808/101897,v1:172.21.15.35:6809/101897] conn(0x7fe3880357e0 0x7fe388037cf0 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T14:43:44.183 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.152+0000 7fe3a11f1700 1 -- 172.21.15.27:0/43772404 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe39c099fa0 msgr2=0x7fe39c135230 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T14:43:44.183 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.152+0000 7fe3a11f1700 1 --2- 172.21.15.27:0/43772404 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe39c099fa0 0x7fe39c135230 crc :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T14:43:44.183 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.153+0000 7fdb8edb6700 1 -- 172.21.15.27:0/2097630216 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fdb881226a0 con 0x7fdb88098c00 2023-01-24T14:43:44.184 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.153+0000 7fdb857fa700 1 -- 172.21.15.27:0/2097630216 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 4) v1 ==== 42729+0+0 (crc 0 0 0) 0x7fdb7c00e040 con 0x7fdb88098c00 2023-01-24T14:43:44.184 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.153+0000 7fe3a11f1700 1 -- 172.21.15.27:0/43772404 shutdown_connections 2023-01-24T14:43:44.184 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.153+0000 7fe3a11f1700 1 --2- 172.21.15.27:0/43772404 >> v2:172.21.15.35:6802/102221 conn(0x7fe38803a780 0x7fe38803cc10 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T14:43:44.184 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.153+0000 7fe3a11f1700 1 --2- 172.21.15.27:0/43772404 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe39c099fa0 0x7fe39c135230 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T14:43:44.185 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.153+0000 7fe3a11f1700 1 --2- 172.21.15.27:0/43772404 >> [v2:172.21.15.35:6808/101897,v1:172.21.15.35:6809/101897] conn(0x7fe3880357e0 0x7fe388037cf0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T14:43:44.185 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.153+0000 7fe3a11f1700 1 -- 172.21.15.27:0/43772404 >> 172.21.15.27:0/43772404 conn(0x7fe39c0a13d0 msgr2=0x7fe39c12a170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T14:43:44.185 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.153+0000 7fdb857fa700 1 --2- 172.21.15.27:0/2097630216 >> [v2:172.21.15.35:6808/101897,v1:172.21.15.35:6809/101897] conn(0x7fdb6c035790 0x7fdb6c037ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T14:43:44.186 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.153+0000 7fe3a11f1700 1 -- 172.21.15.27:0/43772404 shutdown_connections 2023-01-24T14:43:44.186 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.153+0000 7fef6466e700 1 -- 172.21.15.27:0/924089748 --> v2:172.21.15.27:6803/102630 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fef5c121ba0 con 0x7fef4003a720 2023-01-24T14:43:44.186 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.153+0000 7fdb857fa700 1 -- 172.21.15.27:0/2097630216 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 2650+0+0 (crc 0 0 0) 0x7fdb7c017040 con 0x7fdb88098c00 2023-01-24T14:43:44.187 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.153+0000 7fdb857fa700 1 --2- 172.21.15.27:0/2097630216 >> v2:172.21.15.27:6801/102631 conn(0x7fdb6c03a730 0x7fdb6c03cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T14:43:44.187 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.153+0000 7fdb857fa700 1 -- 172.21.15.27:0/2097630216 --> v2:172.21.15.27:6801/102631 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdb6c03d250 con 0x7fdb6c03a730 2023-01-24T14:43:44.187 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.153+0000 7fdb857fa700 1 -- 172.21.15.27:0/2097630216 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_get_version_reply(handle=1 version=14) v2 ==== 24+0+0 (crc 0 0 0) 0x7fdb7c011d30 con 0x7fdb88098c00 2023-01-24T14:43:44.187 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.153+0000 7fe3a11f1700 1 -- 172.21.15.27:0/43772404 wait complete. 2023-01-24T14:43:44.188 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.153+0000 7fdb87fff700 1 --2- 172.21.15.27:0/2097630216 >> [v2:172.21.15.35:6808/101897,v1:172.21.15.35:6809/101897] conn(0x7fdb6c035790 0x7fdb6c037ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T14:43:44.188 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.154+0000 7fdb8d353700 1 --2- 172.21.15.27:0/2097630216 >> v2:172.21.15.27:6801/102631 conn(0x7fdb6c03a730 0x7fdb6c03cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-01-24T14:43:44.188 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.154+0000 7fdb87fff700 1 --2- 172.21.15.27:0/2097630216 >> [v2:172.21.15.35:6808/101897,v1:172.21.15.35:6809/101897] conn(0x7fdb6c035790 0x7fdb6c037ca0 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T14:43:44.189 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.152+0000 7fc97de4f700 1 -- 172.21.15.27:0/667354538 wait complete. 2023-01-24T14:43:44.189 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.154+0000 7fc97de4f700 1 Processor -- start 2023-01-24T14:43:44.189 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.155+0000 7fc97de4f700 1 -- start start 2023-01-24T14:43:44.190 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.155+0000 7fef52ffd700 1 -- 172.21.15.27:0/924089748 <== osd.0 v2:172.21.15.27:6803/102630 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fef5c121ba0 con 0x7fef4003a720 2023-01-24T14:43:44.191 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.155+0000 7fef6466e700 1 -- 172.21.15.27:0/924089748 >> v2:172.21.15.27:6803/102630 conn(0x7fef4003a720 msgr2=0x7fef4003cbb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T14:43:44.191 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.155+0000 7fef6466e700 1 --2- 172.21.15.27:0/924089748 >> v2:172.21.15.27:6803/102630 conn(0x7fef4003a720 0x7fef4003cbb0 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T14:43:44.191 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.155+0000 7fef6466e700 1 -- 172.21.15.27:0/924089748 >> [v2:172.21.15.35:6808/101897,v1:172.21.15.35:6809/101897] conn(0x7fef40035780 msgr2=0x7fef40037c90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T14:43:44.191 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.155+0000 7fef6466e700 1 --2- 172.21.15.27:0/924089748 >> [v2:172.21.15.35:6808/101897,v1:172.21.15.35:6809/101897] conn(0x7fef40035780 0x7fef40037c90 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T14:43:44.191 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.155+0000 7fef6466e700 1 -- 172.21.15.27:0/924089748 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fef5c097a20 msgr2=0x7fef5c11edd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T14:43:44.192 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.155+0000 7fef6466e700 1 --2- 172.21.15.27:0/924089748 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fef5c097a20 0x7fef5c11edd0 crc :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T14:43:44.192 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.156+0000 7fc97de4f700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc97806a280 0x7fc9781314f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T14:43:44.192 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.156+0000 7fc97de4f700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc978133070 con 0x7fc97806a280 2023-01-24T14:43:44.193 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.156+0000 7fef6466e700 1 -- 172.21.15.27:0/924089748 shutdown_connections 2023-01-24T14:43:44.193 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.156+0000 7fef6466e700 1 --2- 172.21.15.27:0/924089748 >> v2:172.21.15.27:6803/102630 conn(0x7fef4003a720 0x7fef4003cbb0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T14:43:44.193 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.156+0000 7fef6466e700 1 --2- 172.21.15.27:0/924089748 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fef5c097a20 0x7fef5c11edd0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T14:43:44.194 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.156+0000 7fef6466e700 1 --2- 172.21.15.27:0/924089748 >> [v2:172.21.15.35:6808/101897,v1:172.21.15.35:6809/101897] conn(0x7fef40035780 0x7fef40037c90 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T14:43:44.194 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.156+0000 7fef6466e700 1 -- 172.21.15.27:0/924089748 >> 172.21.15.27:0/924089748 conn(0x7fef5c0a13d0 msgr2=0x7fef5c09b0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T14:43:44.194 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.156+0000 7fef6466e700 1 -- 172.21.15.27:0/924089748 shutdown_connections 2023-01-24T14:43:44.194 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.156+0000 7fc9777fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc97806a280 0x7fc9781314f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T14:43:44.195 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.156+0000 7fc9777fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc97806a280 0x7fc9781314f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48966/0 (socket says 172.21.15.27:48966) 2023-01-24T14:43:44.195 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.156+0000 7fc9777fe700 1 -- 172.21.15.27:0/888062783 learned_addr learned my addr 172.21.15.27:0/888062783 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-01-24T14:43:44.195 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.156+0000 7fef6466e700 1 -- 172.21.15.27:0/924089748 wait complete. 2023-01-24T14:43:44.196 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.156+0000 7fc9777fe700 1 -- 172.21.15.27:0/888062783 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc978095d50 con 0x7fc97806a280 2023-01-24T14:43:44.196 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.156+0000 7fc9777fe700 1 --2- 172.21.15.27:0/888062783 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc97806a280 0x7fc9781314f0 crc :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T14:43:44.197 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.157+0000 7fc95ffff700 1 -- 172.21.15.27:0/888062783 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fc9680051e0 con 0x7fc97806a280 2023-01-24T14:43:44.197 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.157+0000 7fc95ffff700 1 -- 172.21.15.27:0/888062783 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7fc968005340 con 0x7fc97806a280 2023-01-24T14:43:44.197 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.157+0000 7fc95ffff700 1 -- 172.21.15.27:0/888062783 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fc96800c7a0 con 0x7fc97806a280 2023-01-24T14:43:44.197 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.157+0000 7fc97de4f700 1 -- 172.21.15.27:0/888062783 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc978070a60 con 0x7fc97806a280 2023-01-24T14:43:44.197 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.157+0000 7fc97de4f700 1 -- 172.21.15.27:0/888062783 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc978071a90 con 0x7fc97806a280 2023-01-24T14:43:44.198 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.158+0000 7fc97de4f700 1 -- 172.21.15.27:0/888062783 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fc978134ac0 con 0x7fc97806a280 2023-01-24T14:43:44.198 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.158+0000 7fdb8d353700 1 --2- 172.21.15.27:0/2097630216 >> v2:172.21.15.27:6801/102631 conn(0x7fdb6c03a730 0x7fdb6c03cbc0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T14:43:44.198 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.159+0000 7fdb857fa700 1 -- 172.21.15.27:0/2097630216 <== osd.1 v2:172.21.15.27:6801/102631 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+4964 (crc 0 0 0) 0x7fdb6c03d250 con 0x7fdb6c03a730 2023-01-24T14:43:44.200 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.159+0000 7fc95ffff700 1 -- 172.21.15.27:0/888062783 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 4) v1 ==== 42729+0+0 (crc 0 0 0) 0x7fc96800c950 con 0x7fc97806a280 2023-01-24T14:43:44.200 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.160+0000 7fc95ffff700 1 --2- 172.21.15.27:0/888062783 >> [v2:172.21.15.35:6808/101897,v1:172.21.15.35:6809/101897] conn(0x7fc964035790 0x7fc964037ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T14:43:44.200 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.160+0000 7fc95ffff700 1 -- 172.21.15.27:0/888062783 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 2650+0+0 (crc 0 0 0) 0x7fc968044410 con 0x7fc97806a280 2023-01-24T14:43:44.200 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.160+0000 7fc95ffff700 1 --2- 172.21.15.27:0/888062783 >> v2:172.21.15.35:6800/102222 conn(0x7fc96403a730 0x7fc96403cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T14:43:44.200 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.160+0000 7fc95ffff700 1 -- 172.21.15.27:0/888062783 --> v2:172.21.15.35:6800/102222 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc96403d250 con 0x7fc96403a730 2023-01-24T14:43:44.201 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.160+0000 7fc95ffff700 1 -- 172.21.15.27:0/888062783 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_get_version_reply(handle=1 version=14) v2 ==== 24+0+0 (crc 0 0 0) 0x7fc968044790 con 0x7fc97806a280 2023-01-24T14:43:44.201 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.160+0000 7fc976ffd700 1 --2- 172.21.15.27:0/888062783 >> [v2:172.21.15.35:6808/101897,v1:172.21.15.35:6809/101897] conn(0x7fc964035790 0x7fc964037ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T14:43:44.201 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.160+0000 7fc976ffd700 1 --2- 172.21.15.27:0/888062783 >> [v2:172.21.15.35:6808/101897,v1:172.21.15.35:6809/101897] conn(0x7fc964035790 0x7fc964037ca0 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T14:43:44.201 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.160+0000 7fc977fff700 1 --2- 172.21.15.27:0/888062783 >> v2:172.21.15.35:6800/102222 conn(0x7fc96403a730 0x7fc96403cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-01-24T14:43:44.201 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.162+0000 7fc977fff700 1 --2- 172.21.15.27:0/888062783 >> v2:172.21.15.35:6800/102222 conn(0x7fc96403a730 0x7fc96403cbc0 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T14:43:44.202 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.163+0000 7fc95ffff700 1 -- 172.21.15.27:0/888062783 <== osd.3 v2:172.21.15.35:6800/102222 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+4964 (crc 0 0 0) 0x7fc96403d250 con 0x7fc96403a730 2023-01-24T14:43:44.202 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.166+0000 7fdb8edb6700 1 -- 172.21.15.27:0/2097630216 --> v2:172.21.15.27:6801/102631 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fdb88074980 con 0x7fdb6c03a730 2023-01-24T14:43:44.202 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.168+0000 7fdb857fa700 1 -- 172.21.15.27:0/2097630216 <== osd.1 v2:172.21.15.27:6801/102631 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fdb88074980 con 0x7fdb6c03a730 2023-01-24T14:43:44.203 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.168+0000 7fc97de4f700 1 -- 172.21.15.27:0/888062783 --> v2:172.21.15.35:6800/102222 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fc978131130 con 0x7fc96403a730 2023-01-24T14:43:44.204 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.168+0000 7fdb8edb6700 1 -- 172.21.15.27:0/2097630216 >> v2:172.21.15.27:6801/102631 conn(0x7fdb6c03a730 msgr2=0x7fdb6c03cbc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T14:43:44.204 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.168+0000 7fdb8edb6700 1 --2- 172.21.15.27:0/2097630216 >> v2:172.21.15.27:6801/102631 conn(0x7fdb6c03a730 0x7fdb6c03cbc0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T14:43:44.204 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.168+0000 7fdb8edb6700 1 -- 172.21.15.27:0/2097630216 >> [v2:172.21.15.35:6808/101897,v1:172.21.15.35:6809/101897] conn(0x7fdb6c035790 msgr2=0x7fdb6c037ca0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T14:43:44.205 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.168+0000 7fdb8edb6700 1 --2- 172.21.15.27:0/2097630216 >> [v2:172.21.15.35:6808/101897,v1:172.21.15.35:6809/101897] conn(0x7fdb6c035790 0x7fdb6c037ca0 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T14:43:44.205 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.168+0000 7fdb8edb6700 1 -- 172.21.15.27:0/2097630216 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb88098c00 msgr2=0x7fdb8809f3a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T14:43:44.205 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.168+0000 7fdb8edb6700 1 --2- 172.21.15.27:0/2097630216 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb88098c00 0x7fdb8809f3a0 crc :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T14:43:44.205 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.168+0000 7fdb8edb6700 1 -- 172.21.15.27:0/2097630216 shutdown_connections 2023-01-24T14:43:44.205 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.168+0000 7fdb8edb6700 1 --2- 172.21.15.27:0/2097630216 >> v2:172.21.15.27:6801/102631 conn(0x7fdb6c03a730 0x7fdb6c03cbc0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T14:43:44.206 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.168+0000 7fdb8edb6700 1 --2- 172.21.15.27:0/2097630216 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb88098c00 0x7fdb8809f3a0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T14:43:44.206 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.168+0000 7fdb8edb6700 1 --2- 172.21.15.27:0/2097630216 >> [v2:172.21.15.35:6808/101897,v1:172.21.15.35:6809/101897] conn(0x7fdb6c035790 0x7fdb6c037ca0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T14:43:44.206 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.168+0000 7fdb8edb6700 1 -- 172.21.15.27:0/2097630216 >> 172.21.15.27:0/2097630216 conn(0x7fdb8809da80 msgr2=0x7fdb8812c4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T14:43:44.206 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.168+0000 7fdb8edb6700 1 -- 172.21.15.27:0/2097630216 shutdown_connections 2023-01-24T14:43:44.206 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.169+0000 7fdb8edb6700 1 -- 172.21.15.27:0/2097630216 wait complete. 2023-01-24T14:43:44.207 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.169+0000 7fc95ffff700 1 -- 172.21.15.27:0/888062783 <== osd.3 v2:172.21.15.35:6800/102222 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fc978131130 con 0x7fc96403a730 2023-01-24T14:43:44.207 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.169+0000 7fc97de4f700 1 -- 172.21.15.27:0/888062783 >> v2:172.21.15.35:6800/102222 conn(0x7fc96403a730 msgr2=0x7fc96403cbc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T14:43:44.208 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.169+0000 7fc97de4f700 1 --2- 172.21.15.27:0/888062783 >> v2:172.21.15.35:6800/102222 conn(0x7fc96403a730 0x7fc96403cbc0 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T14:43:44.208 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.169+0000 7fc97de4f700 1 -- 172.21.15.27:0/888062783 >> [v2:172.21.15.35:6808/101897,v1:172.21.15.35:6809/101897] conn(0x7fc964035790 msgr2=0x7fc964037ca0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T14:43:44.208 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.169+0000 7fc97de4f700 1 --2- 172.21.15.27:0/888062783 >> [v2:172.21.15.35:6808/101897,v1:172.21.15.35:6809/101897] conn(0x7fc964035790 0x7fc964037ca0 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T14:43:44.208 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.169+0000 7fc97de4f700 1 -- 172.21.15.27:0/888062783 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc97806a280 msgr2=0x7fc9781314f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T14:43:44.208 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.169+0000 7fc97de4f700 1 --2- 172.21.15.27:0/888062783 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc97806a280 0x7fc9781314f0 crc :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T14:43:44.209 INFO:teuthology.orchestra.run.smithi027.stdout:38654705670 2023-01-24T14:43:44.209 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.169+0000 7fc97de4f700 1 -- 172.21.15.27:0/888062783 shutdown_connections 2023-01-24T14:43:44.210 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.169+0000 7fc97de4f700 1 --2- 172.21.15.27:0/888062783 >> v2:172.21.15.35:6800/102222 conn(0x7fc96403a730 0x7fc96403cbc0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T14:43:44.210 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.169+0000 7fc97de4f700 1 --2- 172.21.15.27:0/888062783 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc97806a280 0x7fc9781314f0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T14:43:44.210 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.169+0000 7fc97de4f700 1 --2- 172.21.15.27:0/888062783 >> [v2:172.21.15.35:6808/101897,v1:172.21.15.35:6809/101897] conn(0x7fc964035790 0x7fc964037ca0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T14:43:44.210 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.169+0000 7fc97de4f700 1 -- 172.21.15.27:0/888062783 >> 172.21.15.27:0/888062783 conn(0x7fc97809f630 msgr2=0x7fc978126780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T14:43:44.211 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-01-24T14:43:44.212 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.169+0000 7fc97de4f700 1 -- 172.21.15.27:0/888062783 shutdown_connections 2023-01-24T14:43:44.212 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.170+0000 7fc97de4f700 1 -- 172.21.15.27:0/888062783 wait complete. 2023-01-24T14:43:44.213 INFO:teuthology.orchestra.run.smithi027.stdout:38654705670 2023-01-24T14:43:44.213 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-01-24T14:43:44.214 INFO:teuthology.orchestra.run.smithi027.stdout:34359738375 2023-01-24T14:43:44.214 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-01-24T14:43:44.215 INFO:teuthology.orchestra.run.smithi027.stdout:38654705670 2023-01-24T14:43:44.216 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-01-24T14:43:44.400 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.398+0000 7fc9ed8db700 1 Processor -- start 2023-01-24T14:43:44.400 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.398+0000 7fc9ed8db700 1 -- start start 2023-01-24T14:43:44.401 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.398+0000 7fe01e182700 1 Processor -- start 2023-01-24T14:43:44.401 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.399+0000 7fe01e182700 1 -- start start 2023-01-24T14:43:44.401 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.399+0000 7fc9ed8db700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc9e81330e0 0x7fc9e8135500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T14:43:44.402 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.399+0000 7fc9ed8db700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9e809b7e0 con 0x7fc9e81300e0 2023-01-24T14:43:44.402 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.400+0000 7fc9ed8db700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9e806a6c0 con 0x7fc9e81330e0 2023-01-24T14:43:44.402 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.400+0000 7fe01e182700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe018097a20 0x7fe018097e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T14:43:44.403 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.400+0000 7fe01e182700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe01809af40 con 0x7fe018098370 2023-01-24T14:43:44.403 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.400+0000 7fe01e182700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe01809a380 con 0x7fe018097a20 2023-01-24T14:43:44.403 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.400+0000 7fc9e67fc700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc9e81330e0 0x7fc9e8135500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T14:43:44.404 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.400+0000 7fc9e67fc700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc9e81330e0 0x7fc9e8135500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48972/0 (socket says 172.21.15.27:48972) 2023-01-24T14:43:44.404 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.400+0000 7fc9e67fc700 1 -- 172.21.15.27:0/3354166255 learned_addr learned my addr 172.21.15.27:0/3354166255 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-01-24T14:43:44.405 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.400+0000 7fe0177fe700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe018097a20 0x7fe018097e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T14:43:44.405 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.400+0000 7fe0177fe700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe018097a20 0x7fe018097e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48988/0 (socket says 172.21.15.27:48988) 2023-01-24T14:43:44.405 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.400+0000 7fe0177fe700 1 -- 172.21.15.27:0/1324814337 learned_addr learned my addr 172.21.15.27:0/1324814337 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-01-24T14:43:44.405 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.400+0000 7fe016ffd700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fe018098370 0x7fe01812af60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:45010/0 (socket says 172.21.15.27:45010) 2023-01-24T14:43:44.405 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.400+0000 7fe0167fc700 1 -- 172.21.15.27:0/1324814337 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3107213283 0 0) 0x7fe01809af40 con 0x7fe018098370 2023-01-24T14:43:44.406 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.400+0000 7fe0167fc700 1 -- 172.21.15.27:0/1324814337 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe000003580 con 0x7fe018098370 2023-01-24T14:43:44.406 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.400+0000 7fe0177fe700 1 -- 172.21.15.27:0/1324814337 >> v1:172.21.15.27:6789/0 conn(0x7fe018098370 legacy=0x7fe01812af60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T14:43:44.406 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.400+0000 7fc9e67fc700 1 -- 172.21.15.27:0/3354166255 >> v1:172.21.15.27:6789/0 conn(0x7fc9e81300e0 legacy=0x7fc9e81304f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T14:43:44.407 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.400+0000 7fe0177fe700 1 -- 172.21.15.27:0/1324814337 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe018071f20 con 0x7fe018097a20 2023-01-24T14:43:44.407 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.400+0000 7fc9e67fc700 1 -- 172.21.15.27:0/3354166255 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9e806cf60 con 0x7fc9e81330e0 2023-01-24T14:43:44.407 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.400+0000 7fc9e67fc700 1 --2- 172.21.15.27:0/3354166255 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc9e81330e0 0x7fc9e8135500 crc :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8acfcaac74c9354b server_cookie=0 in_seq=0 out_seq=0 2023-01-24T14:43:44.408 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.400+0000 7fe0177fe700 1 --2- 172.21.15.27:0/1324814337 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe018097a20 0x7fe018097e30 crc :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ef69201bffeb2ff server_cookie=0 in_seq=0 out_seq=0 2023-01-24T14:43:44.408 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.401+0000 7fe0167fc700 1 -- 172.21.15.27:0/1324814337 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fe00c009d40 con 0x7fe018097a20 2023-01-24T14:43:44.409 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.401+0000 7fc9e5ffb700 1 -- 172.21.15.27:0/3354166255 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fc9d8009d00 con 0x7fc9e81330e0 2023-01-24T14:43:44.409 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.401+0000 7fc9e5ffb700 1 -- 172.21.15.27:0/3354166255 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7fc9d8009e60 con 0x7fc9e81330e0 2023-01-24T14:43:44.409 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.401+0000 7fc9e5ffb700 1 -- 172.21.15.27:0/3354166255 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fc9d800e360 con 0x7fc9e81330e0 2023-01-24T14:43:44.410 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.401+0000 7fe0167fc700 1 -- 172.21.15.27:0/1324814337 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7fe00c00e230 con 0x7fe018097a20 2023-01-24T14:43:44.410 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.401+0000 7fe0167fc700 1 -- 172.21.15.27:0/1324814337 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fe00c009ea0 con 0x7fe018097a20 2023-01-24T14:43:44.411 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.402+0000 7fc9ed8db700 1 -- 172.21.15.27:0/3354166255 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc9e81330e0 msgr2=0x7fc9e8135500 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T14:43:44.412 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.402+0000 7fe01e182700 1 -- 172.21.15.27:0/1324814337 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe018097a20 msgr2=0x7fe018097e30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T14:43:44.412 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.402+0000 7fc9ed8db700 1 --2- 172.21.15.27:0/3354166255 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc9e81330e0 0x7fc9e8135500 crc :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T14:43:44.413 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.402+0000 7fe01e182700 1 --2- 172.21.15.27:0/1324814337 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe018097a20 0x7fe018097e30 crc :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T14:43:44.413 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.402+0000 7fe01e182700 1 -- 172.21.15.27:0/1324814337 shutdown_connections 2023-01-24T14:43:44.413 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.402+0000 7fe01e182700 1 --2- 172.21.15.27:0/1324814337 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe018097a20 0x7fe018097e30 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T14:43:44.414 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.402+0000 7fe01e182700 1 -- 172.21.15.27:0/1324814337 >> 172.21.15.27:0/1324814337 conn(0x7fe0180a13d0 msgr2=0x7fe01809d2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T14:43:44.414 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.402+0000 7fe01e182700 1 -- 172.21.15.27:0/1324814337 shutdown_connections 2023-01-24T14:43:44.414 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.402+0000 7fe01e182700 1 -- 172.21.15.27:0/1324814337 wait complete. 2023-01-24T14:43:44.415 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.402+0000 7fc9ed8db700 1 -- 172.21.15.27:0/3354166255 shutdown_connections 2023-01-24T14:43:44.415 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.402+0000 7fc9ed8db700 1 --2- 172.21.15.27:0/3354166255 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc9e81330e0 0x7fc9e8135500 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T14:43:44.415 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.402+0000 7fc9ed8db700 1 -- 172.21.15.27:0/3354166255 >> 172.21.15.27:0/3354166255 conn(0x7fc9e80a13c0 msgr2=0x7fc9e80a17d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T14:43:44.416 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.402+0000 7fc9ed8db700 1 -- 172.21.15.27:0/3354166255 shutdown_connections 2023-01-24T14:43:44.416 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.403+0000 7fc9ed8db700 1 -- 172.21.15.27:0/3354166255 wait complete. 2023-01-24T14:43:44.416 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.404+0000 7fe01e182700 1 Processor -- start 2023-01-24T14:43:44.417 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.404+0000 7fe01e182700 1 -- start start 2023-01-24T14:43:44.418 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.404+0000 7fc9ed8db700 1 Processor -- start 2023-01-24T14:43:44.418 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.404+0000 7fc9ed8db700 1 -- start start 2023-01-24T14:43:44.418 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.405+0000 7fc9ed8db700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc9e81300e0 0x7fc9e811dc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T14:43:44.418 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.405+0000 7fc9ed8db700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9e811d940 con 0x7fc9e81300e0 2023-01-24T14:43:44.419 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.405+0000 7fc9e6ffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc9e81300e0 0x7fc9e811dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T14:43:44.419 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.405+0000 7fc9e6ffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc9e81300e0 0x7fc9e811dc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48990/0 (socket says 172.21.15.27:48990) 2023-01-24T14:43:44.419 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.405+0000 7fc9e6ffd700 1 -- 172.21.15.27:0/1155704872 learned_addr learned my addr 172.21.15.27:0/1155704872 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-01-24T14:43:44.420 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.405+0000 7fe01e182700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe018098370 0x7fe01812b330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T14:43:44.420 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.405+0000 7fe01e182700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe01809cb10 con 0x7fe018098370 2023-01-24T14:43:44.420 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.405+0000 7fe0177fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe018098370 0x7fe01812b330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T14:43:44.420 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.405+0000 7fe0177fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe018098370 0x7fe01812b330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48998/0 (socket says 172.21.15.27:48998) 2023-01-24T14:43:44.421 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.405+0000 7fe0177fe700 1 -- 172.21.15.27:0/3534383975 learned_addr learned my addr 172.21.15.27:0/3534383975 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-01-24T14:43:44.421 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.405+0000 7fc9e6ffd700 1 -- 172.21.15.27:0/1155704872 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9e806a8e0 con 0x7fc9e81300e0 2023-01-24T14:43:44.421 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.405+0000 7fc9e6ffd700 1 --2- 172.21.15.27:0/1155704872 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc9e81300e0 0x7fc9e811dc60 crc :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T14:43:44.422 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.406+0000 7fe0177fe700 1 -- 172.21.15.27:0/3534383975 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe01809aaa0 con 0x7fe018098370 2023-01-24T14:43:44.423 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.406+0000 7fc9dffff700 1 -- 172.21.15.27:0/1155704872 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fc9e000e040 con 0x7fc9e81300e0 2023-01-24T14:43:44.423 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.406+0000 7fe0177fe700 1 --2- 172.21.15.27:0/3534383975 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe018098370 0x7fe01812b330 crc :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T14:43:44.424 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.406+0000 7fc9dffff700 1 -- 172.21.15.27:0/1155704872 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7fc9e000be30 con 0x7fc9e81300e0 2023-01-24T14:43:44.424 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.406+0000 7fc9ed8db700 1 -- 172.21.15.27:0/1155704872 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9e806c1f0 con 0x7fc9e81300e0 2023-01-24T14:43:44.424 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.406+0000 7fc9dffff700 1 -- 172.21.15.27:0/1155704872 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fc9e0007480 con 0x7fc9e81300e0 2023-01-24T14:43:44.424 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.406+0000 7fc9ed8db700 1 -- 172.21.15.27:0/1155704872 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9e806e940 con 0x7fc9e81300e0 2023-01-24T14:43:44.425 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.406+0000 7fdffffff700 1 -- 172.21.15.27:0/3534383975 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fe00c00ee80 con 0x7fe018098370 2023-01-24T14:43:44.425 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.406+0000 7fe01e182700 1 -- 172.21.15.27:0/3534383975 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe01809a590 con 0x7fe018098370 2023-01-24T14:43:44.426 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.406+0000 7fe01e182700 1 -- 172.21.15.27:0/3534383975 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe01809e040 con 0x7fe018098370 2023-01-24T14:43:44.426 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.407+0000 7fdffffff700 1 -- 172.21.15.27:0/3534383975 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7fe00c014410 con 0x7fe018098370 2023-01-24T14:43:44.426 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.407+0000 7fdffffff700 1 -- 172.21.15.27:0/3534383975 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fe00c014580 con 0x7fe018098370 2023-01-24T14:43:44.426 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.407+0000 7fc9dffff700 1 -- 172.21.15.27:0/1155704872 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 42769+0+0 (crc 0 0 0) 0x7fc9e00039d0 con 0x7fc9e81300e0 2023-01-24T14:43:44.427 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.407+0000 7fc9ed8db700 1 -- 172.21.15.27:0/1155704872 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9e8074980 con 0x7fc9e81300e0 2023-01-24T14:43:44.427 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.407+0000 7fc9dffff700 1 --2- 172.21.15.27:0/1155704872 >> [v2:172.21.15.35:6808/101897,v1:172.21.15.35:6809/101897] conn(0x7fc9c80357e0 0x7fc9c8037cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T14:43:44.427 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.407+0000 7fc9dffff700 1 -- 172.21.15.27:0/1155704872 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 2650+0+0 (crc 0 0 0) 0x7fc9e0044cd0 con 0x7fc9e81300e0 2023-01-24T14:43:44.427 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.408+0000 7fc9e67fc700 1 --2- 172.21.15.27:0/1155704872 >> [v2:172.21.15.35:6808/101897,v1:172.21.15.35:6809/101897] conn(0x7fc9c80357e0 0x7fc9c8037cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T14:43:44.428 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.408+0000 7fdffffff700 1 -- 172.21.15.27:0/3534383975 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 42769+0+0 (crc 0 0 0) 0x7fe00c014780 con 0x7fe018098370 2023-01-24T14:43:44.428 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.408+0000 7fe01e182700 1 -- 172.21.15.27:0/3534383975 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdff80050a0 con 0x7fe018098370 2023-01-24T14:43:44.428 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.408+0000 7fdffffff700 1 --2- 172.21.15.27:0/3534383975 >> [v2:172.21.15.35:6808/101897,v1:172.21.15.35:6809/101897] conn(0x7fe004035790 0x7fe004037ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T14:43:44.429 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.408+0000 7fdffffff700 1 -- 172.21.15.27:0/3534383975 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 2650+0+0 (crc 0 0 0) 0x7fe00c045740 con 0x7fe018098370 2023-01-24T14:43:44.429 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.408+0000 7fc9e67fc700 1 --2- 172.21.15.27:0/1155704872 >> [v2:172.21.15.35:6808/101897,v1:172.21.15.35:6809/101897] conn(0x7fc9c80357e0 0x7fc9c8037cf0 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T14:43:44.430 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.408+0000 7fe016ffd700 1 --2- 172.21.15.27:0/3534383975 >> [v2:172.21.15.35:6808/101897,v1:172.21.15.35:6809/101897] conn(0x7fe004035790 0x7fe004037ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T14:43:44.430 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.409+0000 7fe016ffd700 1 --2- 172.21.15.27:0/3534383975 >> [v2:172.21.15.35:6808/101897,v1:172.21.15.35:6809/101897] conn(0x7fe004035790 0x7fe004037ca0 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T14:43:44.431 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.410+0000 7ff7e9082700 1 Processor -- start 2023-01-24T14:43:44.431 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.410+0000 7ff7e9082700 1 -- start start 2023-01-24T14:43:44.432 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.411+0000 7ff7e9082700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ff7e412ee90 0x7ff7e4133670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T14:43:44.432 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.411+0000 7ff7e9082700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7e412f2a0 con 0x7ff7e4098c00 2023-01-24T14:43:44.432 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.411+0000 7ff7e9082700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7e409cf30 con 0x7ff7e412ee90 2023-01-24T14:43:44.432 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.411+0000 7ff7e259c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ff7e412ee90 0x7ff7e4133670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T14:43:44.432 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.411+0000 7ff7e2d9d700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7ff7e4098c00 0x7ff7e412e780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:45020/0 (socket says 172.21.15.27:45020) 2023-01-24T14:43:44.433 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.411+0000 7ff7e259c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ff7e412ee90 0x7ff7e4133670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:49012/0 (socket says 172.21.15.27:49012) 2023-01-24T14:43:44.433 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.411+0000 7ff7e2d9d700 1 -- 172.21.15.27:0/3894787769 learned_addr learned my addr 172.21.15.27:0/3894787769 (peer_addr_for_me v1:172.21.15.27:0/0) 2023-01-24T14:43:44.433 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.412+0000 7ff7e259c700 1 -- 172.21.15.27:0/3894787769 >> v1:172.21.15.27:6789/0 conn(0x7ff7e4098c00 legacy=0x7ff7e412e780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T14:43:44.433 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.412+0000 7ff7e259c700 1 -- 172.21.15.27:0/3894787769 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7e409cc20 con 0x7ff7e412ee90 2023-01-24T14:43:44.434 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.412+0000 7ff7e259c700 1 --2- 172.21.15.27:0/3894787769 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff7e412ee90 0x7ff7e4133670 crc :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee36d186e9146c82 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T14:43:44.434 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.413+0000 7f406a5f5700 1 Processor -- start 2023-01-24T14:43:44.435 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.413+0000 7f406a5f5700 1 -- start start 2023-01-24T14:43:44.435 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.413+0000 7fc9dffff700 1 -- 172.21.15.27:0/1155704872 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172632 (crc 0 0 0) 0x7fc9e000ee00 con 0x7fc9e81300e0 2023-01-24T14:43:44.436 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.413+0000 7f406a5f5700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f4064098540 0x7f4064133750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T14:43:44.437 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.413+0000 7f406a5f5700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f406409af40 con 0x7f4064097a20 2023-01-24T14:43:44.437 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.413+0000 7f406a5f5700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f406409a380 con 0x7f4064098540 2023-01-24T14:43:44.437 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.414+0000 7f405bfff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f4064098540 0x7f4064133750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T14:43:44.437 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.414+0000 7f4063fff700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f4064097a20 0x7f4064097e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:45034/0 (socket says 172.21.15.27:45034) 2023-01-24T14:43:44.438 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.414+0000 7f4063fff700 1 -- 172.21.15.27:0/2166304319 learned_addr learned my addr 172.21.15.27:0/2166304319 (peer_addr_for_me v1:172.21.15.27:0/0) 2023-01-24T14:43:44.438 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.414+0000 7f405bfff700 1 -- 172.21.15.27:0/2166304319 >> v1:172.21.15.27:6789/0 conn(0x7f4064097a20 legacy=0x7f4064097e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T14:43:44.438 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.414+0000 7f405bfff700 1 -- 172.21.15.27:0/2166304319 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4064071f20 con 0x7f4064098540 2023-01-24T14:43:44.438 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.414+0000 7f405bfff700 1 --2- 172.21.15.27:0/2166304319 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4064098540 0x7f4064133750 crc :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=313c8c29373a34a7 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T14:43:44.438 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.418+0000 7f40637fe700 1 -- 172.21.15.27:0/2166304319 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f404c009cd0 con 0x7f4064098540 2023-01-24T14:43:44.439 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.418+0000 7fdffffff700 1 -- 172.21.15.27:0/3534383975 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172632 (crc 0 0 0) 0x7fe00c00f8f0 con 0x7fe018098370 2023-01-24T14:43:44.440 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.419+0000 7f40637fe700 1 -- 172.21.15.27:0/2166304319 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f404c009e30 con 0x7f4064098540 2023-01-24T14:43:44.440 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.419+0000 7f40637fe700 1 -- 172.21.15.27:0/2166304319 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f404c00e360 con 0x7f4064098540 2023-01-24T14:43:44.440 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.420+0000 7f406a5f5700 1 -- 172.21.15.27:0/2166304319 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4064098540 msgr2=0x7f4064133750 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T14:43:44.440 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.420+0000 7f406a5f5700 1 --2- 172.21.15.27:0/2166304319 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4064098540 0x7f4064133750 crc :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T14:43:44.440 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.420+0000 7f406a5f5700 1 -- 172.21.15.27:0/2166304319 shutdown_connections 2023-01-24T14:43:44.441 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.420+0000 7f406a5f5700 1 --2- 172.21.15.27:0/2166304319 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4064098540 0x7f4064133750 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T14:43:44.441 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.420+0000 7f406a5f5700 1 -- 172.21.15.27:0/2166304319 >> 172.21.15.27:0/2166304319 conn(0x7f40640a13d0 msgr2=0x7f406409d2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T14:43:44.442 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.418+0000 7ff7e1d9b700 1 -- 172.21.15.27:0/3894787769 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7ff7d8009cd0 con 0x7ff7e412ee90 2023-01-24T14:43:44.442 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.418+0000 7ff7e1d9b700 1 -- 172.21.15.27:0/3894787769 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7ff7d8009e30 con 0x7ff7e412ee90 2023-01-24T14:43:44.442 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.418+0000 7ff7e1d9b700 1 -- 172.21.15.27:0/3894787769 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7ff7d800f470 con 0x7ff7e412ee90 2023-01-24T14:43:44.443 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.419+0000 7ff7e9082700 1 -- 172.21.15.27:0/3894787769 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff7e412ee90 msgr2=0x7ff7e4133670 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T14:43:44.443 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.419+0000 7ff7e9082700 1 --2- 172.21.15.27:0/3894787769 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff7e412ee90 0x7ff7e4133670 crc :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T14:43:44.443 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.419+0000 7ff7e9082700 1 -- 172.21.15.27:0/3894787769 shutdown_connections 2023-01-24T14:43:44.443 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.419+0000 7ff7e9082700 1 --2- 172.21.15.27:0/3894787769 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff7e412ee90 0x7ff7e4133670 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T14:43:44.443 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.419+0000 7ff7e9082700 1 -- 172.21.15.27:0/3894787769 >> 172.21.15.27:0/3894787769 conn(0x7ff7e409da80 msgr2=0x7ff7e409dec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T14:43:44.444 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.419+0000 7ff7e9082700 1 -- 172.21.15.27:0/3894787769 shutdown_connections 2023-01-24T14:43:44.444 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.419+0000 7ff7e9082700 1 -- 172.21.15.27:0/3894787769 wait complete. 2023-01-24T14:43:44.445 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.421+0000 7f406a5f5700 1 -- 172.21.15.27:0/2166304319 shutdown_connections 2023-01-24T14:43:44.445 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.421+0000 7f406a5f5700 1 -- 172.21.15.27:0/2166304319 wait complete. 2023-01-24T14:43:44.446 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.421+0000 7ff7e9082700 1 Processor -- start 2023-01-24T14:43:44.446 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.421+0000 7ff7e9082700 1 -- start start 2023-01-24T14:43:44.446 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.422+0000 7ff7e9082700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff7e4098c00 0x7ff7e411b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T14:43:44.446 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.422+0000 7ff7e9082700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7e412cfb0 con 0x7ff7e4098c00 2023-01-24T14:43:44.447 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.422+0000 7f406a5f5700 1 Processor -- start 2023-01-24T14:43:44.448 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.422+0000 7ff7e2d9d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff7e4098c00 0x7ff7e411b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T14:43:44.448 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.422+0000 7ff7e2d9d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff7e4098c00 0x7ff7e411b310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:49038/0 (socket says 172.21.15.27:49038) 2023-01-24T14:43:44.448 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.422+0000 7ff7e2d9d700 1 -- 172.21.15.27:0/1718486145 learned_addr learned my addr 172.21.15.27:0/1718486145 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-01-24T14:43:44.448 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.422+0000 7ff7e2d9d700 1 -- 172.21.15.27:0/1718486145 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7e409a4f0 con 0x7ff7e4098c00 2023-01-24T14:43:44.448 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.422+0000 7ff7e2d9d700 1 --2- 172.21.15.27:0/1718486145 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff7e4098c00 0x7ff7e411b310 crc :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T14:43:44.449 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.422+0000 7ff7cb7fe700 1 -- 172.21.15.27:0/1718486145 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7ff7d4005a60 con 0x7ff7e4098c00 2023-01-24T14:43:44.449 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.422+0000 7ff7cb7fe700 1 -- 172.21.15.27:0/1718486145 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7ff7d4005bc0 con 0x7ff7e4098c00 2023-01-24T14:43:44.449 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.422+0000 7ff7e9082700 1 -- 172.21.15.27:0/1718486145 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7e4099690 con 0x7ff7e4098c00 2023-01-24T14:43:44.449 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.422+0000 7ff7cb7fe700 1 -- 172.21.15.27:0/1718486145 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7ff7d4005e60 con 0x7ff7e4098c00 2023-01-24T14:43:44.449 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.423+0000 7ff7e9082700 1 -- 172.21.15.27:0/1718486145 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7e409a680 con 0x7ff7e4098c00 2023-01-24T14:43:44.450 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.422+0000 7f406a5f5700 1 -- start start 2023-01-24T14:43:44.450 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.423+0000 7f406a5f5700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4064097a20 0x7f406411b680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T14:43:44.451 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.423+0000 7f406a5f5700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f406411c7a0 con 0x7f4064097a20 2023-01-24T14:43:44.451 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.423+0000 7f4063fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4064097a20 0x7f406411b680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T14:43:44.451 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.423+0000 7f4063fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4064097a20 0x7f406411b680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:49050/0 (socket says 172.21.15.27:49050) 2023-01-24T14:43:44.452 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.423+0000 7f4063fff700 1 -- 172.21.15.27:0/852239864 learned_addr learned my addr 172.21.15.27:0/852239864 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-01-24T14:43:44.452 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.424+0000 7ff7cb7fe700 1 -- 172.21.15.27:0/1718486145 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 42769+0+0 (crc 0 0 0) 0x7ff7d400c7b0 con 0x7ff7e4098c00 2023-01-24T14:43:44.453 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.424+0000 7ff7e9082700 1 -- 172.21.15.27:0/1718486145 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7e411cef0 con 0x7ff7e4098c00 2023-01-24T14:43:44.453 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.424+0000 7ff7cb7fe700 1 --2- 172.21.15.27:0/1718486145 >> [v2:172.21.15.35:6808/101897,v1:172.21.15.35:6809/101897] conn(0x7ff7d0035790 0x7ff7d0037ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T14:43:44.453 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.424+0000 7f4063fff700 1 -- 172.21.15.27:0/852239864 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f406409aaa0 con 0x7f4064097a20 2023-01-24T14:43:44.454 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.424+0000 7ff7cb7fe700 1 -- 172.21.15.27:0/1718486145 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 2650+0+0 (crc 0 0 0) 0x7ff7d4044070 con 0x7ff7e4098c00 2023-01-24T14:43:44.454 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.424+0000 7f4063fff700 1 --2- 172.21.15.27:0/852239864 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4064097a20 0x7f406411b680 crc :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T14:43:44.455 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.424+0000 7ff7e259c700 1 --2- 172.21.15.27:0/1718486145 >> [v2:172.21.15.35:6808/101897,v1:172.21.15.35:6809/101897] conn(0x7ff7d0035790 0x7ff7d0037ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T14:43:44.455 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.425+0000 7ff7e259c700 1 --2- 172.21.15.27:0/1718486145 >> [v2:172.21.15.35:6808/101897,v1:172.21.15.35:6809/101897] conn(0x7ff7d0035790 0x7ff7d0037ca0 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T14:43:44.456 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.429+0000 7f40617fa700 1 -- 172.21.15.27:0/852239864 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f4054005a60 con 0x7f4064097a20 2023-01-24T14:43:44.456 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.429+0000 7f40617fa700 1 -- 172.21.15.27:0/852239864 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f4054005bc0 con 0x7f4064097a20 2023-01-24T14:43:44.456 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.429+0000 7f406a5f5700 1 -- 172.21.15.27:0/852239864 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f406409a590 con 0x7f4064097a20 2023-01-24T14:43:44.456 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.429+0000 7ff7cb7fe700 1 -- 172.21.15.27:0/1718486145 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172632 (crc 0 0 0) 0x7ff7d400a120 con 0x7ff7e4098c00 2023-01-24T14:43:44.457 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.429+0000 7f40617fa700 1 -- 172.21.15.27:0/852239864 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f4054005e60 con 0x7f4064097a20 2023-01-24T14:43:44.458 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.429+0000 7f406a5f5700 1 -- 172.21.15.27:0/852239864 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f406409e040 con 0x7f4064097a20 2023-01-24T14:43:44.458 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.430+0000 7f40617fa700 1 -- 172.21.15.27:0/852239864 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 42769+0+0 (crc 0 0 0) 0x7f405400c7b0 con 0x7f4064097a20 2023-01-24T14:43:44.458 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.430+0000 7f40617fa700 1 --2- 172.21.15.27:0/852239864 >> [v2:172.21.15.35:6808/101897,v1:172.21.15.35:6809/101897] conn(0x7f40440357e0 0x7f4044037cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T14:43:44.458 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.430+0000 7f40617fa700 1 -- 172.21.15.27:0/852239864 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 2650+0+0 (crc 0 0 0) 0x7f40540441b0 con 0x7f4064097a20 2023-01-24T14:43:44.459 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.430+0000 7f406a5f5700 1 -- 172.21.15.27:0/852239864 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f406409cb80 con 0x7f4064097a20 2023-01-24T14:43:44.459 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.431+0000 7f405bfff700 1 --2- 172.21.15.27:0/852239864 >> [v2:172.21.15.35:6808/101897,v1:172.21.15.35:6809/101897] conn(0x7f40440357e0 0x7f4044037cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T14:43:44.459 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.431+0000 7f405bfff700 1 --2- 172.21.15.27:0/852239864 >> [v2:172.21.15.35:6808/101897,v1:172.21.15.35:6809/101897] conn(0x7f40440357e0 0x7f4044037cf0 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T14:43:44.459 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.435+0000 7f40617fa700 1 -- 172.21.15.27:0/852239864 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172632 (crc 0 0 0) 0x7f406409cb80 con 0x7f4064097a20 2023-01-24T14:43:44.615 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.613+0000 7fe01e182700 1 -- 172.21.15.27:0/3534383975 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7fdff8005350 con 0x7fe018098370 2023-01-24T14:43:44.615 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.613+0000 7fdffffff700 1 -- 172.21.15.27:0/3534383975 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (crc 0 0 0) 0x7fe00c00f710 con 0x7fe018098370 2023-01-24T14:43:44.615 INFO:teuthology.orchestra.run.smithi027.stdout:38654705670 2023-01-24T14:43:44.617 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.615+0000 7fe01e182700 1 -- 172.21.15.27:0/3534383975 >> [v2:172.21.15.35:6808/101897,v1:172.21.15.35:6809/101897] conn(0x7fe004035790 msgr2=0x7fe004037ca0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T14:43:44.617 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.615+0000 7fe01e182700 1 --2- 172.21.15.27:0/3534383975 >> [v2:172.21.15.35:6808/101897,v1:172.21.15.35:6809/101897] conn(0x7fe004035790 0x7fe004037ca0 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T14:43:44.617 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.615+0000 7fe01e182700 1 -- 172.21.15.27:0/3534383975 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe018098370 msgr2=0x7fe01812b330 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T14:43:44.618 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.615+0000 7fe01e182700 1 --2- 172.21.15.27:0/3534383975 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe018098370 0x7fe01812b330 crc :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T14:43:44.618 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.616+0000 7fe01e182700 1 -- 172.21.15.27:0/3534383975 shutdown_connections 2023-01-24T14:43:44.618 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.616+0000 7fe01e182700 1 --2- 172.21.15.27:0/3534383975 >> [v2:172.21.15.35:6808/101897,v1:172.21.15.35:6809/101897] conn(0x7fe004035790 0x7fe004037ca0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T14:43:44.618 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.616+0000 7fe01e182700 1 --2- 172.21.15.27:0/3534383975 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe018098370 0x7fe01812b330 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T14:43:44.618 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.616+0000 7fe01e182700 1 -- 172.21.15.27:0/3534383975 >> 172.21.15.27:0/3534383975 conn(0x7fe0180a13d0 msgr2=0x7fe01809bb20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T14:43:44.618 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.616+0000 7fe01e182700 1 -- 172.21.15.27:0/3534383975 shutdown_connections 2023-01-24T14:43:44.619 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.616+0000 7fe01e182700 1 -- 172.21.15.27:0/3534383975 wait complete. 2023-01-24T14:43:44.629 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.628+0000 7f406a5f5700 1 -- 172.21.15.27:0/852239864 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f406409cb80 con 0x7f4064097a20 2023-01-24T14:43:44.630 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 38654705670 for osd.0 2023-01-24T14:43:44.630 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.628+0000 7f40617fa700 1 -- 172.21.15.27:0/852239864 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (crc 0 0 0) 0x7f406409cb80 con 0x7f4064097a20 2023-01-24T14:43:44.631 INFO:teuthology.orchestra.run.smithi027.stdout:38654705670 2023-01-24T14:43:44.631 DEBUG:teuthology.parallel:result is None 2023-01-24T14:43:44.632 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.630+0000 7f406a5f5700 1 -- 172.21.15.27:0/852239864 >> [v2:172.21.15.35:6808/101897,v1:172.21.15.35:6809/101897] conn(0x7f40440357e0 msgr2=0x7f4044037cf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T14:43:44.632 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.630+0000 7f406a5f5700 1 --2- 172.21.15.27:0/852239864 >> [v2:172.21.15.35:6808/101897,v1:172.21.15.35:6809/101897] conn(0x7f40440357e0 0x7f4044037cf0 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T14:43:44.633 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.630+0000 7f406a5f5700 1 -- 172.21.15.27:0/852239864 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4064097a20 msgr2=0x7f406411b680 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T14:43:44.633 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.630+0000 7f406a5f5700 1 --2- 172.21.15.27:0/852239864 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4064097a20 0x7f406411b680 crc :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T14:43:44.633 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.630+0000 7f406a5f5700 1 -- 172.21.15.27:0/852239864 shutdown_connections 2023-01-24T14:43:44.633 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.630+0000 7f406a5f5700 1 --2- 172.21.15.27:0/852239864 >> [v2:172.21.15.35:6808/101897,v1:172.21.15.35:6809/101897] conn(0x7f40440357e0 0x7f4044037cf0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T14:43:44.633 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.630+0000 7f406a5f5700 1 --2- 172.21.15.27:0/852239864 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4064097a20 0x7f406411b680 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T14:43:44.633 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.630+0000 7f406a5f5700 1 -- 172.21.15.27:0/852239864 >> 172.21.15.27:0/852239864 conn(0x7f40640a13d0 msgr2=0x7f406406fab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T14:43:44.634 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.630+0000 7f406a5f5700 1 -- 172.21.15.27:0/852239864 shutdown_connections 2023-01-24T14:43:44.634 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.630+0000 7f406a5f5700 1 -- 172.21.15.27:0/852239864 wait complete. 2023-01-24T14:43:44.644 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 38654705670 for osd.2 2023-01-24T14:43:44.644 DEBUG:teuthology.parallel:result is None 2023-01-24T14:43:44.670 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.669+0000 7fc9ed8db700 1 -- 172.21.15.27:0/1155704872 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fc9e811fb40 con 0x7fc9e81300e0 2023-01-24T14:43:44.670 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.669+0000 7fc9dffff700 1 -- 172.21.15.27:0/1155704872 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (crc 0 0 0) 0x7fc9e000f210 con 0x7fc9e81300e0 2023-01-24T14:43:44.671 INFO:teuthology.orchestra.run.smithi027.stdout:38654705670 2023-01-24T14:43:44.673 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.671+0000 7fc9ed8db700 1 -- 172.21.15.27:0/1155704872 >> [v2:172.21.15.35:6808/101897,v1:172.21.15.35:6809/101897] conn(0x7fc9c80357e0 msgr2=0x7fc9c8037cf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T14:43:44.673 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.671+0000 7fc9ed8db700 1 --2- 172.21.15.27:0/1155704872 >> [v2:172.21.15.35:6808/101897,v1:172.21.15.35:6809/101897] conn(0x7fc9c80357e0 0x7fc9c8037cf0 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T14:43:44.673 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.671+0000 7fc9ed8db700 1 -- 172.21.15.27:0/1155704872 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc9e81300e0 msgr2=0x7fc9e811dc60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T14:43:44.673 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.671+0000 7fc9ed8db700 1 --2- 172.21.15.27:0/1155704872 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc9e81300e0 0x7fc9e811dc60 crc :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T14:43:44.674 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.671+0000 7fc9ed8db700 1 -- 172.21.15.27:0/1155704872 shutdown_connections 2023-01-24T14:43:44.674 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.671+0000 7fc9ed8db700 1 --2- 172.21.15.27:0/1155704872 >> [v2:172.21.15.35:6808/101897,v1:172.21.15.35:6809/101897] conn(0x7fc9c80357e0 0x7fc9c8037cf0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T14:43:44.674 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.671+0000 7fc9ed8db700 1 --2- 172.21.15.27:0/1155704872 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc9e81300e0 0x7fc9e811dc60 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T14:43:44.674 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.671+0000 7fc9ed8db700 1 -- 172.21.15.27:0/1155704872 >> 172.21.15.27:0/1155704872 conn(0x7fc9e80a13c0 msgr2=0x7fc9e809f660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T14:43:44.674 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.671+0000 7fc9ed8db700 1 -- 172.21.15.27:0/1155704872 shutdown_connections 2023-01-24T14:43:44.675 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.671+0000 7fc9ed8db700 1 -- 172.21.15.27:0/1155704872 wait complete. 2023-01-24T14:43:44.685 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 38654705670 for osd.1 2023-01-24T14:43:44.686 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.683+0000 7ff7e9082700 1 -- 172.21.15.27:0/1718486145 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7ff7e412cd90 con 0x7ff7e4098c00 2023-01-24T14:43:44.686 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.684+0000 7ff7cb7fe700 1 -- 172.21.15.27:0/1718486145 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (crc 0 0 0) 0x7ff7d400a120 con 0x7ff7e4098c00 2023-01-24T14:43:44.686 DEBUG:teuthology.parallel:result is None 2023-01-24T14:43:44.687 INFO:teuthology.orchestra.run.smithi027.stdout:34359738375 2023-01-24T14:43:44.688 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.686+0000 7ff7e9082700 1 -- 172.21.15.27:0/1718486145 >> [v2:172.21.15.35:6808/101897,v1:172.21.15.35:6809/101897] conn(0x7ff7d0035790 msgr2=0x7ff7d0037ca0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T14:43:44.688 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.686+0000 7ff7e9082700 1 --2- 172.21.15.27:0/1718486145 >> [v2:172.21.15.35:6808/101897,v1:172.21.15.35:6809/101897] conn(0x7ff7d0035790 0x7ff7d0037ca0 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T14:43:44.688 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.686+0000 7ff7e9082700 1 -- 172.21.15.27:0/1718486145 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff7e4098c00 msgr2=0x7ff7e411b310 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T14:43:44.688 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.686+0000 7ff7e9082700 1 --2- 172.21.15.27:0/1718486145 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff7e4098c00 0x7ff7e411b310 crc :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T14:43:44.689 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.686+0000 7ff7e9082700 1 -- 172.21.15.27:0/1718486145 shutdown_connections 2023-01-24T14:43:44.689 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.686+0000 7ff7e9082700 1 --2- 172.21.15.27:0/1718486145 >> [v2:172.21.15.35:6808/101897,v1:172.21.15.35:6809/101897] conn(0x7ff7d0035790 0x7ff7d0037ca0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T14:43:44.689 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.686+0000 7ff7e9082700 1 --2- 172.21.15.27:0/1718486145 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff7e4098c00 0x7ff7e411b310 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T14:43:44.689 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.686+0000 7ff7e9082700 1 -- 172.21.15.27:0/1718486145 >> 172.21.15.27:0/1718486145 conn(0x7ff7e409da80 msgr2=0x7ff7e412db60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T14:43:44.690 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.686+0000 7ff7e9082700 1 -- 172.21.15.27:0/1718486145 shutdown_connections 2023-01-24T14:43:44.690 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.686+0000 7ff7e9082700 1 -- 172.21.15.27:0/1718486145 wait complete. 2023-01-24T14:43:44.700 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738375 got 34359738375 for osd.3 2023-01-24T14:43:44.700 DEBUG:teuthology.parallel:result is None 2023-01-24T14:43:44.700 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-01-24T14:43:44.701 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-01-24T14:43:44.873 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.872+0000 7f93615d8700 1 Processor -- start 2023-01-24T14:43:44.874 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.872+0000 7f93615d8700 1 -- start start 2023-01-24T14:43:44.877 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.873+0000 7f93615d8700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f935c071fc0 0x7f935c12df90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T14:43:44.877 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.873+0000 7f93615d8700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f935c09a9d0 con 0x7f935c12e4d0 2023-01-24T14:43:44.878 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.873+0000 7f93615d8700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f935c099b40 con 0x7f935c071fc0 2023-01-24T14:43:44.878 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.873+0000 7f935affd700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f935c071fc0 0x7f935c12df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T14:43:44.878 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.873+0000 7f9353fff700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f935c12e4d0 0x7f935c097a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:45036/0 (socket says 172.21.15.27:45036) 2023-01-24T14:43:44.879 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.873+0000 7f9353fff700 1 -- 172.21.15.27:0/1437026073 learned_addr learned my addr 172.21.15.27:0/1437026073 (peer_addr_for_me v1:172.21.15.27:0/0) 2023-01-24T14:43:44.879 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.874+0000 7f935affd700 1 -- 172.21.15.27:0/1437026073 >> v1:172.21.15.27:6789/0 conn(0x7f935c12e4d0 legacy=0x7f935c097a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T14:43:44.879 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.874+0000 7f935affd700 1 -- 172.21.15.27:0/1437026073 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f935c099830 con 0x7f935c071fc0 2023-01-24T14:43:44.880 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.874+0000 7f935affd700 1 --2- 172.21.15.27:0/1437026073 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f935c071fc0 0x7f935c12df90 crc :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8eb652558cd7eb25 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T14:43:44.880 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.874+0000 7f935a7fc700 1 -- 172.21.15.27:0/1437026073 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f934c00b600 con 0x7f935c071fc0 2023-01-24T14:43:44.880 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.874+0000 7f935a7fc700 1 -- 172.21.15.27:0/1437026073 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f934c00b760 con 0x7f935c071fc0 2023-01-24T14:43:44.880 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.874+0000 7f935a7fc700 1 -- 172.21.15.27:0/1437026073 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f934c00ba00 con 0x7f935c071fc0 2023-01-24T14:43:44.881 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.874+0000 7f93615d8700 1 -- 172.21.15.27:0/1437026073 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f935c071fc0 msgr2=0x7f935c12df90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T14:43:44.881 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.874+0000 7f93615d8700 1 --2- 172.21.15.27:0/1437026073 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f935c071fc0 0x7f935c12df90 crc :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T14:43:44.881 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.875+0000 7f93615d8700 1 -- 172.21.15.27:0/1437026073 shutdown_connections 2023-01-24T14:43:44.881 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.875+0000 7f93615d8700 1 --2- 172.21.15.27:0/1437026073 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f935c071fc0 0x7f935c12df90 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T14:43:44.881 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.875+0000 7f93615d8700 1 -- 172.21.15.27:0/1437026073 >> 172.21.15.27:0/1437026073 conn(0x7f935c0a13d0 msgr2=0x7f935c09d2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T14:43:44.882 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.875+0000 7f93615d8700 1 -- 172.21.15.27:0/1437026073 shutdown_connections 2023-01-24T14:43:44.882 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.875+0000 7f93615d8700 1 -- 172.21.15.27:0/1437026073 wait complete. 2023-01-24T14:43:44.882 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.876+0000 7f93615d8700 1 Processor -- start 2023-01-24T14:43:44.882 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.876+0000 7f93615d8700 1 -- start start 2023-01-24T14:43:44.885 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.877+0000 7f93615d8700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f935c071fc0 0x7f935c1c4860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T14:43:44.885 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.877+0000 7f93615d8700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f935c1bb310 con 0x7f935c071fc0 2023-01-24T14:43:44.886 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.877+0000 7f935affd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f935c071fc0 0x7f935c1c4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T14:43:44.886 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.877+0000 7f935affd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f935c071fc0 0x7f935c1c4860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:49062/0 (socket says 172.21.15.27:49062) 2023-01-24T14:43:44.886 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.877+0000 7f935affd700 1 -- 172.21.15.27:0/3312980782 learned_addr learned my addr 172.21.15.27:0/3312980782 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-01-24T14:43:44.886 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.877+0000 7f935affd700 1 -- 172.21.15.27:0/3312980782 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f935c09c460 con 0x7f935c071fc0 2023-01-24T14:43:44.886 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.878+0000 7f935affd700 1 --2- 172.21.15.27:0/3312980782 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f935c071fc0 0x7f935c1c4860 crc :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T14:43:44.887 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.878+0000 7f93537fe700 1 -- 172.21.15.27:0/3312980782 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f934c013400 con 0x7f935c071fc0 2023-01-24T14:43:44.887 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.878+0000 7f93537fe700 1 -- 172.21.15.27:0/3312980782 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f934c013560 con 0x7f935c071fc0 2023-01-24T14:43:44.887 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.878+0000 7f93615d8700 1 -- 172.21.15.27:0/3312980782 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f935c09a650 con 0x7f935c071fc0 2023-01-24T14:43:44.887 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.878+0000 7f93537fe700 1 -- 172.21.15.27:0/3312980782 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f934c013800 con 0x7f935c071fc0 2023-01-24T14:43:44.887 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.878+0000 7f93615d8700 1 -- 172.21.15.27:0/3312980782 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f935c067930 con 0x7f935c071fc0 2023-01-24T14:43:44.888 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.878+0000 7f93537fe700 1 -- 172.21.15.27:0/3312980782 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 42769+0+0 (crc 0 0 0) 0x7f934c0139b0 con 0x7f935c071fc0 2023-01-24T14:43:44.888 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.879+0000 7f93537fe700 1 --2- 172.21.15.27:0/3312980782 >> [v2:172.21.15.35:6808/101897,v1:172.21.15.35:6809/101897] conn(0x7f933c0357d0 0x7f933c037ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T14:43:44.888 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.879+0000 7f93537fe700 1 -- 172.21.15.27:0/3312980782 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 2650+0+0 (crc 0 0 0) 0x7f934c018040 con 0x7f935c071fc0 2023-01-24T14:43:44.888 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.879+0000 7f93615d8700 1 -- 172.21.15.27:0/3312980782 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f935c1190f0 con 0x7f935c071fc0 2023-01-24T14:43:44.888 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.879+0000 7f9353fff700 1 --2- 172.21.15.27:0/3312980782 >> [v2:172.21.15.35:6808/101897,v1:172.21.15.35:6809/101897] conn(0x7f933c0357d0 0x7f933c037ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T14:43:44.889 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.879+0000 7f9353fff700 1 --2- 172.21.15.27:0/3312980782 >> [v2:172.21.15.35:6808/101897,v1:172.21.15.35:6809/101897] conn(0x7f933c0357d0 0x7f933c037ce0 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T14:43:44.889 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:44.884+0000 7f93537fe700 1 -- 172.21.15.27:0/3312980782 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172632 (crc 0 0 0) 0x7f934c018a80 con 0x7f935c071fc0 2023-01-24T14:43:45.074 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:45.073+0000 7f93615d8700 1 -- 172.21.15.27:0/3312980782 --> [v2:172.21.15.35:6808/101897,v1:172.21.15.35:6809/101897] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f935c074980 con 0x7f933c0357d0 2023-01-24T14:43:45.075 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:45.075+0000 7f93537fe700 1 -- 172.21.15.27:0/3312980782 <== mgr.4098 v2:172.21.15.35:6808/101897 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+30026 (crc 0 0 0) 0x7f935c074980 con 0x7f933c0357d0 2023-01-24T14:43:45.076 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-24T14:43:45.078 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:45.076+0000 7f93615d8700 1 -- 172.21.15.27:0/3312980782 >> [v2:172.21.15.35:6808/101897,v1:172.21.15.35:6809/101897] conn(0x7f933c0357d0 msgr2=0x7f933c037ce0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T14:43:45.078 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:45.076+0000 7f93615d8700 1 --2- 172.21.15.27:0/3312980782 >> [v2:172.21.15.35:6808/101897,v1:172.21.15.35:6809/101897] conn(0x7f933c0357d0 0x7f933c037ce0 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T14:43:45.079 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:45.076+0000 7f93615d8700 1 -- 172.21.15.27:0/3312980782 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f935c071fc0 msgr2=0x7f935c1c4860 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T14:43:45.079 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:45.076+0000 7f93615d8700 1 --2- 172.21.15.27:0/3312980782 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f935c071fc0 0x7f935c1c4860 crc :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T14:43:45.079 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:45.076+0000 7f93615d8700 1 -- 172.21.15.27:0/3312980782 shutdown_connections 2023-01-24T14:43:45.079 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:45.076+0000 7f93615d8700 1 --2- 172.21.15.27:0/3312980782 >> [v2:172.21.15.35:6808/101897,v1:172.21.15.35:6809/101897] conn(0x7f933c0357d0 0x7f933c037ce0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T14:43:45.079 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:45.077+0000 7f93615d8700 1 --2- 172.21.15.27:0/3312980782 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f935c071fc0 0x7f935c1c4860 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T14:43:45.080 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:45.077+0000 7f93615d8700 1 -- 172.21.15.27:0/3312980782 >> 172.21.15.27:0/3312980782 conn(0x7f935c0a13d0 msgr2=0x7f935c06ff70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T14:43:45.080 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:45.077+0000 7f93615d8700 1 -- 172.21.15.27:0/3312980782 shutdown_connections 2023-01-24T14:43:45.080 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:45.077+0000 7f93615d8700 1 -- 172.21.15.27:0/3312980782 wait complete. 2023-01-24T14:43:45.080 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2023-01-24T14:43:45.090 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":9,"stamp":"2023-01-24T14:43:44.170011+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":39,"num_read_kb":29,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":96,"ondisk_log_size":96,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":419430400,"kb_used":22712,"kb_used_data":1384,"kb_used_omap":0,"kb_used_meta":20288,"kb_avail":419407688,"statfs":{"total":429496729600,"available":429473472512,"internally_reserved":0,"allocated":1417216,"data_stored":1260624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":20774912},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":8,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-01-24T14:43:42.919200+0000","last_change":"2023-01-24T14:43:42.919604+0000","last_active":"2023-01-24T14:43:42.919200+0000","last_peered":"2023-01-24T14:43:42.919200+0000","last_clean":"2023-01-24T14:43:41.796583+0000","last_became_active":"2023-01-24T14:43:42.913821+0000","last_became_peered":"2023-01-24T14:43:42.913821+0000","last_unstale":"2023-01-24T14:43:42.919200+0000","last_undegraded":"2023-01-24T14:43:42.919200+0000","last_fullsized":"2023-01-24T14:43:42.919200+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T14:43:41.796583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T14:43:41.796583+0000","last_clean_scrub_stamp":"2023-01-24T14:43:41.796583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-01-24T14:43:42.936736+0000","last_change":"2023-01-24T14:43:42.937104+0000","last_active":"2023-01-24T14:43:42.936736+0000","last_peered":"2023-01-24T14:43:42.936736+0000","last_clean":"2023-01-24T14:43:41.796583+0000","last_became_active":"2023-01-24T14:43:42.929904+0000","last_became_peered":"2023-01-24T14:43:42.929904+0000","last_unstale":"2023-01-24T14:43:42.936736+0000","last_undegraded":"2023-01-24T14:43:42.936736+0000","last_fullsized":"2023-01-24T14:43:42.936736+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T14:43:41.796583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T14:43:41.796583+0000","last_clean_scrub_stamp":"2023-01-24T14:43:41.796583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":8,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-01-24T14:43:42.914101+0000","last_change":"2023-01-24T14:43:42.914457+0000","last_active":"2023-01-24T14:43:42.914101+0000","last_peered":"2023-01-24T14:43:42.914101+0000","last_clean":"2023-01-24T14:43:41.796583+0000","last_became_active":"2023-01-24T14:43:42.904543+0000","last_became_peered":"2023-01-24T14:43:42.904543+0000","last_unstale":"2023-01-24T14:43:42.914101+0000","last_undegraded":"2023-01-24T14:43:42.914101+0000","last_fullsized":"2023-01-24T14:43:42.914101+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T14:43:41.796583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T14:43:41.796583+0000","last_clean_scrub_stamp":"2023-01-24T14:43:41.796583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-01-24T14:43:42.921090+0000","last_change":"2023-01-24T14:43:42.921526+0000","last_active":"2023-01-24T14:43:42.921090+0000","last_peered":"2023-01-24T14:43:42.921090+0000","last_clean":"2023-01-24T14:43:41.796583+0000","last_became_active":"2023-01-24T14:43:42.916837+0000","last_became_peered":"2023-01-24T14:43:42.916837+0000","last_unstale":"2023-01-24T14:43:42.921090+0000","last_undegraded":"2023-01-24T14:43:42.921090+0000","last_fullsized":"2023-01-24T14:43:42.921090+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T14:43:41.796583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T14:43:41.796583+0000","last_clean_scrub_stamp":"2023-01-24T14:43:41.796583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"13'96","reported_seq":107,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-01-24T14:43:43.328269+0000","last_change":"2023-01-24T14:43:40.850072+0000","last_active":"2023-01-24T14:43:43.328269+0000","last_peered":"2023-01-24T14:43:43.328269+0000","last_clean":"2023-01-24T14:43:43.328269+0000","last_became_active":"2023-01-24T14:43:40.847120+0000","last_became_peered":"2023-01-24T14:43:40.847120+0000","last_unstale":"2023-01-24T14:43:43.328269+0000","last_undegraded":"2023-01-24T14:43:43.328269+0000","last_fullsized":"2023-01-24T14:43:43.328269+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T14:43:39.783371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T14:43:39.783371+0000","last_clean_scrub_stamp":"2023-01-24T14:43:39.783371+0000","objects_scrubbed":0,"log_size":96,"log_dups_size":0,"ondisk_log_size":96,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":39,"num_read_kb":29,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":8,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-01-24T14:43:42.925697+0000","last_change":"2023-01-24T14:43:42.926106+0000","last_active":"2023-01-24T14:43:42.925697+0000","last_peered":"2023-01-24T14:43:42.925697+0000","last_clean":"2023-01-24T14:43:41.796583+0000","last_became_active":"2023-01-24T14:43:42.923603+0000","last_became_peered":"2023-01-24T14:43:42.923603+0000","last_unstale":"2023-01-24T14:43:42.925697+0000","last_undegraded":"2023-01-24T14:43:42.925697+0000","last_fullsized":"2023-01-24T14:43:42.925697+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T14:43:41.796583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T14:43:41.796583+0000","last_clean_scrub_stamp":"2023-01-24T14:43:41.796583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-01-24T14:43:42.945470+0000","last_change":"2023-01-24T14:43:42.945470+0000","last_active":"2023-01-24T14:43:42.945470+0000","last_peered":"2023-01-24T14:43:42.945470+0000","last_clean":"2023-01-24T14:43:42.945470+0000","last_became_active":"2023-01-24T14:43:42.940297+0000","last_became_peered":"2023-01-24T14:43:42.940297+0000","last_unstale":"2023-01-24T14:43:42.945470+0000","last_undegraded":"2023-01-24T14:43:42.945470+0000","last_fullsized":"2023-01-24T14:43:42.945470+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T14:43:41.796583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T14:43:41.796583+0000","last_clean_scrub_stamp":"2023-01-24T14:43:41.796583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-01-24T14:43:42.942138+0000","last_change":"2023-01-24T14:43:42.942521+0000","last_active":"2023-01-24T14:43:42.942138+0000","last_peered":"2023-01-24T14:43:42.942138+0000","last_clean":"2023-01-24T14:43:41.796583+0000","last_became_active":"2023-01-24T14:43:42.936651+0000","last_became_peered":"2023-01-24T14:43:42.936651+0000","last_unstale":"2023-01-24T14:43:42.942138+0000","last_undegraded":"2023-01-24T14:43:42.942138+0000","last_fullsized":"2023-01-24T14:43:42.942138+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T14:43:41.796583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T14:43:41.796583+0000","last_clean_scrub_stamp":"2023-01-24T14:43:41.796583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":8,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-01-24T14:43:42.931320+0000","last_change":"2023-01-24T14:43:42.931711+0000","last_active":"2023-01-24T14:43:42.931320+0000","last_peered":"2023-01-24T14:43:42.931320+0000","last_clean":"2023-01-24T14:43:41.796583+0000","last_became_active":"2023-01-24T14:43:42.921706+0000","last_became_peered":"2023-01-24T14:43:42.921706+0000","last_unstale":"2023-01-24T14:43:42.931320+0000","last_undegraded":"2023-01-24T14:43:42.931320+0000","last_fullsized":"2023-01-24T14:43:42.931320+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T14:43:41.796583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T14:43:41.796583+0000","last_clean_scrub_stamp":"2023-01-24T14:43:41.796583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":39,"num_read_kb":29,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":96,"ondisk_log_size":96,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":8,"seq":34359738375,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":4928,"kb_used_data":56,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":104852672,"statfs":{"total":107374182400,"available":107369136128,"internally_reserved":0,"allocated":57344,"data_stored":19972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":9,"seq":38654705670,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":6460,"kb_used_data":636,"kb_used_omap":0,"kb_used_meta":5312,"kb_avail":104851140,"statfs":{"total":107374182400,"available":107367567360,"internally_reserved":0,"allocated":651264,"data_stored":610340,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5439488},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":9,"seq":38654705670,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":4928,"kb_used_data":56,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":104852672,"statfs":{"total":107374182400,"available":107369136128,"internally_reserved":0,"allocated":57344,"data_stored":19972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705670,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":6396,"kb_used_data":636,"kb_used_omap":0,"kb_used_meta":5248,"kb_avail":104851204,"statfs":{"total":107374182400,"available":107367632896,"internally_reserved":0,"allocated":651264,"data_stored":610340,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5373952},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2023-01-24T14:43:45.091 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-01-24T14:43:45.224 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:45.222+0000 7fb1ef117700 1 Processor -- start 2023-01-24T14:43:45.224 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:45.222+0000 7fb1ef117700 1 -- start start 2023-01-24T14:43:45.226 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:45.223+0000 7fb1ef117700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb1e809d890 0x7fb1e81351b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T14:43:45.227 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:45.223+0000 7fb1ef117700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1e809b2a0 con 0x7fb1e809cd70 2023-01-24T14:43:45.227 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:45.223+0000 7fb1ef117700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1e809edd0 con 0x7fb1e809d890 2023-01-24T14:43:45.227 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:45.223+0000 7fb1eceb3700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fb1e809cd70 0x7fb1e809d180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:45044/0 (socket says 172.21.15.27:45044) 2023-01-24T14:43:45.227 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:45.223+0000 7fb1eceb3700 1 -- 172.21.15.27:0/2951578160 learned_addr learned my addr 172.21.15.27:0/2951578160 (peer_addr_for_me v1:172.21.15.27:0/0) 2023-01-24T14:43:45.227 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:45.223+0000 7fb1dffff700 1 --2- 172.21.15.27:0/2951578160 >> v2:172.21.15.27:3300/0 conn(0x7fb1e809d890 0x7fb1e81351b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T14:43:45.228 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:45.224+0000 7fb1dffff700 1 -- 172.21.15.27:0/2951578160 >> v1:172.21.15.27:6789/0 conn(0x7fb1e809cd70 legacy=0x7fb1e809d180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T14:43:45.228 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:45.224+0000 7fb1df7fe700 1 -- 172.21.15.27:0/2951578160 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4066451697 0 0) 0x7fb1e809b2a0 con 0x7fb1e809cd70 2023-01-24T14:43:45.228 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:45.224+0000 7fb1dffff700 1 -- 172.21.15.27:0/2951578160 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1e809b6e0 con 0x7fb1e809d890 2023-01-24T14:43:45.228 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:45.224+0000 7fb1dffff700 1 --2- 172.21.15.27:0/2951578160 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb1e809d890 0x7fb1e81351b0 crc :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a78c699548d96c server_cookie=0 in_seq=0 out_seq=0 2023-01-24T14:43:45.228 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:45.224+0000 7fb1df7fe700 1 -- 172.21.15.27:0/2951578160 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fb1d8009cd0 con 0x7fb1e809d890 2023-01-24T14:43:45.229 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:45.224+0000 7fb1df7fe700 1 -- 172.21.15.27:0/2951578160 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7fb1d8009e30 con 0x7fb1e809d890 2023-01-24T14:43:45.229 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:45.224+0000 7fb1df7fe700 1 -- 172.21.15.27:0/2951578160 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fb1d800e360 con 0x7fb1e809d890 2023-01-24T14:43:45.229 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:45.225+0000 7fb1ef117700 1 -- 172.21.15.27:0/2951578160 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb1e809d890 msgr2=0x7fb1e81351b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T14:43:45.229 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:45.225+0000 7fb1ef117700 1 --2- 172.21.15.27:0/2951578160 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb1e809d890 0x7fb1e81351b0 crc :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T14:43:45.229 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:45.225+0000 7fb1ef117700 1 -- 172.21.15.27:0/2951578160 shutdown_connections 2023-01-24T14:43:45.230 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:45.225+0000 7fb1ef117700 1 --2- 172.21.15.27:0/2951578160 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb1e809d890 0x7fb1e81351b0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T14:43:45.230 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:45.225+0000 7fb1ef117700 1 -- 172.21.15.27:0/2951578160 >> 172.21.15.27:0/2951578160 conn(0x7fb1e80a2040 msgr2=0x7fb1e809de60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T14:43:45.230 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:45.225+0000 7fb1ef117700 1 -- 172.21.15.27:0/2951578160 shutdown_connections 2023-01-24T14:43:45.230 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:45.225+0000 7fb1ef117700 1 -- 172.21.15.27:0/2951578160 wait complete. 2023-01-24T14:43:45.232 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:45.226+0000 7fb1ef117700 1 Processor -- start 2023-01-24T14:43:45.233 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:45.226+0000 7fb1ef117700 1 -- start start 2023-01-24T14:43:45.233 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:45.227+0000 7fb1ef117700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb1e809cd70 0x7fb1e8119aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T14:43:45.233 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:45.227+0000 7fb1ef117700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1e81197a0 con 0x7fb1e809cd70 2023-01-24T14:43:45.233 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:45.227+0000 7fb1eceb3700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb1e809cd70 0x7fb1e8119aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T14:43:45.233 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:45.227+0000 7fb1eceb3700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb1e809cd70 0x7fb1e8119aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:49070/0 (socket says 172.21.15.27:49070) 2023-01-24T14:43:45.234 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:45.227+0000 7fb1eceb3700 1 -- 172.21.15.27:0/3674327087 learned_addr learned my addr 172.21.15.27:0/3674327087 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-01-24T14:43:45.236 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:45.228+0000 7fb1eceb3700 1 -- 172.21.15.27:0/3674327087 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1e806a7d0 con 0x7fb1e809cd70 2023-01-24T14:43:45.236 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:45.228+0000 7fb1eceb3700 1 --2- 172.21.15.27:0/3674327087 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb1e809cd70 0x7fb1e8119aa0 crc :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T14:43:45.237 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:45.228+0000 7fb1dd7fa700 1 -- 172.21.15.27:0/3674327087 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fb1e4004b80 con 0x7fb1e809cd70 2023-01-24T14:43:45.237 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:45.228+0000 7fb1dd7fa700 1 -- 172.21.15.27:0/3674327087 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7fb1e4004ce0 con 0x7fb1e809cd70 2023-01-24T14:43:45.237 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:45.228+0000 7fb1dd7fa700 1 -- 172.21.15.27:0/3674327087 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7fb1e400c520 con 0x7fb1e809cd70 2023-01-24T14:43:45.237 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:45.228+0000 7fb1ef117700 1 -- 172.21.15.27:0/3674327087 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1e806ea50 con 0x7fb1e809cd70 2023-01-24T14:43:45.237 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:45.228+0000 7fb1ef117700 1 -- 172.21.15.27:0/3674327087 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1e809abb0 con 0x7fb1e809cd70 2023-01-24T14:43:45.238 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:45.229+0000 7fb1dd7fa700 1 -- 172.21.15.27:0/3674327087 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 42769+0+0 (crc 0 0 0) 0x7fb1e400c680 con 0x7fb1e809cd70 2023-01-24T14:43:45.238 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:45.229+0000 7fb1ef117700 1 -- 172.21.15.27:0/3674327087 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1e811ab60 con 0x7fb1e809cd70 2023-01-24T14:43:45.238 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:45.229+0000 7fb1dd7fa700 1 --2- 172.21.15.27:0/3674327087 >> [v2:172.21.15.35:6808/101897,v1:172.21.15.35:6809/101897] conn(0x7fb1d40357e0 0x7fb1d4037cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T14:43:45.238 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:45.229+0000 7fb1dd7fa700 1 -- 172.21.15.27:0/3674327087 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 2650+0+0 (crc 0 0 0) 0x7fb1e4043ff0 con 0x7fb1e809cd70 2023-01-24T14:43:45.238 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:45.229+0000 7fb1dffff700 1 --2- 172.21.15.27:0/3674327087 >> [v2:172.21.15.35:6808/101897,v1:172.21.15.35:6809/101897] conn(0x7fb1d40357e0 0x7fb1d4037cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T14:43:45.239 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:45.230+0000 7fb1dffff700 1 --2- 172.21.15.27:0/3674327087 >> [v2:172.21.15.35:6808/101897,v1:172.21.15.35:6809/101897] conn(0x7fb1d40357e0 0x7fb1d4037cf0 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T14:43:45.239 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:45.234+0000 7fb1dd7fa700 1 -- 172.21.15.27:0/3674327087 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172632 (crc 0 0 0) 0x7fb1e4008c40 con 0x7fb1e809cd70 2023-01-24T14:43:45.424 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:45.422+0000 7fb1ef117700 1 -- 172.21.15.27:0/3674327087 --> [v2:172.21.15.35:6808/101897,v1:172.21.15.35:6809/101897] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb1e806c3a0 con 0x7fb1d40357e0 2023-01-24T14:43:45.425 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:45.424+0000 7fb1dd7fa700 1 -- 172.21.15.27:0/3674327087 <== mgr.4098 v2:172.21.15.35:6808/101897 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+30027 (crc 0 0 0) 0x7fb1e806c3a0 con 0x7fb1d40357e0 2023-01-24T14:43:45.425 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-24T14:43:45.428 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:45.426+0000 7fb1ef117700 1 -- 172.21.15.27:0/3674327087 >> [v2:172.21.15.35:6808/101897,v1:172.21.15.35:6809/101897] conn(0x7fb1d40357e0 msgr2=0x7fb1d4037cf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T14:43:45.428 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:45.426+0000 7fb1ef117700 1 --2- 172.21.15.27:0/3674327087 >> [v2:172.21.15.35:6808/101897,v1:172.21.15.35:6809/101897] conn(0x7fb1d40357e0 0x7fb1d4037cf0 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T14:43:45.428 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:45.426+0000 7fb1ef117700 1 -- 172.21.15.27:0/3674327087 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb1e809cd70 msgr2=0x7fb1e8119aa0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T14:43:45.428 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:45.426+0000 7fb1ef117700 1 --2- 172.21.15.27:0/3674327087 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb1e809cd70 0x7fb1e8119aa0 crc :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T14:43:45.429 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:45.426+0000 7fb1ef117700 1 -- 172.21.15.27:0/3674327087 shutdown_connections 2023-01-24T14:43:45.429 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:45.426+0000 7fb1ef117700 1 --2- 172.21.15.27:0/3674327087 >> [v2:172.21.15.35:6808/101897,v1:172.21.15.35:6809/101897] conn(0x7fb1d40357e0 0x7fb1d4037cf0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T14:43:45.429 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:45.426+0000 7fb1ef117700 1 --2- 172.21.15.27:0/3674327087 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb1e809cd70 0x7fb1e8119aa0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T14:43:45.429 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:45.426+0000 7fb1ef117700 1 -- 172.21.15.27:0/3674327087 >> 172.21.15.27:0/3674327087 conn(0x7fb1e80a2040 msgr2=0x7fb1e81b7eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T14:43:45.429 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:45.426+0000 7fb1ef117700 1 -- 172.21.15.27:0/3674327087 shutdown_connections 2023-01-24T14:43:45.429 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:45.426+0000 7fb1ef117700 1 -- 172.21.15.27:0/3674327087 wait complete. 2023-01-24T14:43:45.430 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2023-01-24T14:43:45.440 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":10,"stamp":"2023-01-24T14:43:45.345006+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":39,"num_read_kb":29,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":96,"ondisk_log_size":96,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":419430400,"kb_used":22712,"kb_used_data":1384,"kb_used_omap":0,"kb_used_meta":20288,"kb_avail":419407688,"statfs":{"total":429496729600,"available":429473472512,"internally_reserved":0,"allocated":1417216,"data_stored":1260624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":20774912},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"1.174994"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":8,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-01-24T14:43:42.919200+0000","last_change":"2023-01-24T14:43:42.919604+0000","last_active":"2023-01-24T14:43:42.919200+0000","last_peered":"2023-01-24T14:43:42.919200+0000","last_clean":"2023-01-24T14:43:41.796583+0000","last_became_active":"2023-01-24T14:43:42.913821+0000","last_became_peered":"2023-01-24T14:43:42.913821+0000","last_unstale":"2023-01-24T14:43:42.919200+0000","last_undegraded":"2023-01-24T14:43:42.919200+0000","last_fullsized":"2023-01-24T14:43:42.919200+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T14:43:41.796583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T14:43:41.796583+0000","last_clean_scrub_stamp":"2023-01-24T14:43:41.796583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-01-24T14:43:42.936736+0000","last_change":"2023-01-24T14:43:42.937104+0000","last_active":"2023-01-24T14:43:42.936736+0000","last_peered":"2023-01-24T14:43:42.936736+0000","last_clean":"2023-01-24T14:43:41.796583+0000","last_became_active":"2023-01-24T14:43:42.929904+0000","last_became_peered":"2023-01-24T14:43:42.929904+0000","last_unstale":"2023-01-24T14:43:42.936736+0000","last_undegraded":"2023-01-24T14:43:42.936736+0000","last_fullsized":"2023-01-24T14:43:42.936736+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T14:43:41.796583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T14:43:41.796583+0000","last_clean_scrub_stamp":"2023-01-24T14:43:41.796583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":8,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-01-24T14:43:42.914101+0000","last_change":"2023-01-24T14:43:42.914457+0000","last_active":"2023-01-24T14:43:42.914101+0000","last_peered":"2023-01-24T14:43:42.914101+0000","last_clean":"2023-01-24T14:43:41.796583+0000","last_became_active":"2023-01-24T14:43:42.904543+0000","last_became_peered":"2023-01-24T14:43:42.904543+0000","last_unstale":"2023-01-24T14:43:42.914101+0000","last_undegraded":"2023-01-24T14:43:42.914101+0000","last_fullsized":"2023-01-24T14:43:42.914101+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T14:43:41.796583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T14:43:41.796583+0000","last_clean_scrub_stamp":"2023-01-24T14:43:41.796583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-01-24T14:43:42.921090+0000","last_change":"2023-01-24T14:43:42.921526+0000","last_active":"2023-01-24T14:43:42.921090+0000","last_peered":"2023-01-24T14:43:42.921090+0000","last_clean":"2023-01-24T14:43:41.796583+0000","last_became_active":"2023-01-24T14:43:42.916837+0000","last_became_peered":"2023-01-24T14:43:42.916837+0000","last_unstale":"2023-01-24T14:43:42.921090+0000","last_undegraded":"2023-01-24T14:43:42.921090+0000","last_fullsized":"2023-01-24T14:43:42.921090+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T14:43:41.796583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T14:43:41.796583+0000","last_clean_scrub_stamp":"2023-01-24T14:43:41.796583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"13'96","reported_seq":107,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-01-24T14:43:43.328269+0000","last_change":"2023-01-24T14:43:40.850072+0000","last_active":"2023-01-24T14:43:43.328269+0000","last_peered":"2023-01-24T14:43:43.328269+0000","last_clean":"2023-01-24T14:43:43.328269+0000","last_became_active":"2023-01-24T14:43:40.847120+0000","last_became_peered":"2023-01-24T14:43:40.847120+0000","last_unstale":"2023-01-24T14:43:43.328269+0000","last_undegraded":"2023-01-24T14:43:43.328269+0000","last_fullsized":"2023-01-24T14:43:43.328269+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T14:43:39.783371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T14:43:39.783371+0000","last_clean_scrub_stamp":"2023-01-24T14:43:39.783371+0000","objects_scrubbed":0,"log_size":96,"log_dups_size":0,"ondisk_log_size":96,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":39,"num_read_kb":29,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":8,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-01-24T14:43:42.925697+0000","last_change":"2023-01-24T14:43:42.926106+0000","last_active":"2023-01-24T14:43:42.925697+0000","last_peered":"2023-01-24T14:43:42.925697+0000","last_clean":"2023-01-24T14:43:41.796583+0000","last_became_active":"2023-01-24T14:43:42.923603+0000","last_became_peered":"2023-01-24T14:43:42.923603+0000","last_unstale":"2023-01-24T14:43:42.925697+0000","last_undegraded":"2023-01-24T14:43:42.925697+0000","last_fullsized":"2023-01-24T14:43:42.925697+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T14:43:41.796583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T14:43:41.796583+0000","last_clean_scrub_stamp":"2023-01-24T14:43:41.796583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-01-24T14:43:42.945470+0000","last_change":"2023-01-24T14:43:42.945470+0000","last_active":"2023-01-24T14:43:42.945470+0000","last_peered":"2023-01-24T14:43:42.945470+0000","last_clean":"2023-01-24T14:43:42.945470+0000","last_became_active":"2023-01-24T14:43:42.940297+0000","last_became_peered":"2023-01-24T14:43:42.940297+0000","last_unstale":"2023-01-24T14:43:42.945470+0000","last_undegraded":"2023-01-24T14:43:42.945470+0000","last_fullsized":"2023-01-24T14:43:42.945470+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T14:43:41.796583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T14:43:41.796583+0000","last_clean_scrub_stamp":"2023-01-24T14:43:41.796583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-01-24T14:43:42.942138+0000","last_change":"2023-01-24T14:43:42.942521+0000","last_active":"2023-01-24T14:43:42.942138+0000","last_peered":"2023-01-24T14:43:42.942138+0000","last_clean":"2023-01-24T14:43:41.796583+0000","last_became_active":"2023-01-24T14:43:42.936651+0000","last_became_peered":"2023-01-24T14:43:42.936651+0000","last_unstale":"2023-01-24T14:43:42.942138+0000","last_undegraded":"2023-01-24T14:43:42.942138+0000","last_fullsized":"2023-01-24T14:43:42.942138+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T14:43:41.796583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T14:43:41.796583+0000","last_clean_scrub_stamp":"2023-01-24T14:43:41.796583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":8,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-01-24T14:43:42.931320+0000","last_change":"2023-01-24T14:43:42.931711+0000","last_active":"2023-01-24T14:43:42.931320+0000","last_peered":"2023-01-24T14:43:42.931320+0000","last_clean":"2023-01-24T14:43:41.796583+0000","last_became_active":"2023-01-24T14:43:42.921706+0000","last_became_peered":"2023-01-24T14:43:42.921706+0000","last_unstale":"2023-01-24T14:43:42.931320+0000","last_undegraded":"2023-01-24T14:43:42.931320+0000","last_fullsized":"2023-01-24T14:43:42.931320+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T14:43:41.796583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T14:43:41.796583+0000","last_clean_scrub_stamp":"2023-01-24T14:43:41.796583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":39,"num_read_kb":29,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":96,"ondisk_log_size":96,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":8,"seq":34359738375,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":4928,"kb_used_data":56,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":104852672,"statfs":{"total":107374182400,"available":107369136128,"internally_reserved":0,"allocated":57344,"data_stored":19972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":9,"seq":38654705670,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":6460,"kb_used_data":636,"kb_used_omap":0,"kb_used_meta":5312,"kb_avail":104851140,"statfs":{"total":107374182400,"available":107367567360,"internally_reserved":0,"allocated":651264,"data_stored":610340,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5439488},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":9,"seq":38654705670,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":4928,"kb_used_data":56,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":104852672,"statfs":{"total":107374182400,"available":107369136128,"internally_reserved":0,"allocated":57344,"data_stored":19972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705670,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":6396,"kb_used_data":636,"kb_used_omap":0,"kb_used_meta":5248,"kb_avail":104851204,"statfs":{"total":107374182400,"available":107367632896,"internally_reserved":0,"allocated":651264,"data_stored":610340,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5373952},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2023-01-24T14:43:45.441 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-01-24T14:43:45.441 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2023-01-24T14:43:45.441 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2023-01-24T14:43:45.442 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-01-24T14:43:45.573 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:45.572+0000 7f00f3dcb700 1 Processor -- start 2023-01-24T14:43:45.573 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:45.572+0000 7f00f3dcb700 1 -- start start 2023-01-24T14:43:45.575 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:45.573+0000 7f00f3dcb700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f00ec098130 0x7f00ec1351b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T14:43:45.576 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:45.573+0000 7f00f3dcb700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00ec098b10 con 0x7f00ec099950 2023-01-24T14:43:45.576 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:45.573+0000 7f00f3dcb700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00ec071670 con 0x7f00ec098130 2023-01-24T14:43:45.576 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:45.573+0000 7f00f1366700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f00ec098130 0x7f00ec1351b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T14:43:45.576 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:45.573+0000 7f00f1b67700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f00ec099950 0x7f00ec097a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:45056/0 (socket says 172.21.15.27:45056) 2023-01-24T14:43:45.577 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:45.573+0000 7f00f1b67700 1 -- 172.21.15.27:0/3876536822 learned_addr learned my addr 172.21.15.27:0/3876536822 (peer_addr_for_me v1:172.21.15.27:0/0) 2023-01-24T14:43:45.577 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:45.573+0000 7f00f1366700 1 -- 172.21.15.27:0/3876536822 >> v1:172.21.15.27:6789/0 conn(0x7f00ec099950 legacy=0x7f00ec097a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T14:43:45.577 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:45.573+0000 7f00f1366700 1 -- 172.21.15.27:0/3876536822 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00ec071360 con 0x7f00ec098130 2023-01-24T14:43:45.577 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:45.573+0000 7f00f0b65700 1 -- 172.21.15.27:0/3876536822 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 203552034 0 0) 0x7f00ec098b10 con 0x7f00ec099950 2023-01-24T14:43:45.577 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:45.573+0000 7f00f1366700 1 --2- 172.21.15.27:0/3876536822 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f00ec098130 0x7f00ec1351b0 crc :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf414c5dd2f79241 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T14:43:45.578 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:45.573+0000 7f00f0b65700 1 -- 172.21.15.27:0/3876536822 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f00e8009cd0 con 0x7f00ec098130 2023-01-24T14:43:45.578 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:45.573+0000 7f00f0b65700 1 -- 172.21.15.27:0/3876536822 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f00e8009e30 con 0x7f00ec098130 2023-01-24T14:43:45.578 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:45.573+0000 7f00f0b65700 1 -- 172.21.15.27:0/3876536822 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f00e800e360 con 0x7f00ec098130 2023-01-24T14:43:45.578 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:45.574+0000 7f00f3dcb700 1 -- 172.21.15.27:0/3876536822 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f00ec098130 msgr2=0x7f00ec1351b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T14:43:45.578 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:45.574+0000 7f00f3dcb700 1 --2- 172.21.15.27:0/3876536822 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f00ec098130 0x7f00ec1351b0 crc :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T14:43:45.578 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:45.574+0000 7f00f3dcb700 1 -- 172.21.15.27:0/3876536822 shutdown_connections 2023-01-24T14:43:45.579 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:45.574+0000 7f00f3dcb700 1 --2- 172.21.15.27:0/3876536822 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f00ec098130 0x7f00ec1351b0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T14:43:45.579 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:45.574+0000 7f00f3dcb700 1 -- 172.21.15.27:0/3876536822 >> 172.21.15.27:0/3876536822 conn(0x7f00ec0a31f0 msgr2=0x7f00ec06f8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T14:43:45.579 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:45.574+0000 7f00f3dcb700 1 -- 172.21.15.27:0/3876536822 shutdown_connections 2023-01-24T14:43:45.579 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:45.574+0000 7f00f3dcb700 1 -- 172.21.15.27:0/3876536822 wait complete. 2023-01-24T14:43:45.582 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:45.576+0000 7f00f3dcb700 1 Processor -- start 2023-01-24T14:43:45.582 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:45.576+0000 7f00f3dcb700 1 -- start start 2023-01-24T14:43:45.582 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:45.577+0000 7f00f3dcb700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f00ec098130 0x7f00ec120060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T14:43:45.582 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:45.577+0000 7f00f3dcb700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00ec121ad0 con 0x7f00ec098130 2023-01-24T14:43:45.582 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:45.577+0000 7f00f1b67700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f00ec098130 0x7f00ec120060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T14:43:45.583 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:45.577+0000 7f00f1b67700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f00ec098130 0x7f00ec120060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:49088/0 (socket says 172.21.15.27:49088) 2023-01-24T14:43:45.583 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:45.577+0000 7f00f1b67700 1 -- 172.21.15.27:0/1686178959 learned_addr learned my addr 172.21.15.27:0/1686178959 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-01-24T14:43:45.583 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:45.577+0000 7f00f1b67700 1 -- 172.21.15.27:0/1686178959 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00ec09a160 con 0x7f00ec098130 2023-01-24T14:43:45.583 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:45.577+0000 7f00f1b67700 1 --2- 172.21.15.27:0/1686178959 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f00ec098130 0x7f00ec120060 crc :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T14:43:45.583 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:45.577+0000 7f00de7fc700 1 -- 172.21.15.27:0/1686178959 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f00e0005a60 con 0x7f00ec098130 2023-01-24T14:43:45.584 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:45.577+0000 7f00de7fc700 1 -- 172.21.15.27:0/1686178959 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (crc 0 0 0) 0x7f00e0005bc0 con 0x7f00ec098130 2023-01-24T14:43:45.584 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:45.577+0000 7f00f3dcb700 1 -- 172.21.15.27:0/1686178959 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00ec09ad70 con 0x7f00ec098130 2023-01-24T14:43:45.585 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:45.577+0000 7f00de7fc700 1 -- 172.21.15.27:0/1686178959 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (crc 0 0 0) 0x7f00e0005e60 con 0x7f00ec098130 2023-01-24T14:43:45.585 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:45.577+0000 7f00f3dcb700 1 -- 172.21.15.27:0/1686178959 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00ec09a2f0 con 0x7f00ec098130 2023-01-24T14:43:45.585 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:45.578+0000 7f00de7fc700 1 -- 172.21.15.27:0/1686178959 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 42769+0+0 (crc 0 0 0) 0x7f00e00070b0 con 0x7f00ec098130 2023-01-24T14:43:45.586 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:45.578+0000 7f00f3dcb700 1 -- 172.21.15.27:0/1686178959 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00ec074980 con 0x7f00ec098130 2023-01-24T14:43:45.586 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:45.578+0000 7f00de7fc700 1 --2- 172.21.15.27:0/1686178959 >> [v2:172.21.15.35:6808/101897,v1:172.21.15.35:6809/101897] conn(0x7f00d00357e0 0x7f00d0037cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T14:43:45.586 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:45.578+0000 7f00de7fc700 1 -- 172.21.15.27:0/1686178959 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 2650+0+0 (crc 0 0 0) 0x7f00e00441e0 con 0x7f00ec098130 2023-01-24T14:43:45.586 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:45.579+0000 7f00f1366700 1 --2- 172.21.15.27:0/1686178959 >> [v2:172.21.15.35:6808/101897,v1:172.21.15.35:6809/101897] conn(0x7f00d00357e0 0x7f00d0037cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T14:43:45.587 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:45.579+0000 7f00f1366700 1 --2- 172.21.15.27:0/1686178959 >> [v2:172.21.15.35:6808/101897,v1:172.21.15.35:6809/101897] conn(0x7f00d00357e0 0x7f00d0037cf0 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T14:43:45.587 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:45.583+0000 7f00de7fc700 1 -- 172.21.15.27:0/1686178959 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172632 (crc 0 0 0) 0x7f00e000a300 con 0x7f00ec098130 2023-01-24T14:43:45.824 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:45.822+0000 7f00f3dcb700 1 -- 172.21.15.27:0/1686178959 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f00ec121140 con 0x7f00ec098130 2023-01-24T14:43:45.824 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:45.822+0000 7f00de7fc700 1 -- 172.21.15.27:0/1686178959 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (crc 0 0 0) 0x7f00e000a120 con 0x7f00ec098130 2023-01-24T14:43:45.824 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-24T14:43:45.825 INFO:teuthology.orchestra.run.smithi027.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-01-24T14:43:45.826 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:45.824+0000 7f00f3dcb700 1 -- 172.21.15.27:0/1686178959 >> [v2:172.21.15.35:6808/101897,v1:172.21.15.35:6809/101897] conn(0x7f00d00357e0 msgr2=0x7f00d0037cf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T14:43:45.826 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:45.824+0000 7f00f3dcb700 1 --2- 172.21.15.27:0/1686178959 >> [v2:172.21.15.35:6808/101897,v1:172.21.15.35:6809/101897] conn(0x7f00d00357e0 0x7f00d0037cf0 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T14:43:45.827 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:45.825+0000 7f00f3dcb700 1 -- 172.21.15.27:0/1686178959 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f00ec098130 msgr2=0x7f00ec120060 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T14:43:45.827 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:45.825+0000 7f00f3dcb700 1 --2- 172.21.15.27:0/1686178959 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f00ec098130 0x7f00ec120060 crc :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T14:43:45.827 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:45.825+0000 7f00f3dcb700 1 -- 172.21.15.27:0/1686178959 shutdown_connections 2023-01-24T14:43:45.827 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:45.825+0000 7f00f3dcb700 1 --2- 172.21.15.27:0/1686178959 >> [v2:172.21.15.35:6808/101897,v1:172.21.15.35:6809/101897] conn(0x7f00d00357e0 0x7f00d0037cf0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T14:43:45.828 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:45.825+0000 7f00f3dcb700 1 --2- 172.21.15.27:0/1686178959 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f00ec098130 0x7f00ec120060 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T14:43:45.828 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:45.825+0000 7f00f3dcb700 1 -- 172.21.15.27:0/1686178959 >> 172.21.15.27:0/1686178959 conn(0x7f00ec0a31f0 msgr2=0x7f00ec118bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T14:43:45.828 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:45.825+0000 7f00f3dcb700 1 -- 172.21.15.27:0/1686178959 shutdown_connections 2023-01-24T14:43:45.828 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-24T14:43:45.825+0000 7f00f3dcb700 1 -- 172.21.15.27:0/1686178959 wait complete. 2023-01-24T14:43:45.838 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2023-01-24T14:43:45.839 INFO:teuthology.run_tasks:Running task workunit... 2023-01-24T14:43:45.857 INFO:tasks.workunit:Pulling workunits from ref 510284b66513490445619d1430aa869868c71a09 2023-01-24T14:43:45.858 INFO:tasks.workunit:Making a separate scratch dir for every client... 2023-01-24T14:43:45.858 DEBUG:teuthology.orchestra.run.smithi027:> stat -- /home/ubuntu/cephtest/mnt.0 2023-01-24T14:43:45.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-01-24T14:43:45.877 INFO:teuthology.orchestra.run.smithi027.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2023-01-24T14:43:45.878 DEBUG:teuthology.orchestra.run.smithi027:> mkdir -- /home/ubuntu/cephtest/mnt.0 2023-01-24T14:43:45.935 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2023-01-24T14:43:45.936 DEBUG:teuthology.orchestra.run.smithi027:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2023-01-24T14:43:45.993 INFO:tasks.workunit:timeout=3h 2023-01-24T14:43:45.994 INFO:tasks.workunit:cleanup=True 2023-01-24T14:43:45.994 DEBUG:teuthology.orchestra.run.smithi027:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 510284b66513490445619d1430aa869868c71a09 2023-01-24T14:43:46.053 INFO:tasks.workunit.client.0.smithi027.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2023-01-24T14:44:31.214 INFO:tasks.workunit.client.0.smithi027.stderr:Updating files: 93% (10931/11669) Updating files: 94% (10969/11669) Updating files: 95% (11086/11669) Updating files: 96% (11203/11669) Updating files: 97% (11319/11669) Updating files: 98% (11436/11669) Updating files: 99% (11553/11669) Updating files: 100% (11669/11669) Updating files: 100% (11669/11669), done. 2023-01-24T14:44:31.381 INFO:tasks.workunit.client.0.smithi027.stderr:Note: switching to '510284b66513490445619d1430aa869868c71a09'. 2023-01-24T14:44:31.382 INFO:tasks.workunit.client.0.smithi027.stderr: 2023-01-24T14:44:31.382 INFO:tasks.workunit.client.0.smithi027.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2023-01-24T14:44:31.382 INFO:tasks.workunit.client.0.smithi027.stderr:changes and commit them, and you can discard any commits you make in this 2023-01-24T14:44:31.382 INFO:tasks.workunit.client.0.smithi027.stderr:state without impacting any branches by switching back to a branch. 2023-01-24T14:44:31.383 INFO:tasks.workunit.client.0.smithi027.stderr: 2023-01-24T14:44:31.383 INFO:tasks.workunit.client.0.smithi027.stderr:If you want to create a new branch to retain commits you create, you may 2023-01-24T14:44:31.383 INFO:tasks.workunit.client.0.smithi027.stderr:do so (now or later) by using -c with the switch command. Example: 2023-01-24T14:44:31.383 INFO:tasks.workunit.client.0.smithi027.stderr: 2023-01-24T14:44:31.383 INFO:tasks.workunit.client.0.smithi027.stderr: git switch -c 2023-01-24T14:44:31.384 INFO:tasks.workunit.client.0.smithi027.stderr: 2023-01-24T14:44:31.384 INFO:tasks.workunit.client.0.smithi027.stderr:Or undo this operation with: 2023-01-24T14:44:31.384 INFO:tasks.workunit.client.0.smithi027.stderr: 2023-01-24T14:44:31.384 INFO:tasks.workunit.client.0.smithi027.stderr: git switch - 2023-01-24T14:44:31.384 INFO:tasks.workunit.client.0.smithi027.stderr: 2023-01-24T14:44:31.385 INFO:tasks.workunit.client.0.smithi027.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2023-01-24T14:44:31.385 INFO:tasks.workunit.client.0.smithi027.stderr: 2023-01-24T14:44:31.385 INFO:tasks.workunit.client.0.smithi027.stderr:HEAD is now at 510284b6651 Merge pull request #49791 from idryomov/wip-doc-exclusive-lock-transitions 2023-01-24T14:44:31.386 DEBUG:teuthology.orchestra.run.smithi027:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2023-01-24T14:44:31.456 INFO:tasks.workunit.client.0.smithi027.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2023-01-24T14:44:31.458 INFO:tasks.workunit.client.0.smithi027.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2023-01-24T14:44:31.459 INFO:tasks.workunit.client.0.smithi027.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2023-01-24T14:44:32.002 INFO:tasks.workunit.client.0.smithi027.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2023-01-24T14:44:32.061 INFO:tasks.workunit.client.0.smithi027.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2023-01-24T14:44:32.101 INFO:tasks.workunit.client.0.smithi027.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2023-01-24T14:44:32.102 INFO:tasks.workunit.client.0.smithi027.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2023-01-24T14:44:32.103 INFO:tasks.workunit.client.0.smithi027.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2023-01-24T14:44:32.141 INFO:tasks.workunit.client.0.smithi027.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2023-01-24T14:44:32.144 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2023-01-24T14:44:32.144 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2023-01-24T14:44:32.202 INFO:tasks.workunit:Running workunits matching rados/test.sh on client.0... 2023-01-24T14:44:32.204 INFO:tasks.workunit:Running workunit rados/test.sh... 2023-01-24T14:44:32.204 DEBUG:teuthology.orchestra.run.smithi027:workunit test rados/test.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=510284b66513490445619d1430aa869868c71a09 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 CRIMSON_COMPAT=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh 2023-01-24T14:44:32.265 INFO:tasks.workunit.client.0.smithi027.stderr:+ parallel=1 2023-01-24T14:44:32.265 INFO:tasks.workunit.client.0.smithi027.stderr:+ '[' '' = --serial ']' 2023-01-24T14:44:32.266 INFO:tasks.workunit.client.0.smithi027.stderr:+ color= 2023-01-24T14:44:32.266 INFO:tasks.workunit.client.0.smithi027.stderr:+ '[' -t 1 ']' 2023-01-24T14:44:32.266 INFO:tasks.workunit.client.0.smithi027.stderr:+ trap cleanup EXIT ERR HUP INT QUIT 2023-01-24T14:44:32.266 INFO:tasks.workunit.client.0.smithi027.stderr:+ declare -A pids 2023-01-24T14:44:32.266 INFO:tasks.workunit.client.0.smithi027.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-01-24T14:44:32.267 INFO:tasks.workunit.client.0.smithi027.stderr:+ '[' 1 -eq 1 ']' 2023-01-24T14:44:32.267 INFO:tasks.workunit.client.0.smithi027.stderr:++ printf %25s api_aio 2023-01-24T14:44:32.267 INFO:tasks.workunit.client.0.smithi027.stderr:+ r=' api_aio' 2023-01-24T14:44:32.267 INFO:tasks.workunit.client.0.smithi027.stderr:++ echo api_aio 2023-01-24T14:44:32.267 INFO:tasks.workunit.client.0.smithi027.stderr:++ awk '{print $1}' 2023-01-24T14:44:32.268 INFO:tasks.workunit.client.0.smithi027.stderr:+ ff=api_aio 2023-01-24T14:44:32.268 INFO:tasks.workunit.client.0.smithi027.stderr:+ pid=104561 2023-01-24T14:44:32.268 INFO:tasks.workunit.client.0.smithi027.stderr:+ echo 'test api_aio on pid 104561' 2023-01-24T14:44:32.269 INFO:tasks.workunit.client.0.smithi027.stdout:test api_aio on pid 104561 2023-01-24T14:44:32.270 INFO:tasks.workunit.client.0.smithi027.stderr:+ pids[$f]=104561 2023-01-24T14:44:32.271 INFO:tasks.workunit.client.0.smithi027.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-01-24T14:44:32.271 INFO:tasks.workunit.client.0.smithi027.stderr:+ '[' 1 -eq 1 ']' 2023-01-24T14:44:32.271 INFO:tasks.workunit.client.0.smithi027.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_aio 2>&1 | tee ceph_test_rados_api_aio.log | sed "s/^/ api_aio: /"' 2023-01-24T14:44:32.271 INFO:tasks.workunit.client.0.smithi027.stderr:++ printf %25s api_aio_pp 2023-01-24T14:44:32.272 INFO:tasks.workunit.client.0.smithi027.stderr:+ r=' api_aio_pp' 2023-01-24T14:44:32.272 INFO:tasks.workunit.client.0.smithi027.stderr:++ echo api_aio_pp 2023-01-24T14:44:32.272 INFO:tasks.workunit.client.0.smithi027.stderr:++ awk '{print $1}' 2023-01-24T14:44:32.272 INFO:tasks.workunit.client.0.smithi027.stderr:+ ceph_test_rados_api_aio 2023-01-24T14:44:32.272 INFO:tasks.workunit.client.0.smithi027.stderr:+ tee ceph_test_rados_api_aio.log 2023-01-24T14:44:32.272 INFO:tasks.workunit.client.0.smithi027.stderr:+ sed 's/^/ api_aio: /' 2023-01-24T14:44:32.273 INFO:tasks.workunit.client.0.smithi027.stderr:+ ff=api_aio_pp 2023-01-24T14:44:32.273 INFO:tasks.workunit.client.0.smithi027.stderr:+ pid=104569 2023-01-24T14:44:32.273 INFO:tasks.workunit.client.0.smithi027.stderr:+ echo 'test api_aio_pp on pid 104569' 2023-01-24T14:44:32.273 INFO:tasks.workunit.client.0.smithi027.stdout:test api_aio_pp on pid 104569 2023-01-24T14:44:32.275 INFO:tasks.workunit.client.0.smithi027.stderr:+ pids[$f]=104569 2023-01-24T14:44:32.275 INFO:tasks.workunit.client.0.smithi027.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-01-24T14:44:32.275 INFO:tasks.workunit.client.0.smithi027.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_aio_pp 2>&1 | tee ceph_test_rados_api_aio_pp.log | sed "s/^/ api_aio_pp: /"' 2023-01-24T14:44:32.276 INFO:tasks.workunit.client.0.smithi027.stderr:+ '[' 1 -eq 1 ']' 2023-01-24T14:44:32.276 INFO:tasks.workunit.client.0.smithi027.stderr:++ printf %25s api_io 2023-01-24T14:44:32.276 INFO:tasks.workunit.client.0.smithi027.stderr:+ r=' api_io' 2023-01-24T14:44:32.276 INFO:tasks.workunit