2023-01-24T14:18:18.671 INFO:root:teuthology version: 0.0.1.dev22+g8e1bba1 2023-01-24T14:18:18.672 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_rzarzynski@teuthology --description crimson-rados/basic/{centos_latest clusters/fixed-2 crimson_qa_overrides deploy/ceph tasks/rados_python} --name rzarzynski-2023-01-24_14:08:39-crimson-rados-main-distro-default-smithi --archive /home/teuthworker/archive/rzarzynski-2023-01-24_14:08:39-crimson-rados-main-distro-default-smithi/7135315 --verbose -- /home/teuthworker/archive/rzarzynski-2023-01-24_14:08:39-crimson-rados-main-distro-default-smithi/7135315/orig.config.yaml 2023-01-24T14:18:18.699 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2023-01-24T14:18:18.760 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/rzarzynski-2023-01-24_14:08:39-crimson-rados-main-distro-default-smithi/7135315 branch: main description: crimson-rados/basic/{centos_latest clusters/fixed-2 crimson_qa_overrides deploy/ceph tasks/rados_python} email: null first_in_suite: false job_id: '7135315' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: rzarzynski-2023-01-24_14:08:39-crimson-rados-main-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 8.stream overrides: admin_socket: branch: main ceph: conf: global: ms client mode: crc ms cluster mode: crc ms mon client mode: crc ms mon cluster mode: crc ms mon service mode: crc ms service mode: crc mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 osd shutdown pgref assert: true flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - but it is still running - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(PG_ - \(OSD_ - \(OBJECT_ - \(POOL_APP_NOT_ENABLED\) sha1: 1a671c198559f54389cd5afa22085a0c7b29a1ef ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: flavor: crimson sha1: 1a671c198559f54389cd5afa22085a0c7b29a1ef selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: main env: CRIMSON_COMPAT: '1' sha1: 510284b66513490445619d1430aa869868c71a09 owner: scheduled_rzarzynski@teuthology priority: 60 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - osd.0 - osd.1 - client.0 - node-exporter.a - - mgr.x - osd.2 - osd.3 - client.1 - prometheus.a - node-exporter.b sha1: 1a671c198559f54389cd5afa22085a0c7b29a1ef sleep_before_teardown: 0 suite: crimson-rados suite_branch: main suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_510284b66513490445619d1430aa869868c71a09/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 510284b66513490445619d1430aa869868c71a09 targets: smithi002.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJx0+gMikQqwdjPEHuD51w6FELOppJvYH1UJ4tUlNN9c5SvEHEwW7Upjb8SVQrE3vpg0ftDnWrtyoUpxLS6Mnq8= smithi039.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC9DJRQAR3yJe+Cm2yKBWSi166E6Ri+4Ig4WOqTjM+nJWiRQ+TQp4q28NtgDtMBJFykmeBqA98y5mpifcEN10Dt852OyQQWDWCT7t8oWA6w32C/jc8llTrodbuFvZwuCcFBOn9TWziNmysTJNLMd0BfvEmTc4L1wrYy/ds3R/29K41E5Q1RWhkb+E5Z5byG/kWkrgb2jomXexJZZHt5lFgx7mGDxr+pyIGM8PmSfSoiRhyQ2Tf8Sd6KbiNwmFjkglOOkweLwvjbykAmLoKjItl/o+AWDvgc+0/n/gkzBCtDXFpijbWN4ZHLrv6vtiGkoU6rtg7l5eRs5kxK6QZ8HDAv9P1pUpSfIMGCbtI1Z1LXWFHnsRsuFJfPhLyK/CF1sG9svHBBbOzlC3K4OfS6Xv8MF1V0iDhd7R3Thcih/XkYQ2nF0dWD/ktc5+tvVmhTIsuoexAKdjhqv82HYVQgPpwl/wynbhB1UebXQWl1oHrwF4LhYC90Whkpg4CLTNnvHVc= tasks: - install: null - ceph: conf: mon: mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 paxos service trim min: 10 osd: debug monc: 20 flavor: crimson - workunit: clients: client.0: - rados/test_python.sh --eval-attr 'not (wait or tier or ec or bench or stats)' timeout: 1h teuthology_branch: main teuthology_sha1: 8e1bba12e584b7ae912e92e94afe9c075ab51fa3 timestamp: 2023-01-24_14:08:39 tube: smithi user: rzarzynski verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.17735 2023-01-24T14:18:18.760 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_510284b66513490445619d1430aa869868c71a09/qa; will attempt to use it 2023-01-24T14:18:18.761 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_510284b66513490445619d1430aa869868c71a09/qa/tasks 2023-01-24T14:18:18.761 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-01-24T14:18:18.763 INFO:teuthology.task.internal:Checking packages... 2023-01-24T14:18:18.790 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'crimson' and ceph hash '1a671c198559f54389cd5afa22085a0c7b29a1ef' 2023-01-24T14:18:18.790 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-01-24T14:18:18.791 INFO:teuthology.packaging:ref: None 2023-01-24T14:18:18.791 INFO:teuthology.packaging:tag: None 2023-01-24T14:18:18.791 INFO:teuthology.packaging:branch: main 2023-01-24T14:18:18.792 INFO:teuthology.packaging:sha1: 1a671c198559f54389cd5afa22085a0c7b29a1ef 2023-01-24T14:18:18.792 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F8%2Fx86_64&ref=main 2023-01-24T14:18:19.018 INFO:teuthology.task.internal:Found packages for ceph version 18.0.0-1954.g1a671c19 2023-01-24T14:18:19.019 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-01-24T14:18:19.026 INFO:teuthology.task.internal:no buildpackages task found 2023-01-24T14:18:19.026 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-01-24T14:18:19.044 INFO:teuthology.task.internal:Saving configuration 2023-01-24T14:18:19.054 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-01-24T14:18:19.060 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-01-24T14:18:19.084 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi002.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rzarzynski-2023-01-24_14:08:39-crimson-rados-main-distro-default-smithi/7135315', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-01-24 14:12:36.496868', 'locked_by': 'scheduled_rzarzynski@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJx0+gMikQqwdjPEHuD51w6FELOppJvYH1UJ4tUlNN9c5SvEHEwW7Upjb8SVQrE3vpg0ftDnWrtyoUpxLS6Mnq8='} 2023-01-24T14:18:19.107 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi039.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rzarzynski-2023-01-24_14:08:39-crimson-rados-main-distro-default-smithi/7135315', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-01-24 14:12:36.495330', 'locked_by': 'scheduled_rzarzynski@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBP3wq/O2d21q66rST0GwpPmSqmuDbHPBY1Vca6VjHl0G3CePnD241ckM6o03mX1m2XC38uMrzousKR9q8gzoMJs='} 2023-01-24T14:18:19.107 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-01-24T14:18:19.114 INFO:teuthology.task.internal:roles: ubuntu@smithi002.front.sepia.ceph.com - ['mon.a', 'osd.0', 'osd.1', 'client.0', 'node-exporter.a'] 2023-01-24T14:18:19.114 INFO:teuthology.task.internal:roles: ubuntu@smithi039.front.sepia.ceph.com - ['mgr.x', 'osd.2', 'osd.3', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-24T14:18:19.114 INFO:teuthology.run_tasks:Running task console_log... 2023-01-24T14:18:19.184 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fa07394f5e0>, signals=[15]) 2023-01-24T14:18:19.185 INFO:teuthology.run_tasks:Running task internal.connect... 2023-01-24T14:18:19.191 INFO:teuthology.task.internal:Opening connections... 2023-01-24T14:18:19.191 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi002.front.sepia.ceph.com 2023-01-24T14:18:19.193 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi002.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-01-24T14:18:19.261 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi039.front.sepia.ceph.com 2023-01-24T14:18:19.262 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi039.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-01-24T14:18:19.330 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-01-24T14:18:19.338 DEBUG:teuthology.orchestra.run.smithi002:> uname -m 2023-01-24T14:18:19.358 INFO:teuthology.orchestra.run.smithi002.stdout:x86_64 2023-01-24T14:18:19.359 DEBUG:teuthology.orchestra.run.smithi002:> cat /etc/os-release 2023-01-24T14:18:19.418 INFO:teuthology.orchestra.run.smithi002.stdout:NAME="CentOS Stream" 2023-01-24T14:18:19.418 INFO:teuthology.orchestra.run.smithi002.stdout:VERSION="8" 2023-01-24T14:18:19.418 INFO:teuthology.orchestra.run.smithi002.stdout:ID="centos" 2023-01-24T14:18:19.418 INFO:teuthology.orchestra.run.smithi002.stdout:ID_LIKE="rhel fedora" 2023-01-24T14:18:19.419 INFO:teuthology.orchestra.run.smithi002.stdout:VERSION_ID="8" 2023-01-24T14:18:19.419 INFO:teuthology.orchestra.run.smithi002.stdout:PLATFORM_ID="platform:el8" 2023-01-24T14:18:19.419 INFO:teuthology.orchestra.run.smithi002.stdout:PRETTY_NAME="CentOS Stream 8" 2023-01-24T14:18:19.419 INFO:teuthology.orchestra.run.smithi002.stdout:ANSI_COLOR="0;31" 2023-01-24T14:18:19.419 INFO:teuthology.orchestra.run.smithi002.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2023-01-24T14:18:19.420 INFO:teuthology.orchestra.run.smithi002.stdout:HOME_URL="https://centos.org/" 2023-01-24T14:18:19.420 INFO:teuthology.orchestra.run.smithi002.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-01-24T14:18:19.420 INFO:teuthology.orchestra.run.smithi002.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2023-01-24T14:18:19.420 INFO:teuthology.orchestra.run.smithi002.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-01-24T14:18:19.421 INFO:teuthology.lock.ops:Updating smithi002.front.sepia.ceph.com on lock server 2023-01-24T14:18:19.437 DEBUG:teuthology.orchestra.run.smithi039:> uname -m 2023-01-24T14:18:19.457 INFO:teuthology.orchestra.run.smithi039.stdout:x86_64 2023-01-24T14:18:19.457 DEBUG:teuthology.orchestra.run.smithi039:> cat /etc/os-release 2023-01-24T14:18:19.517 INFO:teuthology.orchestra.run.smithi039.stdout:NAME="CentOS Stream" 2023-01-24T14:18:19.517 INFO:teuthology.orchestra.run.smithi039.stdout:VERSION="8" 2023-01-24T14:18:19.517 INFO:teuthology.orchestra.run.smithi039.stdout:ID="centos" 2023-01-24T14:18:19.517 INFO:teuthology.orchestra.run.smithi039.stdout:ID_LIKE="rhel fedora" 2023-01-24T14:18:19.518 INFO:teuthology.orchestra.run.smithi039.stdout:VERSION_ID="8" 2023-01-24T14:18:19.518 INFO:teuthology.orchestra.run.smithi039.stdout:PLATFORM_ID="platform:el8" 2023-01-24T14:18:19.518 INFO:teuthology.orchestra.run.smithi039.stdout:PRETTY_NAME="CentOS Stream 8" 2023-01-24T14:18:19.518 INFO:teuthology.orchestra.run.smithi039.stdout:ANSI_COLOR="0;31" 2023-01-24T14:18:19.518 INFO:teuthology.orchestra.run.smithi039.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2023-01-24T14:18:19.519 INFO:teuthology.orchestra.run.smithi039.stdout:HOME_URL="https://centos.org/" 2023-01-24T14:18:19.519 INFO:teuthology.orchestra.run.smithi039.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-01-24T14:18:19.519 INFO:teuthology.orchestra.run.smithi039.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2023-01-24T14:18:19.519 INFO:teuthology.orchestra.run.smithi039.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-01-24T14:18:19.519 INFO:teuthology.lock.ops:Updating smithi039.front.sepia.ceph.com on lock server 2023-01-24T14:18:19.537 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-01-24T14:18:19.546 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-01-24T14:18:19.554 INFO:teuthology.task.internal:Checking for old test directory... 2023-01-24T14:18:19.554 DEBUG:teuthology.orchestra.run.smithi002:> test '!' -e /home/ubuntu/cephtest 2023-01-24T14:18:19.557 DEBUG:teuthology.orchestra.run.smithi039:> test '!' -e /home/ubuntu/cephtest 2023-01-24T14:18:19.575 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-01-24T14:18:19.582 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-01-24T14:18:19.582 DEBUG:teuthology.orchestra.run.smithi002:> test -z $(ls -A /var/lib/ceph) 2023-01-24T14:18:19.615 DEBUG:teuthology.orchestra.run.smithi039:> test -z $(ls -A /var/lib/ceph) 2023-01-24T14:18:19.640 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-01-24T14:18:19.698 INFO:teuthology.run_tasks:Running task kernel... 2023-01-24T14:18:19.712 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-01-24T14:18:19.712 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mgr.x': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-01-24T14:18:19.713 DEBUG:teuthology.orchestra.run.smithi002:> test -f /run/.containerenv -o -f /.dockerenv 2023-01-24T14:18:19.714 DEBUG:teuthology.orchestra.run.smithi039:> test -f /run/.containerenv -o -f /.dockerenv 2023-01-24T14:18:19.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-01-24T14:18:19.732 DEBUG:teuthology.orchestra.run.smithi002:> uname -r 2023-01-24T14:18:19.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-01-24T14:18:19.733 DEBUG:teuthology.orchestra.run.smithi039:> uname -r 2023-01-24T14:18:19.790 INFO:teuthology.orchestra.run.smithi002.stdout:4.18.0-408.el8.x86_64 2023-01-24T14:18:19.790 INFO:teuthology.task.kernel:Running kernel on smithi002: 4.18.0-408.el8.x86_64 2023-01-24T14:18:19.791 DEBUG:teuthology.orchestra.run.smithi002:> sudo yum install -y kernel 2023-01-24T14:18:19.792 INFO:teuthology.orchestra.run.smithi039.stdout:4.18.0-408.el8.x86_64 2023-01-24T14:18:19.792 INFO:teuthology.task.kernel:Running kernel on smithi039: 4.18.0-408.el8.x86_64 2023-01-24T14:18:19.792 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum install -y kernel 2023-01-24T14:18:23.455 INFO:teuthology.orchestra.run.smithi039.stdout:CentOS-8 - AppStream 32 MB/s | 27 MB 00:00 2023-01-24T14:18:23.482 INFO:teuthology.orchestra.run.smithi002.stdout:CentOS-8 - AppStream 32 MB/s | 27 MB 00:00 2023-01-24T14:18:29.456 INFO:teuthology.orchestra.run.smithi039.stdout:CentOS-8 - BaseOS 38 MB/s | 27 MB 00:00 2023-01-24T14:18:29.591 INFO:teuthology.orchestra.run.smithi002.stdout:CentOS-8 - BaseOS 34 MB/s | 27 MB 00:00 2023-01-24T14:18:34.004 INFO:teuthology.orchestra.run.smithi039.stdout:CentOS-8 - Extras 108 kB/s | 18 kB 00:00 2023-01-24T14:18:34.114 INFO:teuthology.orchestra.run.smithi002.stdout:CentOS-8 - Extras 134 kB/s | 18 kB 00:00 2023-01-24T14:18:34.431 INFO:teuthology.orchestra.run.smithi039.stdout:CentOS-8 - PowerTools 20 MB/s | 5.6 MB 00:00 2023-01-24T14:18:34.525 INFO:teuthology.orchestra.run.smithi002.stdout:CentOS-8 - PowerTools 21 MB/s | 5.6 MB 00:00 2023-01-24T14:18:35.723 INFO:teuthology.orchestra.run.smithi039.stdout:Copr repo for python3-asyncssh owned by ceph 29 kB/s | 3.5 kB 00:00 2023-01-24T14:18:35.807 INFO:teuthology.orchestra.run.smithi002.stdout:Copr repo for python3-asyncssh owned by ceph 33 kB/s | 3.5 kB 00:00 2023-01-24T14:18:36.848 INFO:teuthology.orchestra.run.smithi039.stdout:Extra Packages for Enterprise Linux 14 MB/s | 13 MB 00:00 2023-01-24T14:18:36.908 INFO:teuthology.orchestra.run.smithi002.stdout:Extra Packages for Enterprise Linux 14 MB/s | 13 MB 00:00 2023-01-24T14:18:40.332 INFO:teuthology.orchestra.run.smithi039.stdout:lab-extras 450 kB/s | 24 kB 00:00 2023-01-24T14:18:40.458 INFO:teuthology.orchestra.run.smithi002.stdout:lab-extras 469 kB/s | 24 kB 00:00 2023-01-24T14:18:43.630 INFO:teuthology.orchestra.run.smithi039.stdout:Package kernel-4.18.0-408.el8.x86_64 is already installed. 2023-01-24T14:18:43.631 INFO:teuthology.orchestra.run.smithi039.stdout:Package kernel-4.18.0-305.3.1.el8.x86_64 is already installed. 2023-01-24T14:18:43.789 INFO:teuthology.orchestra.run.smithi002.stdout:Package kernel-4.18.0-408.el8.x86_64 is already installed. 2023-01-24T14:18:43.790 INFO:teuthology.orchestra.run.smithi002.stdout:Package kernel-4.18.0-305.3.1.el8.x86_64 is already installed. 2023-01-24T14:18:43.815 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2023-01-24T14:18:43.816 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2023-01-24T14:18:43.816 INFO:teuthology.orchestra.run.smithi039.stdout: Package Arch Version Repository Size 2023-01-24T14:18:43.816 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2023-01-24T14:18:43.817 INFO:teuthology.orchestra.run.smithi039.stdout:Installing: 2023-01-24T14:18:43.817 INFO:teuthology.orchestra.run.smithi039.stdout: kernel x86_64 4.18.0-448.el8 CentOS-BaseOS 9.2 M 2023-01-24T14:18:43.817 INFO:teuthology.orchestra.run.smithi039.stdout: kernel-core x86_64 4.18.0-448.el8 CentOS-BaseOS 41 M 2023-01-24T14:18:43.817 INFO:teuthology.orchestra.run.smithi039.stdout: kernel-modules x86_64 4.18.0-448.el8 CentOS-BaseOS 34 M 2023-01-24T14:18:43.817 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-01-24T14:18:43.818 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction Summary 2023-01-24T14:18:43.818 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2023-01-24T14:18:43.818 INFO:teuthology.orchestra.run.smithi039.stdout:Install 3 Packages 2023-01-24T14:18:43.818 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-01-24T14:18:43.819 INFO:teuthology.orchestra.run.smithi039.stdout:Total download size: 85 M 2023-01-24T14:18:43.819 INFO:teuthology.orchestra.run.smithi039.stdout:Installed size: 94 M 2023-01-24T14:18:43.819 INFO:teuthology.orchestra.run.smithi039.stdout:Downloading Packages: 2023-01-24T14:18:43.967 INFO:teuthology.orchestra.run.smithi002.stdout:Dependencies resolved. 2023-01-24T14:18:43.968 INFO:teuthology.orchestra.run.smithi002.stdout:================================================================================ 2023-01-24T14:18:43.968 INFO:teuthology.orchestra.run.smithi002.stdout: Package Arch Version Repository Size 2023-01-24T14:18:43.968 INFO:teuthology.orchestra.run.smithi002.stdout:================================================================================ 2023-01-24T14:18:43.969 INFO:teuthology.orchestra.run.smithi002.stdout:Installing: 2023-01-24T14:18:43.969 INFO:teuthology.orchestra.run.smithi002.stdout: kernel x86_64 4.18.0-448.el8 CentOS-BaseOS 9.2 M 2023-01-24T14:18:43.969 INFO:teuthology.orchestra.run.smithi002.stdout: kernel-core x86_64 4.18.0-448.el8 CentOS-BaseOS 41 M 2023-01-24T14:18:43.969 INFO:teuthology.orchestra.run.smithi002.stdout: kernel-modules x86_64 4.18.0-448.el8 CentOS-BaseOS 34 M 2023-01-24T14:18:43.970 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-01-24T14:18:43.970 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction Summary 2023-01-24T14:18:43.970 INFO:teuthology.orchestra.run.smithi002.stdout:================================================================================ 2023-01-24T14:18:43.970 INFO:teuthology.orchestra.run.smithi002.stdout:Install 3 Packages 2023-01-24T14:18:43.970 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-01-24T14:18:43.971 INFO:teuthology.orchestra.run.smithi002.stdout:Total download size: 85 M 2023-01-24T14:18:43.971 INFO:teuthology.orchestra.run.smithi002.stdout:Installed size: 94 M 2023-01-24T14:18:43.971 INFO:teuthology.orchestra.run.smithi002.stdout:Downloading Packages: 2023-01-24T14:18:44.092 INFO:teuthology.orchestra.run.smithi039.stdout:(1/3): kernel-4.18.0-448.el8.x86_64.rpm 34 MB/s | 9.2 MB 00:00 2023-01-24T14:18:44.276 INFO:teuthology.orchestra.run.smithi002.stdout:(1/3): kernel-4.18.0-448.el8.x86_64.rpm 30 MB/s | 9.2 MB 00:00 2023-01-24T14:18:45.309 INFO:teuthology.orchestra.run.smithi039.stdout:(2/3): kernel-core-4.18.0-448.el8.x86_64.rpm 28 MB/s | 41 MB 00:01 2023-01-24T14:18:45.650 INFO:teuthology.orchestra.run.smithi039.stdout:(3/3): kernel-modules-4.18.0-448.el8.x86_64.rpm 19 MB/s | 34 MB 00:01 2023-01-24T14:18:45.653 INFO:teuthology.orchestra.run.smithi039.stdout:-------------------------------------------------------------------------------- 2023-01-24T14:18:45.653 INFO:teuthology.orchestra.run.smithi039.stdout:Total 46 MB/s | 85 MB 00:01 2023-01-24T14:18:45.710 INFO:teuthology.orchestra.run.smithi002.stdout:(2/3): kernel-core-4.18.0-448.el8.x86_64.rpm 24 MB/s | 41 MB 00:01 2023-01-24T14:18:46.193 INFO:teuthology.orchestra.run.smithi002.stdout:(3/3): kernel-modules-4.18.0-448.el8.x86_64.rpm 15 MB/s | 34 MB 00:02 2023-01-24T14:18:46.196 INFO:teuthology.orchestra.run.smithi002.stdout:-------------------------------------------------------------------------------- 2023-01-24T14:18:46.196 INFO:teuthology.orchestra.run.smithi002.stdout:Total 38 MB/s | 85 MB 00:02 2023-01-24T14:18:46.321 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction check 2023-01-24T14:18:46.724 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction check succeeded. 2023-01-24T14:18:46.724 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction test 2023-01-24T14:18:46.855 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction check 2023-01-24T14:18:47.263 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction check succeeded. 2023-01-24T14:18:47.264 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction test 2023-01-24T14:18:48.799 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction test succeeded. 2023-01-24T14:18:48.833 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction 2023-01-24T14:18:49.293 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction test succeeded. 2023-01-24T14:18:49.335 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction 2023-01-24T14:18:51.706 INFO:teuthology.orchestra.run.smithi039.stdout: Preparing : 1/1 2023-01-24T14:18:52.219 INFO:teuthology.orchestra.run.smithi002.stdout: Preparing : 1/1 2023-01-24T14:18:52.287 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : kernel-core-4.18.0-448.el8.x86_64 1/3 2023-01-24T14:18:52.784 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : kernel-core-4.18.0-448.el8.x86_64 1/3 2023-01-24T14:18:56.266 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: kernel-core-4.18.0-448.el8.x86_64 1/3 2023-01-24T14:18:56.749 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: kernel-core-4.18.0-448.el8.x86_64 1/3 2023-01-24T14:18:56.861 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : kernel-modules-4.18.0-448.el8.x86_64 2/3 2023-01-24T14:18:57.167 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : kernel-modules-4.18.0-448.el8.x86_64 2/3 2023-01-24T14:19:00.412 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: kernel-modules-4.18.0-448.el8.x86_64 2/3 2023-01-24T14:19:00.657 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : kernel-4.18.0-448.el8.x86_64 3/3 2023-01-24T14:19:00.718 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: kernel-modules-4.18.0-448.el8.x86_64 2/3 2023-01-24T14:19:00.980 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : kernel-4.18.0-448.el8.x86_64 3/3 2023-01-24T14:19:23.803 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: kernel-core-4.18.0-448.el8.x86_64 3/3 2023-01-24T14:19:23.830 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: kernel-modules-4.18.0-448.el8.x86_64 3/3 2023-01-24T14:19:24.268 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: kernel-4.18.0-448.el8.x86_64 3/3 2023-01-24T14:19:24.269 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : kernel-4.18.0-448.el8.x86_64 1/3 2023-01-24T14:19:24.269 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : kernel-core-4.18.0-448.el8.x86_64 2/3 2023-01-24T14:19:24.320 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: kernel-core-4.18.0-448.el8.x86_64 3/3 2023-01-24T14:19:24.347 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: kernel-modules-4.18.0-448.el8.x86_64 3/3 2023-01-24T14:19:24.797 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: kernel-4.18.0-448.el8.x86_64 3/3 2023-01-24T14:19:24.797 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : kernel-4.18.0-448.el8.x86_64 1/3 2023-01-24T14:19:24.797 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : kernel-core-4.18.0-448.el8.x86_64 2/3 2023-01-24T14:19:24.914 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : kernel-modules-4.18.0-448.el8.x86_64 3/3 2023-01-24T14:19:24.914 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-01-24T14:19:24.915 INFO:teuthology.orchestra.run.smithi039.stdout:Installed: 2023-01-24T14:19:24.915 INFO:teuthology.orchestra.run.smithi039.stdout: kernel-4.18.0-448.el8.x86_64 kernel-core-4.18.0-448.el8.x86_64 2023-01-24T14:19:24.915 INFO:teuthology.orchestra.run.smithi039.stdout: kernel-modules-4.18.0-448.el8.x86_64 2023-01-24T14:19:24.915 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-01-24T14:19:24.915 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2023-01-24T14:19:25.026 DEBUG:teuthology.orchestra.run.smithi039:> rpm -q kernel --last | head -n 1 2023-01-24T14:19:25.231 INFO:teuthology.orchestra.run.smithi039.stdout:kernel-4.18.0-448.el8.x86_64 Tue 24 Jan 2023 02:19:00 PM UTC 2023-01-24T14:19:25.233 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-448.el8.x86_64 2023-01-24T14:19:25.233 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-408.el8.x86_64 Expected: 4.18.0-448.el8.x86_64 2023-01-24T14:19:25.234 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-01-24T14:19:25.234 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-01-24T14:19:25.234 INFO:teuthology.task.kernel:Installing distro kernel on mgr.x... 2023-01-24T14:19:25.234 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi039.front.sepia.ceph.com, path=None, version=distro) 2023-01-24T14:19:25.235 DEBUG:teuthology.orchestra.run.smithi039:> rpm -q kernel --last | head -n 1 2023-01-24T14:19:25.413 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : kernel-modules-4.18.0-448.el8.x86_64 3/3 2023-01-24T14:19:25.413 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-01-24T14:19:25.413 INFO:teuthology.orchestra.run.smithi002.stdout:Installed: 2023-01-24T14:19:25.413 INFO:teuthology.orchestra.run.smithi002.stdout: kernel-4.18.0-448.el8.x86_64 kernel-core-4.18.0-448.el8.x86_64 2023-01-24T14:19:25.414 INFO:teuthology.orchestra.run.smithi002.stdout: kernel-modules-4.18.0-448.el8.x86_64 2023-01-24T14:19:25.414 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-01-24T14:19:25.414 INFO:teuthology.orchestra.run.smithi002.stdout:Complete! 2023-01-24T14:19:25.431 INFO:teuthology.orchestra.run.smithi039.stdout:kernel-4.18.0-448.el8.x86_64 Tue 24 Jan 2023 02:19:00 PM UTC 2023-01-24T14:19:25.433 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-448.el8.x86_64 2023-01-24T14:19:25.433 DEBUG:teuthology.orchestra.run.smithi039:> sudo rpm -qi grub2-tools 2023-01-24T14:19:25.468 INFO:teuthology.orchestra.run.smithi039.stdout:Name : grub2-tools 2023-01-24T14:19:25.468 INFO:teuthology.orchestra.run.smithi039.stdout:Epoch : 1 2023-01-24T14:19:25.468 INFO:teuthology.orchestra.run.smithi039.stdout:Version : 2.02 2023-01-24T14:19:25.469 INFO:teuthology.orchestra.run.smithi039.stdout:Release : 129.el8 2023-01-24T14:19:25.469 INFO:teuthology.orchestra.run.smithi039.stdout:Architecture: x86_64 2023-01-24T14:19:25.469 INFO:teuthology.orchestra.run.smithi039.stdout:Install Date: Sun 28 Aug 2022 01:03:50 PM UTC 2023-01-24T14:19:25.469 INFO:teuthology.orchestra.run.smithi039.stdout:Group : System Environment/Base 2023-01-24T14:19:25.469 INFO:teuthology.orchestra.run.smithi039.stdout:Size : 9265586 2023-01-24T14:19:25.470 INFO:teuthology.orchestra.run.smithi039.stdout:License : GPLv3+ 2023-01-24T14:19:25.470 INFO:teuthology.orchestra.run.smithi039.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2023-01-24T14:19:25.470 INFO:teuthology.orchestra.run.smithi039.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2023-01-24T14:19:25.470 INFO:teuthology.orchestra.run.smithi039.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2023-01-24T14:19:25.470 INFO:teuthology.orchestra.run.smithi039.stdout:Build Host : kbuilder.bsys.centos.org 2023-01-24T14:19:25.471 INFO:teuthology.orchestra.run.smithi039.stdout:Relocations : (not relocatable) 2023-01-24T14:19:25.471 INFO:teuthology.orchestra.run.smithi039.stdout:Packager : CentOS BuildSystem 2023-01-24T14:19:25.471 INFO:teuthology.orchestra.run.smithi039.stdout:Vendor : CentOS 2023-01-24T14:19:25.471 INFO:teuthology.orchestra.run.smithi039.stdout:URL : http://www.gnu.org/software/grub/ 2023-01-24T14:19:25.471 INFO:teuthology.orchestra.run.smithi039.stdout:Summary : Support tools for GRUB. 2023-01-24T14:19:25.472 INFO:teuthology.orchestra.run.smithi039.stdout:Description : 2023-01-24T14:19:25.472 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-01-24T14:19:25.472 INFO:teuthology.orchestra.run.smithi039.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-01-24T14:19:25.472 INFO:teuthology.orchestra.run.smithi039.stdout:customizable bootloader with modular architecture. It supports a rich 2023-01-24T14:19:25.472 INFO:teuthology.orchestra.run.smithi039.stdout:variety of kernel formats, file systems, computer architectures and 2023-01-24T14:19:25.473 INFO:teuthology.orchestra.run.smithi039.stdout:hardware devices. 2023-01-24T14:19:25.473 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-01-24T14:19:25.473 INFO:teuthology.orchestra.run.smithi039.stdout:This subpackage provides tools for support of all platforms. 2023-01-24T14:19:25.475 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-01-24T14:19:25.475 INFO:teuthology.task.kernel:Updating grub on smithi039 to boot 4.18.0-448.el8.x86_64 2023-01-24T14:19:25.475 DEBUG:teuthology.orchestra.run.smithi039:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-01-24T14:19:25.517 DEBUG:teuthology.orchestra.run.smithi002:> rpm -q kernel --last | head -n 1 2023-01-24T14:19:25.728 INFO:teuthology.orchestra.run.smithi002.stdout:kernel-4.18.0-448.el8.x86_64 Tue 24 Jan 2023 02:19:00 PM UTC 2023-01-24T14:19:25.730 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-448.el8.x86_64 2023-01-24T14:19:25.730 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-408.el8.x86_64 Expected: 4.18.0-448.el8.x86_64 2023-01-24T14:19:25.730 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-01-24T14:19:25.730 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-01-24T14:19:25.731 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2023-01-24T14:19:25.731 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi002.front.sepia.ceph.com, path=None, version=distro) 2023-01-24T14:19:25.731 DEBUG:teuthology.orchestra.run.smithi002:> rpm -q kernel --last | head -n 1 2023-01-24T14:19:25.926 INFO:teuthology.orchestra.run.smithi002.stdout:kernel-4.18.0-448.el8.x86_64 Tue 24 Jan 2023 02:19:00 PM UTC 2023-01-24T14:19:25.928 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-448.el8.x86_64 2023-01-24T14:19:25.928 DEBUG:teuthology.orchestra.run.smithi002:> sudo rpm -qi grub2-tools 2023-01-24T14:19:25.962 INFO:teuthology.orchestra.run.smithi002.stdout:Name : grub2-tools 2023-01-24T14:19:25.963 INFO:teuthology.orchestra.run.smithi002.stdout:Epoch : 1 2023-01-24T14:19:25.963 INFO:teuthology.orchestra.run.smithi002.stdout:Version : 2.02 2023-01-24T14:19:25.963 INFO:teuthology.orchestra.run.smithi002.stdout:Release : 129.el8 2023-01-24T14:19:25.963 INFO:teuthology.orchestra.run.smithi002.stdout:Architecture: x86_64 2023-01-24T14:19:25.963 INFO:teuthology.orchestra.run.smithi002.stdout:Install Date: Sun 28 Aug 2022 01:03:50 PM UTC 2023-01-24T14:19:25.964 INFO:teuthology.orchestra.run.smithi002.stdout:Group : System Environment/Base 2023-01-24T14:19:25.964 INFO:teuthology.orchestra.run.smithi002.stdout:Size : 9265586 2023-01-24T14:19:25.964 INFO:teuthology.orchestra.run.smithi002.stdout:License : GPLv3+ 2023-01-24T14:19:25.964 INFO:teuthology.orchestra.run.smithi002.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2023-01-24T14:19:25.964 INFO:teuthology.orchestra.run.smithi002.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2023-01-24T14:19:25.965 INFO:teuthology.orchestra.run.smithi002.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2023-01-24T14:19:25.965 INFO:teuthology.orchestra.run.smithi002.stdout:Build Host : kbuilder.bsys.centos.org 2023-01-24T14:19:25.965 INFO:teuthology.orchestra.run.smithi002.stdout:Relocations : (not relocatable) 2023-01-24T14:19:25.965 INFO:teuthology.orchestra.run.smithi002.stdout:Packager : CentOS BuildSystem 2023-01-24T14:19:25.966 INFO:teuthology.orchestra.run.smithi002.stdout:Vendor : CentOS 2023-01-24T14:19:25.966 INFO:teuthology.orchestra.run.smithi002.stdout:URL : http://www.gnu.org/software/grub/ 2023-01-24T14:19:25.966 INFO:teuthology.orchestra.run.smithi002.stdout:Summary : Support tools for GRUB. 2023-01-24T14:19:25.966 INFO:teuthology.orchestra.run.smithi002.stdout:Description : 2023-01-24T14:19:25.966 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-01-24T14:19:25.967 INFO:teuthology.orchestra.run.smithi002.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-01-24T14:19:25.967 INFO:teuthology.orchestra.run.smithi002.stdout:customizable bootloader with modular architecture. It supports a rich 2023-01-24T14:19:25.967 INFO:teuthology.orchestra.run.smithi002.stdout:variety of kernel formats, file systems, computer architectures and 2023-01-24T14:19:25.967 INFO:teuthology.orchestra.run.smithi002.stdout:hardware devices. 2023-01-24T14:19:25.967 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-01-24T14:19:25.968 INFO:teuthology.orchestra.run.smithi002.stdout:This subpackage provides tools for support of all platforms. 2023-01-24T14:19:25.969 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-01-24T14:19:25.969 INFO:teuthology.task.kernel:Updating grub on smithi002 to boot 4.18.0-448.el8.x86_64 2023-01-24T14:19:25.969 DEBUG:teuthology.orchestra.run.smithi002:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-01-24T14:19:26.810 INFO:teuthology.orchestra.run.smithi039.stderr:Generating grub configuration file ... 2023-01-24T14:19:27.257 INFO:teuthology.orchestra.run.smithi002.stderr:Generating grub configuration file ... 2023-01-24T14:19:28.927 INFO:teuthology.orchestra.run.smithi039.stderr:done 2023-01-24T14:19:28.928 DEBUG:teuthology.orchestra.run.smithi039:> mktemp 2023-01-24T14:19:28.960 INFO:teuthology.orchestra.run.smithi039.stdout:/tmp/tmp.vyBPHueucz 2023-01-24T14:19:28.960 DEBUG:teuthology.orchestra.run.smithi039:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.vyBPHueucz 2023-01-24T14:19:29.042 DEBUG:teuthology.orchestra.run.smithi039:> sudo chmod 0666 /tmp/tmp.vyBPHueucz 2023-01-24T14:19:29.191 DEBUG:teuthology.orchestra.remote:smithi039:/tmp/tmp.vyBPHueucz is 6KB 2023-01-24T14:19:29.240 DEBUG:teuthology.orchestra.run.smithi039:> rm -fr /tmp/tmp.vyBPHueucz 2023-01-24T14:19:29.258 DEBUG:teuthology.orchestra.run.smithi039:> sudo /bin/ls /boot/loader/entries || true 2023-01-24T14:19:29.282 INFO:teuthology.orchestra.run.smithi002.stderr:done 2023-01-24T14:19:29.283 DEBUG:teuthology.orchestra.run.smithi002:> mktemp 2023-01-24T14:19:29.335 INFO:teuthology.orchestra.run.smithi039.stdout:2ff282740cf84d4fa02ff60ed95fbbff-0-rescue.conf 2023-01-24T14:19:29.336 INFO:teuthology.orchestra.run.smithi039.stdout:2ff282740cf84d4fa02ff60ed95fbbff-4.18.0-305.3.1.el8.x86_64.conf 2023-01-24T14:19:29.336 INFO:teuthology.orchestra.run.smithi039.stdout:2ff282740cf84d4fa02ff60ed95fbbff-4.18.0-408.el8.x86_64.conf 2023-01-24T14:19:29.336 INFO:teuthology.orchestra.run.smithi039.stdout:2ff282740cf84d4fa02ff60ed95fbbff-4.18.0-448.el8.x86_64.conf 2023-01-24T14:19:29.337 DEBUG:teuthology.orchestra.run.smithi039:> sudo grub2-set-default 2ff282740cf84d4fa02ff60ed95fbbff-4.18.0-448.el8.x86_64 2023-01-24T14:19:29.339 INFO:teuthology.orchestra.run.smithi002.stdout:/tmp/tmp.Gnjl8unmEh 2023-01-24T14:19:29.339 DEBUG:teuthology.orchestra.run.smithi002:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.Gnjl8unmEh 2023-01-24T14:19:29.371 DEBUG:teuthology.orchestra.run.smithi002:> sudo chmod 0666 /tmp/tmp.Gnjl8unmEh 2023-01-24T14:19:29.485 DEBUG:teuthology.orchestra.run.smithi039:> sudo shutdown -r now 2023-01-24T14:19:29.516 DEBUG:teuthology.orchestra.remote:smithi002:/tmp/tmp.Gnjl8unmEh is 6KB 2023-01-24T14:19:29.564 DEBUG:teuthology.orchestra.run.smithi002:> rm -fr /tmp/tmp.Gnjl8unmEh 2023-01-24T14:19:29.582 DEBUG:teuthology.orchestra.run.smithi002:> sudo /bin/ls /boot/loader/entries || true 2023-01-24T14:19:29.648 INFO:teuthology.orchestra.run.smithi002.stdout:2ff282740cf84d4fa02ff60ed95fbbff-0-rescue.conf 2023-01-24T14:19:29.649 INFO:teuthology.orchestra.run.smithi002.stdout:2ff282740cf84d4fa02ff60ed95fbbff-4.18.0-305.3.1.el8.x86_64.conf 2023-01-24T14:19:29.649 INFO:teuthology.orchestra.run.smithi002.stdout:2ff282740cf84d4fa02ff60ed95fbbff-4.18.0-408.el8.x86_64.conf 2023-01-24T14:19:29.649 INFO:teuthology.orchestra.run.smithi002.stdout:2ff282740cf84d4fa02ff60ed95fbbff-4.18.0-448.el8.x86_64.conf 2023-01-24T14:19:29.650 DEBUG:teuthology.orchestra.run.smithi002:> sudo grub2-set-default 2ff282740cf84d4fa02ff60ed95fbbff-4.18.0-448.el8.x86_64 2023-01-24T14:19:29.839 DEBUG:teuthology.orchestra.run.smithi002:> sudo shutdown -r now 2023-01-24T14:19:59.516 INFO:teuthology.task.kernel:Checking client mgr.x for new kernel version... 2023-01-24T14:19:59.517 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi039.front.sepia.ceph.com' 2023-01-24T14:19:59.518 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi039.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-01-24T14:19:59.842 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2023-01-24T14:19:59.843 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi002.front.sepia.ceph.com' 2023-01-24T14:19:59.843 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi002.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-01-24T14:20:17.900 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.39 2023-01-24T14:20:18.408 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.2 2023-01-24T14:20:47.928 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi039.front.sepia.ceph.com' 2023-01-24T14:20:47.929 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi039.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-01-24T14:20:48.410 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi002.front.sepia.ceph.com' 2023-01-24T14:20:48.411 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi002.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-01-24T14:20:50.984 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.39 2023-01-24T14:21:04.131 DEBUG:teuthology.orchestra.run.smithi002:> true 2023-01-24T14:21:04.432 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi002.front.sepia.ceph.com' 2023-01-24T14:21:04.432 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "4.18.0-448.el8.x86_64"... 2023-01-24T14:21:04.433 DEBUG:teuthology.orchestra.run.smithi002:> uname -r 2023-01-24T14:21:04.451 INFO:teuthology.orchestra.run.smithi002.stdout:4.18.0-448.el8.x86_64 2023-01-24T14:21:04.455 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-448.el8.x86_64 vs 4.18.0-448.el8.x86_64 2023-01-24T14:21:04.455 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-01-24T14:21:04.456 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-01-24T14:21:05.456 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2023-01-24T14:21:05.457 DEBUG:teuthology.orchestra.run.smithi002:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-01-24T14:21:05.550 INFO:teuthology.orchestra.run.smithi002.stdout:ttyS1 2023-01-24T14:21:05.577 DEBUG:teuthology.parallel:result is None 2023-01-24T14:21:20.986 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi039.front.sepia.ceph.com' 2023-01-24T14:21:20.986 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi039.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-01-24T14:21:28.365 DEBUG:teuthology.orchestra.run.smithi039:> true 2023-01-24T14:21:28.729 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi039.front.sepia.ceph.com' 2023-01-24T14:21:28.729 INFO:teuthology.task.kernel:Checking kernel version of mgr.x, want "4.18.0-448.el8.x86_64"... 2023-01-24T14:21:28.730 DEBUG:teuthology.orchestra.run.smithi039:> uname -r 2023-01-24T14:21:28.748 INFO:teuthology.orchestra.run.smithi039.stdout:4.18.0-448.el8.x86_64 2023-01-24T14:21:28.748 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-448.el8.x86_64 vs 4.18.0-448.el8.x86_64 2023-01-24T14:21:28.748 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-01-24T14:21:28.749 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-01-24T14:21:29.750 INFO:teuthology.task.kernel:Enabling kdb on mgr.x... 2023-01-24T14:21:29.751 DEBUG:teuthology.orchestra.run.smithi039:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-01-24T14:21:29.845 INFO:teuthology.orchestra.run.smithi039.stdout:ttyS1 2023-01-24T14:21:29.871 DEBUG:teuthology.parallel:result is None 2023-01-24T14:21:29.872 INFO:teuthology.run_tasks:Running task internal.base... 2023-01-24T14:21:29.879 INFO:teuthology.task.internal:Creating test directory... 2023-01-24T14:21:29.880 DEBUG:teuthology.orchestra.run.smithi002:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-01-24T14:21:29.883 DEBUG:teuthology.orchestra.run.smithi039:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-01-24T14:21:29.903 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-01-24T14:21:29.910 INFO:teuthology.run_tasks:Running task internal.archive... 2023-01-24T14:21:29.916 INFO:teuthology.task.internal:Creating archive directory... 2023-01-24T14:21:29.917 DEBUG:teuthology.orchestra.run.smithi002:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-01-24T14:21:29.944 DEBUG:teuthology.orchestra.run.smithi039:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-01-24T14:21:29.976 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-01-24T14:21:29.983 INFO:teuthology.task.internal:Enabling coredump saving... 2023-01-24T14:21:29.983 DEBUG:teuthology.orchestra.run.smithi002:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-01-24T14:21:30.009 DEBUG:teuthology.orchestra.run.smithi039:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-01-24T14:21:30.048 INFO:teuthology.orchestra.run.smithi039.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-01-24T14:21:30.050 INFO:teuthology.orchestra.run.smithi002.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-01-24T14:21:30.059 INFO:teuthology.orchestra.run.smithi039.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-01-24T14:21:30.061 INFO:teuthology.orchestra.run.smithi002.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-01-24T14:21:30.063 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-01-24T14:21:30.070 INFO:teuthology.task.internal:Configuring sudo... 2023-01-24T14:21:30.070 DEBUG:teuthology.orchestra.run.smithi002:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-01-24T14:21:30.105 DEBUG:teuthology.orchestra.run.smithi039:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-01-24T14:21:30.134 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-01-24T14:21:30.144 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-01-24T14:21:30.144 DEBUG:teuthology.orchestra.run.smithi002:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-01-24T14:21:30.175 DEBUG:teuthology.orchestra.run.smithi039:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-01-24T14:21:30.193 DEBUG:teuthology.orchestra.run.smithi002:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-01-24T14:21:30.275 DEBUG:teuthology.orchestra.run.smithi002:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-01-24T14:21:30.331 DEBUG:teuthology.orchestra.run.smithi002:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-01-24T14:21:30.412 DEBUG:teuthology.orchestra.run.smithi002:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-01-24T14:21:30.459 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2023-01-24T14:21:30.459 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-01-24T14:21:30.527 DEBUG:teuthology.orchestra.run.smithi039:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-01-24T14:21:30.568 DEBUG:teuthology.orchestra.run.smithi039:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-01-24T14:21:30.618 DEBUG:teuthology.orchestra.run.smithi039:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-01-24T14:21:30.700 DEBUG:teuthology.orchestra.run.smithi039:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-01-24T14:21:30.746 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2023-01-24T14:21:30.746 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-01-24T14:21:30.817 DEBUG:teuthology.orchestra.run.smithi002:> sudo service rsyslog restart 2023-01-24T14:21:30.819 DEBUG:teuthology.orchestra.run.smithi039:> sudo service rsyslog restart 2023-01-24T14:21:30.894 INFO:teuthology.orchestra.run.smithi002.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-01-24T14:21:30.925 INFO:teuthology.orchestra.run.smithi039.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-01-24T14:21:32.639 INFO:teuthology.run_tasks:Running task internal.timer... 2023-01-24T14:21:32.647 INFO:teuthology.task.internal:Starting timer... 2023-01-24T14:21:32.647 INFO:teuthology.run_tasks:Running task pcp... 2023-01-24T14:21:32.667 INFO:teuthology.run_tasks:Running task selinux... 2023-01-24T14:21:32.675 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2023-01-24T14:21:32.676 DEBUG:teuthology.orchestra.run.smithi002:> sudo service auditd rotate 2023-01-24T14:21:32.755 INFO:teuthology.orchestra.run.smithi002.stdout:Rotating logs: 2023-01-24T14:21:32.757 DEBUG:teuthology.orchestra.run.smithi039:> sudo service auditd rotate 2023-01-24T14:21:32.830 INFO:teuthology.orchestra.run.smithi039.stdout:Rotating logs: 2023-01-24T14:21:32.832 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-01-24T14:21:32.832 DEBUG:teuthology.orchestra.run.smithi002:> /usr/sbin/getenforce 2023-01-24T14:21:32.864 INFO:teuthology.orchestra.run.smithi002.stdout:Permissive 2023-01-24T14:21:32.864 DEBUG:teuthology.orchestra.run.smithi039:> /usr/sbin/getenforce 2023-01-24T14:21:32.893 INFO:teuthology.orchestra.run.smithi039.stdout:Permissive 2023-01-24T14:21:32.893 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi002.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi039.front.sepia.ceph.com': 'permissive'} 2023-01-24T14:21:32.894 DEBUG:teuthology.orchestra.run.smithi002:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|comm="ksmtuned"\|comm="sssd"\|comm="sss_cache"\|context=system_u:system_r:NetworkManager_dispatcher_t:s0\)' 2023-01-24T14:21:32.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-01-24T14:21:32.934 DEBUG:teuthology.orchestra.run.smithi039:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|comm="ksmtuned"\|comm="sssd"\|comm="sss_cache"\|context=system_u:system_r:NetworkManager_dispatcher_t:s0\)' 2023-01-24T14:21:32.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-01-24T14:21:32.965 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-01-24T14:21:32.965 DEBUG:teuthology.orchestra.run.smithi002:> sudo /usr/sbin/setenforce permissive 2023-01-24T14:21:33.003 DEBUG:teuthology.orchestra.run.smithi039:> sudo /usr/sbin/setenforce permissive 2023-01-24T14:21:33.043 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-01-24T14:21:33.054 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2023-01-24T14:21:33.060 INFO:teuthology.repo_utils:Fetching main from origin 2023-01-24T14:21:33.170 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-01-24T14:21:33.186 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-01-24T14:21:33.188 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi002.front.sepia.ceph.com,smithi039.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-01-24T14:28:09.632 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi002.front.sepia.ceph.com'), Remote(name='ubuntu@smithi039.front.sepia.ceph.com')] 2023-01-24T14:28:09.634 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi002.front.sepia.ceph.com' 2023-01-24T14:28:09.635 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi002.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-01-24T14:28:09.706 DEBUG:teuthology.orchestra.run.smithi002:> true 2023-01-24T14:28:09.780 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi039.front.sepia.ceph.com' 2023-01-24T14:28:09.781 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi039.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-01-24T14:28:09.847 DEBUG:teuthology.orchestra.run.smithi039:> true 2023-01-24T14:28:09.918 INFO:teuthology.run_tasks:Running task clock... 2023-01-24T14:28:09.931 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-01-24T14:28:09.932 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-01-24T14:28:09.933 DEBUG:teuthology.orchestra.run.smithi002:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-01-24T14:28:09.936 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-01-24T14:28:09.936 DEBUG:teuthology.orchestra.run.smithi039:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-01-24T14:28:09.968 INFO:teuthology.orchestra.run.smithi002.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-01-24T14:28:09.982 INFO:teuthology.orchestra.run.smithi002.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-01-24T14:28:09.992 INFO:teuthology.orchestra.run.smithi039.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-01-24T14:28:10.007 INFO:teuthology.orchestra.run.smithi039.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-01-24T14:28:10.052 INFO:teuthology.orchestra.run.smithi002.stderr:sudo: ntpd: command not found 2023-01-24T14:28:10.064 INFO:teuthology.orchestra.run.smithi002.stdout:506 Cannot talk to daemon 2023-01-24T14:28:10.079 INFO:teuthology.orchestra.run.smithi002.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-01-24T14:28:10.093 INFO:teuthology.orchestra.run.smithi002.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-01-24T14:28:10.159 INFO:teuthology.orchestra.run.smithi002.stderr:bash: ntpq: command not found 2023-01-24T14:28:10.162 INFO:teuthology.orchestra.run.smithi002.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-01-24T14:28:10.162 INFO:teuthology.orchestra.run.smithi002.stdout:=============================================================================== 2023-01-24T14:28:10.162 INFO:teuthology.orchestra.run.smithi002.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-01-24T14:28:10.163 INFO:teuthology.orchestra.run.smithi002.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-01-24T14:28:10.163 INFO:teuthology.orchestra.run.smithi002.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-01-24T14:28:10.163 INFO:teuthology.orchestra.run.smithi002.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-01-24T14:28:10.209 INFO:teuthology.orchestra.run.smithi039.stderr:sudo: ntpd: command not found 2023-01-24T14:28:10.220 INFO:teuthology.orchestra.run.smithi039.stdout:506 Cannot talk to daemon 2023-01-24T14:28:10.235 INFO:teuthology.orchestra.run.smithi039.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-01-24T14:28:10.249 INFO:teuthology.orchestra.run.smithi039.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-01-24T14:28:10.322 INFO:teuthology.orchestra.run.smithi039.stderr:bash: ntpq: command not found 2023-01-24T14:28:10.326 INFO:teuthology.orchestra.run.smithi039.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-01-24T14:28:10.326 INFO:teuthology.orchestra.run.smithi039.stdout:=============================================================================== 2023-01-24T14:28:10.326 INFO:teuthology.orchestra.run.smithi039.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-01-24T14:28:10.326 INFO:teuthology.orchestra.run.smithi039.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-01-24T14:28:10.326 INFO:teuthology.orchestra.run.smithi039.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-01-24T14:28:10.327 INFO:teuthology.orchestra.run.smithi039.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-01-24T14:28:10.328 INFO:teuthology.run_tasks:Running task install... 2023-01-24T14:28:10.340 DEBUG:teuthology.task.install:project ceph 2023-01-24T14:28:10.340 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'crimson', 'sha1': '1a671c198559f54389cd5afa22085a0c7b29a1ef'}} 2023-01-24T14:28:10.341 DEBUG:teuthology.task.install:config {'flavor': 'crimson', 'sha1': '1a671c198559f54389cd5afa22085a0c7b29a1ef'} 2023-01-24T14:28:10.341 INFO:teuthology.task.install:Using flavor: crimson 2023-01-24T14:28:10.347 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-01-24T14:28:10.347 INFO:teuthology.task.install:extra packages: [] 2023-01-24T14:28:10.347 DEBUG:teuthology.orchestra.run.smithi002:> sudo lsb_release -is 2023-01-24T14:28:10.386 INFO:teuthology.orchestra.run.smithi002.stdout:CentOSStream 2023-01-24T14:28:10.387 DEBUG:teuthology.misc:System to be installed: CentOSStream 2023-01-24T14:28:10.387 DEBUG:teuthology.orchestra.run.smithi039:> sudo lsb_release -is 2023-01-24T14:28:10.388 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'flavor': 'crimson', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '1a671c198559f54389cd5afa22085a0c7b29a1ef', 'tag': None, 'wait_for_package': False} 2023-01-24T14:28:10.389 DEBUG:teuthology.orchestra.run.smithi002:> sudo lsb_release -is 2023-01-24T14:28:10.427 INFO:teuthology.orchestra.run.smithi039.stdout:CentOSStream 2023-01-24T14:28:10.428 DEBUG:teuthology.misc:System to be installed: CentOSStream 2023-01-24T14:28:10.428 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'flavor': 'crimson', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '1a671c198559f54389cd5afa22085a0c7b29a1ef', 'tag': None, 'wait_for_package': False} 2023-01-24T14:28:10.429 DEBUG:teuthology.orchestra.run.smithi039:> sudo lsb_release -is 2023-01-24T14:28:10.467 INFO:teuthology.orchestra.run.smithi002.stdout:CentOSStream 2023-01-24T14:28:10.468 DEBUG:teuthology.misc:System to be installed: CentOSStream 2023-01-24T14:28:10.468 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F8%2Fx86_64&sha1=1a671c198559f54389cd5afa22085a0c7b29a1ef 2023-01-24T14:28:10.508 INFO:teuthology.orchestra.run.smithi039.stdout:CentOSStream 2023-01-24T14:28:10.509 DEBUG:teuthology.misc:System to be installed: CentOSStream 2023-01-24T14:28:10.510 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F8%2Fx86_64&sha1=1a671c198559f54389cd5afa22085a0c7b29a1ef 2023-01-24T14:28:10.617 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/main/1a671c198559f54389cd5afa22085a0c7b29a1ef/centos/8/flavors/crimson/ 2023-01-24T14:28:10.618 INFO:teuthology.task.install.rpm:Package version is 18.0.0-1954.g1a671c19 2023-01-24T14:28:10.663 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/main/1a671c198559f54389cd5afa22085a0c7b29a1ef/centos/8/flavors/crimson/ 2023-01-24T14:28:10.664 INFO:teuthology.task.install.rpm:Package version is 18.0.0-1954.g1a671c19 2023-01-24T14:28:10.743 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/main/1a671c198559f54389cd5afa22085a0c7b29a1ef/centos/8/flavors/crimson/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/main/1a671c198559f54389cd5afa22085a0c7b29a1ef/centos/8/flavors/crimson/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/main/1a671c198559f54389cd5afa22085a0c7b29a1ef/centos/8/flavors/crimson/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-01-24T14:28:10.744 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2023-01-24T14:28:10.745 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-01-24T14:28:10.774 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2023-01-24T14:28:10.775 DEBUG:teuthology.orchestra.run.smithi002:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/1a671c198559f54389cd5afa22085a0c7b29a1ef/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-01-24T14:28:10.791 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/main/1a671c198559f54389cd5afa22085a0c7b29a1ef/centos/8/flavors/crimson/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/main/1a671c198559f54389cd5afa22085a0c7b29a1ef/centos/8/flavors/crimson/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/main/1a671c198559f54389cd5afa22085a0c7b29a1ef/centos/8/flavors/crimson/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-01-24T14:28:10.792 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2023-01-24T14:28:10.792 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-01-24T14:28:10.821 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2023-01-24T14:28:10.822 DEBUG:teuthology.orchestra.run.smithi039:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/1a671c198559f54389cd5afa22085a0c7b29a1ef/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-01-24T14:28:10.845 DEBUG:teuthology.orchestra.run.smithi002:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-01-24T14:28:10.892 DEBUG:teuthology.orchestra.run.smithi039:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-01-24T14:28:10.924 DEBUG:teuthology.orchestra.run.smithi002:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-01-24T14:28:10.971 DEBUG:teuthology.orchestra.run.smithi039:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-01-24T14:28:10.992 INFO:teuthology.orchestra.run.smithi002.stdout:check_obsoletes = 1 2023-01-24T14:28:10.993 DEBUG:teuthology.orchestra.run.smithi002:> sudo yum clean all 2023-01-24T14:28:11.039 INFO:teuthology.orchestra.run.smithi039.stdout:check_obsoletes = 1 2023-01-24T14:28:11.041 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum clean all 2023-01-24T14:28:11.253 INFO:teuthology.orchestra.run.smithi002.stdout:61 files removed 2023-01-24T14:28:11.282 DEBUG:teuthology.orchestra.run.smithi002:> sudo yum -y install ceph-radosgw 2023-01-24T14:28:11.301 INFO:teuthology.orchestra.run.smithi039.stdout:61 files removed 2023-01-24T14:28:11.330 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum -y install ceph-radosgw 2023-01-24T14:28:12.005 INFO:teuthology.orchestra.run.smithi002.stdout:ceph packages for x86_64 283 kB/s | 83 kB 00:00 2023-01-24T14:28:12.045 INFO:teuthology.orchestra.run.smithi039.stdout:ceph packages for x86_64 295 kB/s | 83 kB 00:00 2023-01-24T14:28:12.297 INFO:teuthology.orchestra.run.smithi002.stdout:ceph noarch packages 37 kB/s | 9.3 kB 00:00 2023-01-24T14:28:12.337 INFO:teuthology.orchestra.run.smithi039.stdout:ceph noarch packages 37 kB/s | 9.3 kB 00:00 2023-01-24T14:28:12.639 INFO:teuthology.orchestra.run.smithi002.stdout:ceph source packages 5.1 kB/s | 1.6 kB 00:00 2023-01-24T14:28:12.654 INFO:teuthology.orchestra.run.smithi039.stdout:ceph source packages 5.5 kB/s | 1.6 kB 00:00 2023-01-24T14:28:13.400 INFO:teuthology.orchestra.run.smithi002.stdout:CentOS-8 - AppStream 38 MB/s | 27 MB 00:00 2023-01-24T14:28:13.499 INFO:teuthology.orchestra.run.smithi039.stdout:CentOS-8 - AppStream 34 MB/s | 27 MB 00:00 2023-01-24T14:28:19.476 INFO:teuthology.orchestra.run.smithi002.stdout:CentOS-8 - BaseOS 38 MB/s | 27 MB 00:00 2023-01-24T14:28:19.724 INFO:teuthology.orchestra.run.smithi039.stdout:CentOS-8 - BaseOS 32 MB/s | 27 MB 00:00 2023-01-24T14:28:24.025 INFO:teuthology.orchestra.run.smithi002.stdout:CentOS-8 - Extras 134 kB/s | 18 kB 00:00 2023-01-24T14:28:24.248 INFO:teuthology.orchestra.run.smithi039.stdout:CentOS-8 - Extras 146 kB/s | 18 kB 00:00 2023-01-24T14:28:24.443 INFO:teuthology.orchestra.run.smithi002.stdout:CentOS-8 - PowerTools 21 MB/s | 5.6 MB 00:00 2023-01-24T14:28:24.659 INFO:teuthology.orchestra.run.smithi039.stdout:CentOS-8 - PowerTools 21 MB/s | 5.6 MB 00:00 2023-01-24T14:28:25.751 INFO:teuthology.orchestra.run.smithi002.stdout:Copr repo for python3-asyncssh owned by ceph 28 kB/s | 3.5 kB 00:00 2023-01-24T14:28:25.941 INFO:teuthology.orchestra.run.smithi039.stdout:Copr repo for python3-asyncssh owned by ceph 28 kB/s | 3.5 kB 00:00 2023-01-24T14:28:27.285 INFO:teuthology.orchestra.run.smithi002.stdout:Extra Packages for Enterprise Linux 9.6 MB/s | 13 MB 00:01 2023-01-24T14:28:27.483 INFO:teuthology.orchestra.run.smithi039.stdout:Extra Packages for Enterprise Linux 9.5 MB/s | 13 MB 00:01 2023-01-24T14:28:30.743 INFO:teuthology.orchestra.run.smithi002.stdout:lab-extras 528 kB/s | 24 kB 00:00 2023-01-24T14:28:30.967 INFO:teuthology.orchestra.run.smithi039.stdout:lab-extras 528 kB/s | 24 kB 00:00 2023-01-24T14:28:34.395 INFO:teuthology.orchestra.run.smithi002.stdout:Dependencies resolved. 2023-01-24T14:28:34.397 INFO:teuthology.orchestra.run.smithi002.stdout:================================================================================ 2023-01-24T14:28:34.398 INFO:teuthology.orchestra.run.smithi002.stdout: Package Arch Version Repository Size 2023-01-24T14:28:34.398 INFO:teuthology.orchestra.run.smithi002.stdout:================================================================================ 2023-01-24T14:28:34.398 INFO:teuthology.orchestra.run.smithi002.stdout:Installing: 2023-01-24T14:28:34.398 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-radosgw x86_64 2:18.0.0-1954.g1a671c19.el8 ceph 14 M 2023-01-24T14:28:34.399 INFO:teuthology.orchestra.run.smithi002.stdout:Upgrading: 2023-01-24T14:28:34.399 INFO:teuthology.orchestra.run.smithi002.stdout: librados2 x86_64 2:18.0.0-1954.g1a671c19.el8 ceph 3.3 M 2023-01-24T14:28:34.399 INFO:teuthology.orchestra.run.smithi002.stdout: librbd1 x86_64 2:18.0.0-1954.g1a671c19.el8 ceph 3.5 M 2023-01-24T14:28:34.399 INFO:teuthology.orchestra.run.smithi002.stdout:Installing dependencies: 2023-01-24T14:28:34.399 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-base x86_64 2:18.0.0-1954.g1a671c19.el8 ceph 6.3 M 2023-01-24T14:28:34.400 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-common x86_64 2:18.0.0-1954.g1a671c19.el8 ceph 27 M 2023-01-24T14:28:34.400 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-selinux x86_64 2:18.0.0-1954.g1a671c19.el8 ceph 24 k 2023-01-24T14:28:34.400 INFO:teuthology.orchestra.run.smithi002.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-01-24T14:28:34.400 INFO:teuthology.orchestra.run.smithi002.stdout: libcephfs2 x86_64 2:18.0.0-1954.g1a671c19.el8 ceph 741 k 2023-01-24T14:28:34.400 INFO:teuthology.orchestra.run.smithi002.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-01-24T14:28:34.401 INFO:teuthology.orchestra.run.smithi002.stdout: librabbitmq x86_64 0.9.0-3.el8 CentOS-BaseOS 47 k 2023-01-24T14:28:34.401 INFO:teuthology.orchestra.run.smithi002.stdout: libradosstriper1 x86_64 2:18.0.0-1954.g1a671c19.el8 ceph 461 k 2023-01-24T14:28:34.401 INFO:teuthology.orchestra.run.smithi002.stdout: librdkafka x86_64 0.11.4-3.el8 CentOS-AppStream 354 k 2023-01-24T14:28:34.401 INFO:teuthology.orchestra.run.smithi002.stdout: librgw2 x86_64 2:18.0.0-1954.g1a671c19.el8 ceph 7.2 M 2023-01-24T14:28:34.401 INFO:teuthology.orchestra.run.smithi002.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-01-24T14:28:34.402 INFO:teuthology.orchestra.run.smithi002.stdout: lttng-ust x86_64 2.8.1-11.el8 CentOS-AppStream 259 k 2023-01-24T14:28:34.402 INFO:teuthology.orchestra.run.smithi002.stdout: python3-ceph-argparse 2023-01-24T14:28:34.402 INFO:teuthology.orchestra.run.smithi002.stdout: x86_64 2:18.0.0-1954.g1a671c19.el8 ceph 45 k 2023-01-24T14:28:34.402 INFO:teuthology.orchestra.run.smithi002.stdout: python3-ceph-common x86_64 2:18.0.0-1954.g1a671c19.el8 ceph 115 k 2023-01-24T14:28:34.402 INFO:teuthology.orchestra.run.smithi002.stdout: python3-cephfs x86_64 2:18.0.0-1954.g1a671c19.el8 ceph 190 k 2023-01-24T14:28:34.403 INFO:teuthology.orchestra.run.smithi002.stdout: python3-prettytable noarch 0.7.2-14.el8 CentOS-AppStream 44 k 2023-01-24T14:28:34.403 INFO:teuthology.orchestra.run.smithi002.stdout: python3-rados x86_64 2:18.0.0-1954.g1a671c19.el8 ceph 371 k 2023-01-24T14:28:34.403 INFO:teuthology.orchestra.run.smithi002.stdout: python3-rbd x86_64 2:18.0.0-1954.g1a671c19.el8 ceph 364 k 2023-01-24T14:28:34.403 INFO:teuthology.orchestra.run.smithi002.stdout: python3-rgw x86_64 2:18.0.0-1954.g1a671c19.el8 ceph 106 k 2023-01-24T14:28:34.403 INFO:teuthology.orchestra.run.smithi002.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-01-24T14:28:34.404 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-01-24T14:28:34.404 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction Summary 2023-01-24T14:28:34.404 INFO:teuthology.orchestra.run.smithi002.stdout:================================================================================ 2023-01-24T14:28:34.404 INFO:teuthology.orchestra.run.smithi002.stdout:Install 21 Packages 2023-01-24T14:28:34.404 INFO:teuthology.orchestra.run.smithi002.stdout:Upgrade 2 Packages 2023-01-24T14:28:34.405 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-01-24T14:28:34.405 INFO:teuthology.orchestra.run.smithi002.stdout:Total download size: 67 M 2023-01-24T14:28:34.405 INFO:teuthology.orchestra.run.smithi002.stdout:Downloading Packages: 2023-01-24T14:28:34.619 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2023-01-24T14:28:34.622 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2023-01-24T14:28:34.622 INFO:teuthology.orchestra.run.smithi039.stdout: Package Arch Version Repository Size 2023-01-24T14:28:34.622 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2023-01-24T14:28:34.623 INFO:teuthology.orchestra.run.smithi039.stdout:Installing: 2023-01-24T14:28:34.623 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-radosgw x86_64 2:18.0.0-1954.g1a671c19.el8 ceph 14 M 2023-01-24T14:28:34.623 INFO:teuthology.orchestra.run.smithi039.stdout:Upgrading: 2023-01-24T14:28:34.623 INFO:teuthology.orchestra.run.smithi039.stdout: librados2 x86_64 2:18.0.0-1954.g1a671c19.el8 ceph 3.3 M 2023-01-24T14:28:34.623 INFO:teuthology.orchestra.run.smithi039.stdout: librbd1 x86_64 2:18.0.0-1954.g1a671c19.el8 ceph 3.5 M 2023-01-24T14:28:34.624 INFO:teuthology.orchestra.run.smithi039.stdout:Installing dependencies: 2023-01-24T14:28:34.624 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-base x86_64 2:18.0.0-1954.g1a671c19.el8 ceph 6.3 M 2023-01-24T14:28:34.624 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-common x86_64 2:18.0.0-1954.g1a671c19.el8 ceph 27 M 2023-01-24T14:28:34.624 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-selinux x86_64 2:18.0.0-1954.g1a671c19.el8 ceph 24 k 2023-01-24T14:28:34.625 INFO:teuthology.orchestra.run.smithi039.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-01-24T14:28:34.625 INFO:teuthology.orchestra.run.smithi039.stdout: libcephfs2 x86_64 2:18.0.0-1954.g1a671c19.el8 ceph 741 k 2023-01-24T14:28:34.625 INFO:teuthology.orchestra.run.smithi039.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-01-24T14:28:34.625 INFO:teuthology.orchestra.run.smithi039.stdout: librabbitmq x86_64 0.9.0-3.el8 CentOS-BaseOS 47 k 2023-01-24T14:28:34.625 INFO:teuthology.orchestra.run.smithi039.stdout: libradosstriper1 x86_64 2:18.0.0-1954.g1a671c19.el8 ceph 461 k 2023-01-24T14:28:34.625 INFO:teuthology.orchestra.run.smithi039.stdout: librdkafka x86_64 0.11.4-3.el8 CentOS-AppStream 354 k 2023-01-24T14:28:34.626 INFO:teuthology.orchestra.run.smithi039.stdout: librgw2 x86_64 2:18.0.0-1954.g1a671c19.el8 ceph 7.2 M 2023-01-24T14:28:34.626 INFO:teuthology.orchestra.run.smithi039.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-01-24T14:28:34.626 INFO:teuthology.orchestra.run.smithi039.stdout: lttng-ust x86_64 2.8.1-11.el8 CentOS-AppStream 259 k 2023-01-24T14:28:34.626 INFO:teuthology.orchestra.run.smithi039.stdout: python3-ceph-argparse 2023-01-24T14:28:34.626 INFO:teuthology.orchestra.run.smithi039.stdout: x86_64 2:18.0.0-1954.g1a671c19.el8 ceph 45 k 2023-01-24T14:28:34.627 INFO:teuthology.orchestra.run.smithi039.stdout: python3-ceph-common x86_64 2:18.0.0-1954.g1a671c19.el8 ceph 115 k 2023-01-24T14:28:34.627 INFO:teuthology.orchestra.run.smithi039.stdout: python3-cephfs x86_64 2:18.0.0-1954.g1a671c19.el8 ceph 190 k 2023-01-24T14:28:34.627 INFO:teuthology.orchestra.run.smithi039.stdout: python3-prettytable noarch 0.7.2-14.el8 CentOS-AppStream 44 k 2023-01-24T14:28:34.627 INFO:teuthology.orchestra.run.smithi039.stdout: python3-rados x86_64 2:18.0.0-1954.g1a671c19.el8 ceph 371 k 2023-01-24T14:28:34.628 INFO:teuthology.orchestra.run.smithi039.stdout: python3-rbd x86_64 2:18.0.0-1954.g1a671c19.el8 ceph 364 k 2023-01-24T14:28:34.628 INFO:teuthology.orchestra.run.smithi039.stdout: python3-rgw x86_64 2:18.0.0-1954.g1a671c19.el8 ceph 106 k 2023-01-24T14:28:34.628 INFO:teuthology.orchestra.run.smithi039.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-01-24T14:28:34.628 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-01-24T14:28:34.628 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction Summary 2023-01-24T14:28:34.628 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2023-01-24T14:28:34.629 INFO:teuthology.orchestra.run.smithi039.stdout:Install 21 Packages 2023-01-24T14:28:34.629 INFO:teuthology.orchestra.run.smithi039.stdout:Upgrade 2 Packages 2023-01-24T14:28:34.629 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-01-24T14:28:34.630 INFO:teuthology.orchestra.run.smithi039.stdout:Total download size: 67 M 2023-01-24T14:28:34.630 INFO:teuthology.orchestra.run.smithi039.stdout:Downloading Packages: 2023-01-24T14:28:35.453 INFO:teuthology.orchestra.run.smithi002.stdout:(1/23): ceph-base-18.0.0-1954.g1a671c19.el8.x86 6.0 MB/s | 6.3 MB 00:01 2023-01-24T14:28:35.555 INFO:teuthology.orchestra.run.smithi002.stdout:(2/23): ceph-selinux-18.0.0-1954.g1a671c19.el8. 238 kB/s | 24 kB 00:00 2023-01-24T14:28:36.063 INFO:teuthology.orchestra.run.smithi002.stdout:(3/23): libcephfs2-18.0.0-1954.g1a671c19.el8.x8 1.4 MB/s | 741 kB 00:00 2023-01-24T14:28:36.214 INFO:teuthology.orchestra.run.smithi002.stdout:(4/23): ceph-radosgw-18.0.0-1954.g1a671c19.el8. 7.9 MB/s | 14 MB 00:01 2023-01-24T14:28:36.247 INFO:teuthology.orchestra.run.smithi002.stdout:(5/23): libradosstriper1-18.0.0-1954.g1a671c19. 2.5 MB/s | 461 kB 00:00 2023-01-24T14:28:36.414 INFO:teuthology.orchestra.run.smithi002.stdout:(6/23): python3-ceph-argparse-18.0.0-1954.g1a67 267 kB/s | 45 kB 00:00 2023-01-24T14:28:36.540 INFO:teuthology.orchestra.run.smithi002.stdout:(7/23): python3-ceph-common-18.0.0-1954.g1a671c 921 kB/s | 115 kB 00:00 2023-01-24T14:28:36.648 INFO:teuthology.orchestra.run.smithi002.stdout:(8/23): python3-cephfs-18.0.0-1954.g1a671c19.el 1.7 MB/s | 190 kB 00:00 2023-01-24T14:28:36.748 INFO:teuthology.orchestra.run.smithi002.stdout:(9/23): python3-rados-18.0.0-1954.g1a671c19.el8 3.6 MB/s | 371 kB 00:00 2023-01-24T14:28:37.032 INFO:teuthology.orchestra.run.smithi002.stdout:(10/23): ceph-common-18.0.0-1954.g1a671c19.el8. 10 MB/s | 27 MB 00:02 2023-01-24T14:28:37.057 INFO:teuthology.orchestra.run.smithi002.stdout:(11/23): python3-rbd-18.0.0-1954.g1a671c19.el8. 1.2 MB/s | 364 kB 00:00 2023-01-24T14:28:37.091 INFO:teuthology.orchestra.run.smithi002.stdout:(12/23): python3-rgw-18.0.0-1954.g1a671c19.el8. 1.8 MB/s | 106 kB 00:00 2023-01-24T14:28:37.124 INFO:teuthology.orchestra.run.smithi002.stdout:(13/23): librdkafka-0.11.4-3.el8.x86_64.rpm 5.2 MB/s | 354 kB 00:00 2023-01-24T14:28:37.150 INFO:teuthology.orchestra.run.smithi002.stdout:(14/23): python3-prettytable-0.7.2-14.el8.noarc 1.9 MB/s | 44 kB 00:00 2023-01-24T14:28:37.175 INFO:teuthology.orchestra.run.smithi002.stdout:(15/23): librabbitmq-0.9.0-3.el8.x86_64.rpm 1.8 MB/s | 47 kB 00:00 2023-01-24T14:28:37.200 INFO:teuthology.orchestra.run.smithi002.stdout:(16/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 2.4 MB/s | 259 kB 00:00 2023-01-24T14:28:37.292 INFO:teuthology.orchestra.run.smithi002.stdout:(17/23): librgw2-18.0.0-1954.g1a671c19.el8.x86_ 6.6 MB/s | 7.2 MB 00:01 2023-01-24T14:28:37.394 INFO:teuthology.orchestra.run.smithi039.stdout:(1/23): ceph-base-18.0.0-1954.g1a671c19.el8.x86 2.3 MB/s | 6.3 MB 00:02 2023-01-24T14:28:37.426 INFO:teuthology.orchestra.run.smithi002.stdout:(18/23): liboath-2.6.2-3.el8.x86_64.rpm 261 kB/s | 59 kB 00:00 2023-01-24T14:28:37.443 INFO:teuthology.orchestra.run.smithi039.stdout:(2/23): ceph-selinux-18.0.0-1954.g1a671c19.el8. 488 kB/s | 24 kB 00:00 2023-01-24T14:28:37.459 INFO:teuthology.orchestra.run.smithi002.stdout:(19/23): libunwind-1.3.1-3.el8.x86_64.rpm 450 kB/s | 75 kB 00:00 2023-01-24T14:28:37.543 INFO:teuthology.orchestra.run.smithi002.stdout:(20/23): gperftools-libs-2.7-9.el8.x86_64.rpm 833 kB/s | 306 kB 00:00 2023-01-24T14:28:37.644 INFO:teuthology.orchestra.run.smithi039.stdout:(3/23): libcephfs2-18.0.0-1954.g1a671c19.el8.x8 3.6 MB/s | 741 kB 00:00 2023-01-24T14:28:37.860 INFO:teuthology.orchestra.run.smithi039.stdout:(4/23): libradosstriper1-18.0.0-1954.g1a671c19. 2.1 MB/s | 461 kB 00:00 2023-01-24T14:28:37.885 INFO:teuthology.orchestra.run.smithi002.stdout:(21/23): librados2-18.0.0-1954.g1a671c19.el8.x8 7.8 MB/s | 3.3 MB 00:00 2023-01-24T14:28:37.944 INFO:teuthology.orchestra.run.smithi002.stdout:(22/23): librbd1-18.0.0-1954.g1a671c19.el8.x86_ 8.9 MB/s | 3.5 MB 00:00 2023-01-24T14:28:38.310 INFO:teuthology.orchestra.run.smithi002.stdout:(23/23): thrift-0.13.0-2.el8.x86_64.rpm 2.0 MB/s | 1.7 MB 00:00 2023-01-24T14:28:38.314 INFO:teuthology.orchestra.run.smithi002.stdout:-------------------------------------------------------------------------------- 2023-01-24T14:28:38.315 INFO:teuthology.orchestra.run.smithi002.stdout:Total 17 MB/s | 67 MB 00:03 2023-01-24T14:28:38.357 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction check 2023-01-24T14:28:38.406 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction check succeeded. 2023-01-24T14:28:38.406 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction test 2023-01-24T14:28:39.073 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction test succeeded. 2023-01-24T14:28:39.076 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction 2023-01-24T14:28:39.844 INFO:teuthology.orchestra.run.smithi039.stdout:(5/23): librgw2-18.0.0-1954.g1a671c19.el8.x86_6 3.6 MB/s | 7.2 MB 00:01 2023-01-24T14:28:39.944 INFO:teuthology.orchestra.run.smithi039.stdout:(6/23): python3-ceph-argparse-18.0.0-1954.g1a67 447 kB/s | 45 kB 00:00 2023-01-24T14:28:40.036 INFO:teuthology.orchestra.run.smithi039.stdout:(7/23): python3-ceph-common-18.0.0-1954.g1a671c 1.2 MB/s | 115 kB 00:00 2023-01-24T14:28:40.178 INFO:teuthology.orchestra.run.smithi039.stdout:(8/23): python3-cephfs-18.0.0-1954.g1a671c19.el 1.3 MB/s | 190 kB 00:00 2023-01-24T14:28:40.212 INFO:teuthology.orchestra.run.smithi002.stdout: Preparing : 1/1 2023-01-24T14:28:40.505 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : liboath-2.6.2-3.el8.x86_64 1/25 2023-01-24T14:28:40.612 INFO:teuthology.orchestra.run.smithi039.stdout:(9/23): python3-rados-18.0.0-1954.g1a671c19.el8 856 kB/s | 371 kB 00:00 2023-01-24T14:28:40.675 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 2/25 2023-01-24T14:28:41.072 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 2/25 2023-01-24T14:28:41.275 INFO:teuthology.orchestra.run.smithi002.stdout: Upgrading : librados2-2:18.0.0-1954.g1a671c19.el8.x86_64 3/25 2023-01-24T14:28:41.337 INFO:teuthology.orchestra.run.smithi039.stdout:(10/23): ceph-radosgw-18.0.0-1954.g1a671c19.el8 2.1 MB/s | 14 MB 00:06 2023-01-24T14:28:41.362 INFO:teuthology.orchestra.run.smithi039.stdout:(11/23): python3-rbd-18.0.0-1954.g1a671c19.el8. 486 kB/s | 364 kB 00:00 2023-01-24T14:28:41.386 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: librados2-2:18.0.0-1954.g1a671c19.el8.x86_64 3/25 2023-01-24T14:28:41.421 INFO:teuthology.orchestra.run.smithi039.stdout:(12/23): librdkafka-0.11.4-3.el8.x86_64.rpm 5.9 MB/s | 354 kB 00:00 2023-01-24T14:28:41.446 INFO:teuthology.orchestra.run.smithi039.stdout:(13/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 10 MB/s | 259 kB 00:00 2023-01-24T14:28:41.463 INFO:teuthology.orchestra.run.smithi039.stdout:(14/23): python3-prettytable-0.7.2-14.el8.noarc 2.6 MB/s | 44 kB 00:00 2023-01-24T14:28:41.480 INFO:teuthology.orchestra.run.smithi039.stdout:(15/23): librabbitmq-0.9.0-3.el8.x86_64.rpm 2.8 MB/s | 47 kB 00:00 2023-01-24T14:28:41.505 INFO:teuthology.orchestra.run.smithi039.stdout:(16/23): python3-rgw-18.0.0-1954.g1a671c19.el8. 635 kB/s | 106 kB 00:00 2023-01-24T14:28:41.631 INFO:teuthology.orchestra.run.smithi039.stdout:(17/23): liboath-2.6.2-3.el8.x86_64.rpm 470 kB/s | 59 kB 00:00 2023-01-24T14:28:41.660 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-rados-2:18.0.0-1954.g1a671c19.el8.x86_64 4/25 2023-01-24T14:28:41.715 INFO:teuthology.orchestra.run.smithi039.stdout:(18/23): libunwind-1.3.1-3.el8.x86_64.rpm 901 kB/s | 75 kB 00:00 2023-01-24T14:28:41.740 INFO:teuthology.orchestra.run.smithi039.stdout:(19/23): gperftools-libs-2.7-9.el8.x86_64.rpm 1.2 MB/s | 306 kB 00:00 2023-01-24T14:28:41.825 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : libcephfs2-2:18.0.0-1954.g1a671c19.el8.x86_64 5/25 2023-01-24T14:28:42.247 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: libcephfs2-2:18.0.0-1954.g1a671c19.el8.x86_64 5/25 2023-01-24T14:28:42.315 INFO:teuthology.orchestra.run.smithi039.stdout:(20/23): thrift-0.13.0-2.el8.x86_64.rpm 2.9 MB/s | 1.7 MB 00:00 2023-01-24T14:28:42.416 INFO:teuthology.orchestra.run.smithi002.stdout: Upgrading : librbd1-2:18.0.0-1954.g1a671c19.el8.x86_64 6/25 2023-01-24T14:28:42.624 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: librbd1-2:18.0.0-1954.g1a671c19.el8.x86_64 6/25 2023-01-24T14:28:42.830 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : thrift-0.13.0-2.el8.x86_64 7/25 2023-01-24T14:28:43.000 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 8/25 2023-01-24T14:28:43.125 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 9/25 2023-01-24T14:28:43.194 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 9/25 2023-01-24T14:28:43.328 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-ceph-argparse-2:18.0.0-1954.g1a671c19.el8. 10/25 2023-01-24T14:28:43.532 INFO:teuthology.orchestra.run.smithi039.stdout:(21/23): ceph-common-18.0.0-1954.g1a671c19.el8. 3.0 MB/s | 27 MB 00:08 2023-01-24T14:28:43.562 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-cephfs-2:18.0.0-1954.g1a671c19.el8.x86_64 11/25 2023-01-24T14:28:43.715 INFO:teuthology.orchestra.run.smithi039.stdout:(22/23): librados2-18.0.0-1954.g1a671c19.el8.x8 1.7 MB/s | 3.3 MB 00:01 2023-01-24T14:28:43.796 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-rbd-2:18.0.0-1954.g1a671c19.el8.x86_64 12/25 2023-01-24T14:28:43.932 INFO:teuthology.orchestra.run.smithi039.stdout:(23/23): librbd1-18.0.0-1954.g1a671c19.el8.x86_ 2.2 MB/s | 3.5 MB 00:01 2023-01-24T14:28:43.935 INFO:teuthology.orchestra.run.smithi039.stdout:-------------------------------------------------------------------------------- 2023-01-24T14:28:43.935 INFO:teuthology.orchestra.run.smithi039.stdout:Total 7.2 MB/s | 67 MB 00:09 2023-01-24T14:28:43.949 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : libradosstriper1-2:18.0.0-1954.g1a671c19.el8.x86_6 13/25 2023-01-24T14:28:43.979 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction check 2023-01-24T14:28:44.028 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction check succeeded. 2023-01-24T14:28:44.028 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction test 2023-01-24T14:28:44.048 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: libradosstriper1-2:18.0.0-1954.g1a671c19.el8.x86_6 13/25 2023-01-24T14:28:44.248 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 14/25 2023-01-24T14:28:44.653 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction test succeeded. 2023-01-24T14:28:44.657 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction 2023-01-24T14:28:45.227 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 15/25 2023-01-24T14:28:45.632 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : librgw2-2:18.0.0-1954.g1a671c19.el8.x86_64 16/25 2023-01-24T14:28:45.697 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: librgw2-2:18.0.0-1954.g1a671c19.el8.x86_64 16/25 2023-01-24T14:28:45.790 INFO:teuthology.orchestra.run.smithi039.stdout: Preparing : 1/1 2023-01-24T14:28:45.909 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-rgw-2:18.0.0-1954.g1a671c19.el8.x86_64 17/25 2023-01-24T14:28:46.048 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : liboath-2.6.2-3.el8.x86_64 1/25 2023-01-24T14:28:46.112 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 18/25 2023-01-24T14:28:46.213 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 2/25 2023-01-24T14:28:46.316 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-ceph-common-2:18.0.0-1954.g1a671c19.el8.x8 19/25 2023-01-24T14:28:46.617 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 2/25 2023-01-24T14:28:46.837 INFO:teuthology.orchestra.run.smithi039.stdout: Upgrading : librados2-2:18.0.0-1954.g1a671c19.el8.x86_64 3/25 2023-01-24T14:28:46.933 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: librados2-2:18.0.0-1954.g1a671c19.el8.x86_64 3/25 2023-01-24T14:28:47.222 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-rados-2:18.0.0-1954.g1a671c19.el8.x86_64 4/25 2023-01-24T14:28:47.404 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : libcephfs2-2:18.0.0-1954.g1a671c19.el8.x86_64 5/25 2023-01-24T14:28:47.845 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: libcephfs2-2:18.0.0-1954.g1a671c19.el8.x86_64 5/25 2023-01-24T14:28:48.004 INFO:teuthology.orchestra.run.smithi039.stdout: Upgrading : librbd1-2:18.0.0-1954.g1a671c19.el8.x86_64 6/25 2023-01-24T14:28:48.229 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: librbd1-2:18.0.0-1954.g1a671c19.el8.x86_64 6/25 2023-01-24T14:28:48.417 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : thrift-0.13.0-2.el8.x86_64 7/25 2023-01-24T14:28:48.570 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 8/25 2023-01-24T14:28:48.720 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 9/25 2023-01-24T14:28:48.800 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 9/25 2023-01-24T14:28:48.940 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-ceph-argparse-2:18.0.0-1954.g1a671c19.el8. 10/25 2023-01-24T14:28:49.275 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-cephfs-2:18.0.0-1954.g1a671c19.el8.x86_64 11/25 2023-01-24T14:28:49.542 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-rbd-2:18.0.0-1954.g1a671c19.el8.x86_64 12/25 2023-01-24T14:28:49.687 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : libradosstriper1-2:18.0.0-1954.g1a671c19.el8.x86_6 13/25 2023-01-24T14:28:49.755 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: libradosstriper1-2:18.0.0-1954.g1a671c19.el8.x86_6 13/25 2023-01-24T14:28:49.931 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 14/25 2023-01-24T14:28:50.831 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: ceph-common-2:18.0.0-1954.g1a671c19.el8.x86_64 20/25 2023-01-24T14:28:50.901 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 15/25 2023-01-24T14:28:51.103 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : librgw2-2:18.0.0-1954.g1a671c19.el8.x86_64 16/25 2023-01-24T14:28:51.186 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: librgw2-2:18.0.0-1954.g1a671c19.el8.x86_64 16/25 2023-01-24T14:28:51.249 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : ceph-common-2:18.0.0-1954.g1a671c19.el8.x86_64 20/25 2023-01-24T14:28:51.378 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-rgw-2:18.0.0-1954.g1a671c19.el8.x86_64 17/25 2023-01-24T14:28:51.572 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 18/25 2023-01-24T14:28:51.779 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-ceph-common-2:18.0.0-1954.g1a671c19.el8.x8 19/25 2023-01-24T14:28:52.039 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: ceph-common-2:18.0.0-1954.g1a671c19.el8.x86_64 20/25 2023-01-24T14:28:52.383 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : ceph-base-2:18.0.0-1954.g1a671c19.el8.x86_64 21/25 2023-01-24T14:28:52.477 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: ceph-base-2:18.0.0-1954.g1a671c19.el8.x86_64 21/25 2023-01-24T14:28:52.649 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : ceph-selinux-2:18.0.0-1954.g1a671c19.el8.x86_64 22/25 2023-01-24T14:28:56.282 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-common-2:18.0.0-1954.g1a671c19.el8.x86_64 20/25 2023-01-24T14:28:56.636 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-common-2:18.0.0-1954.g1a671c19.el8.x86_64 20/25 2023-01-24T14:28:57.409 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-common-2:18.0.0-1954.g1a671c19.el8.x86_64 20/25 2023-01-24T14:28:57.736 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-base-2:18.0.0-1954.g1a671c19.el8.x86_64 21/25 2023-01-24T14:28:57.836 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-base-2:18.0.0-1954.g1a671c19.el8.x86_64 21/25 2023-01-24T14:28:57.994 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-selinux-2:18.0.0-1954.g1a671c19.el8.x86_64 22/25 2023-01-24T14:29:12.956 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: ceph-selinux-2:18.0.0-1954.g1a671c19.el8.x86_64 22/25 2023-01-24T14:29:12.956 INFO:teuthology.orchestra.run.smithi002.stdout:skipping the directory /sys 2023-01-24T14:29:12.957 INFO:teuthology.orchestra.run.smithi002.stdout:skipping the directory /proc 2023-01-24T14:29:12.957 INFO:teuthology.orchestra.run.smithi002.stdout:skipping the directory /dev 2023-01-24T14:29:12.957 INFO:teuthology.orchestra.run.smithi002.stdout:skipping the directory /run 2023-01-24T14:29:12.957 INFO:teuthology.orchestra.run.smithi002.stdout:skipping the directory /mnt 2023-01-24T14:29:12.957 INFO:teuthology.orchestra.run.smithi002.stdout:skipping the directory /var/tmp 2023-01-24T14:29:12.958 INFO:teuthology.orchestra.run.smithi002.stdout:skipping the directory /home 2023-01-24T14:29:12.958 INFO:teuthology.orchestra.run.smithi002.stdout:skipping the directory /tmp 2023-01-24T14:29:12.958 INFO:teuthology.orchestra.run.smithi002.stdout:skipping the directory /dev 2023-01-24T14:29:12.958 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-01-24T14:29:14.705 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : ceph-radosgw-2:18.0.0-1954.g1a671c19.el8.x86_64 23/25 2023-01-24T14:29:14.727 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-1954.g1a671c19.el8.x86_64 23/25 2023-01-24T14:29:14.727 INFO:teuthology.orchestra.run.smithi002.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2023-01-24T14:29:14.922 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2023-01-24T14:29:14.923 INFO:teuthology.orchestra.run.smithi002.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2023-01-24T14:29:15.691 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2023-01-24T14:29:15.692 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : ceph-base-2:18.0.0-1954.g1a671c19.el8.x86_64 1/25 2023-01-24T14:29:15.692 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : ceph-common-2:18.0.0-1954.g1a671c19.el8.x86_64 2/25 2023-01-24T14:29:15.692 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : ceph-radosgw-2:18.0.0-1954.g1a671c19.el8.x86_64 3/25 2023-01-24T14:29:15.693 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : ceph-selinux-2:18.0.0-1954.g1a671c19.el8.x86_64 4/25 2023-01-24T14:29:15.693 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : libcephfs2-2:18.0.0-1954.g1a671c19.el8.x86_64 5/25 2023-01-24T14:29:15.693 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : libradosstriper1-2:18.0.0-1954.g1a671c19.el8.x86_6 6/25 2023-01-24T14:29:15.693 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : librgw2-2:18.0.0-1954.g1a671c19.el8.x86_64 7/25 2023-01-24T14:29:15.693 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-ceph-argparse-2:18.0.0-1954.g1a671c19.el8. 8/25 2023-01-24T14:29:15.694 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-ceph-common-2:18.0.0-1954.g1a671c19.el8.x8 9/25 2023-01-24T14:29:15.694 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-cephfs-2:18.0.0-1954.g1a671c19.el8.x86_64 10/25 2023-01-24T14:29:15.694 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-rados-2:18.0.0-1954.g1a671c19.el8.x86_64 11/25 2023-01-24T14:29:15.694 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-rbd-2:18.0.0-1954.g1a671c19.el8.x86_64 12/25 2023-01-24T14:29:15.695 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-rgw-2:18.0.0-1954.g1a671c19.el8.x86_64 13/25 2023-01-24T14:29:15.695 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 14/25 2023-01-24T14:29:15.695 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 15/25 2023-01-24T14:29:15.695 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 16/25 2023-01-24T14:29:15.695 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 17/25 2023-01-24T14:29:15.696 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 18/25 2023-01-24T14:29:15.696 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 19/25 2023-01-24T14:29:15.696 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 20/25 2023-01-24T14:29:15.696 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 21/25 2023-01-24T14:29:15.696 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : librados2-2:18.0.0-1954.g1a671c19.el8.x86_64 22/25 2023-01-24T14:29:15.697 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2023-01-24T14:29:15.697 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : librbd1-2:18.0.0-1954.g1a671c19.el8.x86_64 24/25 2023-01-24T14:29:16.563 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2023-01-24T14:29:16.564 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-01-24T14:29:16.564 INFO:teuthology.orchestra.run.smithi002.stdout:Upgraded: 2023-01-24T14:29:16.564 INFO:teuthology.orchestra.run.smithi002.stdout: librados2-2:18.0.0-1954.g1a671c19.el8.x86_64 2023-01-24T14:29:16.564 INFO:teuthology.orchestra.run.smithi002.stdout: librbd1-2:18.0.0-1954.g1a671c19.el8.x86_64 2023-01-24T14:29:16.565 INFO:teuthology.orchestra.run.smithi002.stdout:Installed: 2023-01-24T14:29:16.565 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-base-2:18.0.0-1954.g1a671c19.el8.x86_64 2023-01-24T14:29:16.565 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-common-2:18.0.0-1954.g1a671c19.el8.x86_64 2023-01-24T14:29:16.565 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-radosgw-2:18.0.0-1954.g1a671c19.el8.x86_64 2023-01-24T14:29:16.565 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-selinux-2:18.0.0-1954.g1a671c19.el8.x86_64 2023-01-24T14:29:16.566 INFO:teuthology.orchestra.run.smithi002.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-01-24T14:29:16.566 INFO:teuthology.orchestra.run.smithi002.stdout: libcephfs2-2:18.0.0-1954.g1a671c19.el8.x86_64 2023-01-24T14:29:16.566 INFO:teuthology.orchestra.run.smithi002.stdout: liboath-2.6.2-3.el8.x86_64 2023-01-24T14:29:16.566 INFO:teuthology.orchestra.run.smithi002.stdout: librabbitmq-0.9.0-3.el8.x86_64 2023-01-24T14:29:16.566 INFO:teuthology.orchestra.run.smithi002.stdout: libradosstriper1-2:18.0.0-1954.g1a671c19.el8.x86_64 2023-01-24T14:29:16.567 INFO:teuthology.orchestra.run.smithi002.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-01-24T14:29:16.567 INFO:teuthology.orchestra.run.smithi002.stdout: librgw2-2:18.0.0-1954.g1a671c19.el8.x86_64 2023-01-24T14:29:16.567 INFO:teuthology.orchestra.run.smithi002.stdout: libunwind-1.3.1-3.el8.x86_64 2023-01-24T14:29:16.567 INFO:teuthology.orchestra.run.smithi002.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-01-24T14:29:16.567 INFO:teuthology.orchestra.run.smithi002.stdout: python3-ceph-argparse-2:18.0.0-1954.g1a671c19.el8.x86_64 2023-01-24T14:29:16.568 INFO:teuthology.orchestra.run.smithi002.stdout: python3-ceph-common-2:18.0.0-1954.g1a671c19.el8.x86_64 2023-01-24T14:29:16.568 INFO:teuthology.orchestra.run.smithi002.stdout: python3-cephfs-2:18.0.0-1954.g1a671c19.el8.x86_64 2023-01-24T14:29:16.568 INFO:teuthology.orchestra.run.smithi002.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-01-24T14:29:16.568 INFO:teuthology.orchestra.run.smithi002.stdout: python3-rados-2:18.0.0-1954.g1a671c19.el8.x86_64 2023-01-24T14:29:16.569 INFO:teuthology.orchestra.run.smithi002.stdout: python3-rbd-2:18.0.0-1954.g1a671c19.el8.x86_64 2023-01-24T14:29:16.569 INFO:teuthology.orchestra.run.smithi002.stdout: python3-rgw-2:18.0.0-1954.g1a671c19.el8.x86_64 2023-01-24T14:29:16.569 INFO:teuthology.orchestra.run.smithi002.stdout: thrift-0.13.0-2.el8.x86_64 2023-01-24T14:29:16.569 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-01-24T14:29:16.569 INFO:teuthology.orchestra.run.smithi002.stdout:Complete! 2023-01-24T14:29:16.852 DEBUG:teuthology.orchestra.run.smithi002:> sudo yum -y install ceph-test 2023-01-24T14:29:17.619 INFO:teuthology.orchestra.run.smithi002.stdout:Last metadata expiration check: 0:00:47 ago on Tue 24 Jan 2023 02:28:30 PM UTC. 2023-01-24T14:29:18.412 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-selinux-2:18.0.0-1954.g1a671c19.el8.x86_64 22/25 2023-01-24T14:29:18.412 INFO:teuthology.orchestra.run.smithi039.stdout:skipping the directory /sys 2023-01-24T14:29:18.412 INFO:teuthology.orchestra.run.smithi039.stdout:skipping the directory /proc 2023-01-24T14:29:18.413 INFO:teuthology.orchestra.run.smithi039.stdout:skipping the directory /dev 2023-01-24T14:29:18.413 INFO:teuthology.orchestra.run.smithi039.stdout:skipping the directory /run 2023-01-24T14:29:18.413 INFO:teuthology.orchestra.run.smithi039.stdout:skipping the directory /mnt 2023-01-24T14:29:18.413 INFO:teuthology.orchestra.run.smithi039.stdout:skipping the directory /var/tmp 2023-01-24T14:29:18.413 INFO:teuthology.orchestra.run.smithi039.stdout:skipping the directory /home 2023-01-24T14:29:18.414 INFO:teuthology.orchestra.run.smithi039.stdout:skipping the directory /tmp 2023-01-24T14:29:18.414 INFO:teuthology.orchestra.run.smithi039.stdout:skipping the directory /dev 2023-01-24T14:29:18.414 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-01-24T14:29:18.835 INFO:teuthology.orchestra.run.smithi002.stdout:Dependencies resolved. 2023-01-24T14:29:18.836 INFO:teuthology.orchestra.run.smithi002.stdout:================================================================================ 2023-01-24T14:29:18.836 INFO:teuthology.orchestra.run.smithi002.stdout: Package Arch Version Repository Size 2023-01-24T14:29:18.836 INFO:teuthology.orchestra.run.smithi002.stdout:================================================================================ 2023-01-24T14:29:18.837 INFO:teuthology.orchestra.run.smithi002.stdout:Installing: 2023-01-24T14:29:18.837 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-test x86_64 2:18.0.0-1954.g1a671c19.el8 ceph 149 M 2023-01-24T14:29:18.837 INFO:teuthology.orchestra.run.smithi002.stdout:Installing dependencies: 2023-01-24T14:29:18.837 INFO:teuthology.orchestra.run.smithi002.stdout: cryptopp x86_64 8.6.0-1.el8 epel 1.3 M 2023-01-24T14:29:18.838 INFO:teuthology.orchestra.run.smithi002.stdout: jq x86_64 1.6-6.el8 CentOS-AppStream 202 k 2023-01-24T14:29:18.838 INFO:teuthology.orchestra.run.smithi002.stdout: libasan6 x86_64 11.2.1-9.1.el8 CentOS-AppStream 402 k 2023-01-24T14:29:18.838 INFO:teuthology.orchestra.run.smithi002.stdout: libatomic x86_64 8.5.0-18.el8 CentOS-BaseOS 25 k 2023-01-24T14:29:18.838 INFO:teuthology.orchestra.run.smithi002.stdout: libcephsqlite x86_64 2:18.0.0-1954.g1a671c19.el8 ceph 134 k 2023-01-24T14:29:18.838 INFO:teuthology.orchestra.run.smithi002.stdout: libubsan x86_64 8.5.0-18.el8 CentOS-BaseOS 152 k 2023-01-24T14:29:18.839 INFO:teuthology.orchestra.run.smithi002.stdout: oniguruma x86_64 6.8.2-2.el8 CentOS-AppStream 187 k 2023-01-24T14:29:18.839 INFO:teuthology.orchestra.run.smithi002.stdout: socat x86_64 1.7.4.1-1.el8 CentOS-AppStream 323 k 2023-01-24T14:29:18.839 INFO:teuthology.orchestra.run.smithi002.stdout: xmlstarlet x86_64 1.6.1-20.el8 CentOS-AppStream 71 k 2023-01-24T14:29:18.839 INFO:teuthology.orchestra.run.smithi002.stdout: yaml-cpp x86_64 0.6.2-3.el8 epel 171 k 2023-01-24T14:29:18.839 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-01-24T14:29:18.840 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction Summary 2023-01-24T14:29:18.840 INFO:teuthology.orchestra.run.smithi002.stdout:================================================================================ 2023-01-24T14:29:18.840 INFO:teuthology.orchestra.run.smithi002.stdout:Install 11 Packages 2023-01-24T14:29:18.840 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-01-24T14:29:18.841 INFO:teuthology.orchestra.run.smithi002.stdout:Total download size: 152 M 2023-01-24T14:29:18.841 INFO:teuthology.orchestra.run.smithi002.stdout:Installed size: 1.7 G 2023-01-24T14:29:18.841 INFO:teuthology.orchestra.run.smithi002.stdout:Downloading Packages: 2023-01-24T14:29:18.899 INFO:teuthology.orchestra.run.smithi002.stdout:(1/11): jq-1.6-6.el8.x86_64.rpm 3.4 MB/s | 202 kB 00:00 2023-01-24T14:29:18.941 INFO:teuthology.orchestra.run.smithi002.stdout:(2/11): libasan6-11.2.1-9.1.el8.x86_64.rpm 12 MB/s | 402 kB 00:00 2023-01-24T14:29:18.983 INFO:teuthology.orchestra.run.smithi002.stdout:(3/11): oniguruma-6.8.2-2.el8.x86_64.rpm 4.4 MB/s | 187 kB 00:00 2023-01-24T14:29:19.016 INFO:teuthology.orchestra.run.smithi002.stdout:(4/11): socat-1.7.4.1-1.el8.x86_64.rpm 9.7 MB/s | 323 kB 00:00 2023-01-24T14:29:19.042 INFO:teuthology.orchestra.run.smithi002.stdout:(5/11): xmlstarlet-1.6.1-20.el8.x86_64.rpm 2.8 MB/s | 71 kB 00:00 2023-01-24T14:29:19.076 INFO:teuthology.orchestra.run.smithi002.stdout:(6/11): libatomic-8.5.0-18.el8.x86_64.rpm 754 kB/s | 25 kB 00:00 2023-01-24T14:29:19.101 INFO:teuthology.orchestra.run.smithi002.stdout:(7/11): libubsan-8.5.0-18.el8.x86_64.rpm 5.9 MB/s | 152 kB 00:00 2023-01-24T14:29:19.251 INFO:teuthology.orchestra.run.smithi002.stdout:(8/11): libcephsqlite-18.0.0-1954.g1a671c19.el8 327 kB/s | 134 kB 00:00 2023-01-24T14:29:19.560 INFO:teuthology.orchestra.run.smithi002.stdout:(9/11): yaml-cpp-0.6.2-3.el8.x86_64.rpm 553 kB/s | 171 kB 00:00 2023-01-24T14:29:19.793 INFO:teuthology.orchestra.run.smithi002.stdout:(10/11): cryptopp-8.6.0-1.el8.x86_64.rpm 1.8 MB/s | 1.3 MB 00:00 2023-01-24T14:29:20.266 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-radosgw-2:18.0.0-1954.g1a671c19.el8.x86_64 23/25 2023-01-24T14:29:20.289 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-1954.g1a671c19.el8.x86_64 23/25 2023-01-24T14:29:20.289 INFO:teuthology.orchestra.run.smithi039.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2023-01-24T14:29:20.510 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2023-01-24T14:29:20.510 INFO:teuthology.orchestra.run.smithi039.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2023-01-24T14:29:21.317 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2023-01-24T14:29:21.318 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-base-2:18.0.0-1954.g1a671c19.el8.x86_64 1/25 2023-01-24T14:29:21.318 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-common-2:18.0.0-1954.g1a671c19.el8.x86_64 2/25 2023-01-24T14:29:21.319 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-radosgw-2:18.0.0-1954.g1a671c19.el8.x86_64 3/25 2023-01-24T14:29:21.319 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-selinux-2:18.0.0-1954.g1a671c19.el8.x86_64 4/25 2023-01-24T14:29:21.319 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libcephfs2-2:18.0.0-1954.g1a671c19.el8.x86_64 5/25 2023-01-24T14:29:21.320 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libradosstriper1-2:18.0.0-1954.g1a671c19.el8.x86_6 6/25 2023-01-24T14:29:21.320 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : librgw2-2:18.0.0-1954.g1a671c19.el8.x86_64 7/25 2023-01-24T14:29:21.320 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-ceph-argparse-2:18.0.0-1954.g1a671c19.el8. 8/25 2023-01-24T14:29:21.321 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-ceph-common-2:18.0.0-1954.g1a671c19.el8.x8 9/25 2023-01-24T14:29:21.321 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-cephfs-2:18.0.0-1954.g1a671c19.el8.x86_64 10/25 2023-01-24T14:29:21.321 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-rados-2:18.0.0-1954.g1a671c19.el8.x86_64 11/25 2023-01-24T14:29:21.322 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-rbd-2:18.0.0-1954.g1a671c19.el8.x86_64 12/25 2023-01-24T14:29:21.322 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-rgw-2:18.0.0-1954.g1a671c19.el8.x86_64 13/25 2023-01-24T14:29:21.322 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 14/25 2023-01-24T14:29:21.322 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 15/25 2023-01-24T14:29:21.323 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 16/25 2023-01-24T14:29:21.323 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 17/25 2023-01-24T14:29:21.323 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 18/25 2023-01-24T14:29:21.323 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 19/25 2023-01-24T14:29:21.323 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 20/25 2023-01-24T14:29:21.324 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 21/25 2023-01-24T14:29:21.324 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : librados2-2:18.0.0-1954.g1a671c19.el8.x86_64 22/25 2023-01-24T14:29:21.324 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2023-01-24T14:29:21.324 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : librbd1-2:18.0.0-1954.g1a671c19.el8.x86_64 24/25 2023-01-24T14:29:22.185 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2023-01-24T14:29:22.186 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-01-24T14:29:22.186 INFO:teuthology.orchestra.run.smithi039.stdout:Upgraded: 2023-01-24T14:29:22.186 INFO:teuthology.orchestra.run.smithi039.stdout: librados2-2:18.0.0-1954.g1a671c19.el8.x86_64 2023-01-24T14:29:22.186 INFO:teuthology.orchestra.run.smithi039.stdout: librbd1-2:18.0.0-1954.g1a671c19.el8.x86_64 2023-01-24T14:29:22.187 INFO:teuthology.orchestra.run.smithi039.stdout:Installed: 2023-01-24T14:29:22.187 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-base-2:18.0.0-1954.g1a671c19.el8.x86_64 2023-01-24T14:29:22.187 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-common-2:18.0.0-1954.g1a671c19.el8.x86_64 2023-01-24T14:29:22.187 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-radosgw-2:18.0.0-1954.g1a671c19.el8.x86_64 2023-01-24T14:29:22.187 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-selinux-2:18.0.0-1954.g1a671c19.el8.x86_64 2023-01-24T14:29:22.188 INFO:teuthology.orchestra.run.smithi039.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-01-24T14:29:22.188 INFO:teuthology.orchestra.run.smithi039.stdout: libcephfs2-2:18.0.0-1954.g1a671c19.el8.x86_64 2023-01-24T14:29:22.188 INFO:teuthology.orchestra.run.smithi039.stdout: liboath-2.6.2-3.el8.x86_64 2023-01-24T14:29:22.188 INFO:teuthology.orchestra.run.smithi039.stdout: librabbitmq-0.9.0-3.el8.x86_64 2023-01-24T14:29:22.188 INFO:teuthology.orchestra.run.smithi039.stdout: libradosstriper1-2:18.0.0-1954.g1a671c19.el8.x86_64 2023-01-24T14:29:22.189 INFO:teuthology.orchestra.run.smithi039.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-01-24T14:29:22.189 INFO:teuthology.orchestra.run.smithi039.stdout: librgw2-2:18.0.0-1954.g1a671c19.el8.x86_64 2023-01-24T14:29:22.189 INFO:teuthology.orchestra.run.smithi039.stdout: libunwind-1.3.1-3.el8.x86_64 2023-01-24T14:29:22.189 INFO:teuthology.orchestra.run.smithi039.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-01-24T14:29:22.189 INFO:teuthology.orchestra.run.smithi039.stdout: python3-ceph-argparse-2:18.0.0-1954.g1a671c19.el8.x86_64 2023-01-24T14:29:22.190 INFO:teuthology.orchestra.run.smithi039.stdout: python3-ceph-common-2:18.0.0-1954.g1a671c19.el8.x86_64 2023-01-24T14:29:22.190 INFO:teuthology.orchestra.run.smithi039.stdout: python3-cephfs-2:18.0.0-1954.g1a671c19.el8.x86_64 2023-01-24T14:29:22.190 INFO:teuthology.orchestra.run.smithi039.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-01-24T14:29:22.190 INFO:teuthology.orchestra.run.smithi039.stdout: python3-rados-2:18.0.0-1954.g1a671c19.el8.x86_64 2023-01-24T14:29:22.190 INFO:teuthology.orchestra.run.smithi039.stdout: python3-rbd-2:18.0.0-1954.g1a671c19.el8.x86_64 2023-01-24T14:29:22.191 INFO:teuthology.orchestra.run.smithi039.stdout: python3-rgw-2:18.0.0-1954.g1a671c19.el8.x86_64 2023-01-24T14:29:22.191 INFO:teuthology.orchestra.run.smithi039.stdout: thrift-0.13.0-2.el8.x86_64 2023-01-24T14:29:22.191 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-01-24T14:29:22.191 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2023-01-24T14:29:22.473 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum -y install ceph-test 2023-01-24T14:29:23.232 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:00:53 ago on Tue 24 Jan 2023 02:28:30 PM UTC. 2023-01-24T14:29:24.457 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2023-01-24T14:29:24.458 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2023-01-24T14:29:24.458 INFO:teuthology.orchestra.run.smithi039.stdout: Package Arch Version Repository Size 2023-01-24T14:29:24.458 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2023-01-24T14:29:24.458 INFO:teuthology.orchestra.run.smithi039.stdout:Installing: 2023-01-24T14:29:24.459 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-test x86_64 2:18.0.0-1954.g1a671c19.el8 ceph 149 M 2023-01-24T14:29:24.459 INFO:teuthology.orchestra.run.smithi039.stdout:Installing dependencies: 2023-01-24T14:29:24.459 INFO:teuthology.orchestra.run.smithi039.stdout: cryptopp x86_64 8.6.0-1.el8 epel 1.3 M 2023-01-24T14:29:24.459 INFO:teuthology.orchestra.run.smithi039.stdout: jq x86_64 1.6-6.el8 CentOS-AppStream 202 k 2023-01-24T14:29:24.459 INFO:teuthology.orchestra.run.smithi039.stdout: libasan6 x86_64 11.2.1-9.1.el8 CentOS-AppStream 402 k 2023-01-24T14:29:24.460 INFO:teuthology.orchestra.run.smithi039.stdout: libatomic x86_64 8.5.0-18.el8 CentOS-BaseOS 25 k 2023-01-24T14:29:24.460 INFO:teuthology.orchestra.run.smithi039.stdout: libcephsqlite x86_64 2:18.0.0-1954.g1a671c19.el8 ceph 134 k 2023-01-24T14:29:24.460 INFO:teuthology.orchestra.run.smithi039.stdout: libubsan x86_64 8.5.0-18.el8 CentOS-BaseOS 152 k 2023-01-24T14:29:24.460 INFO:teuthology.orchestra.run.smithi039.stdout: oniguruma x86_64 6.8.2-2.el8 CentOS-AppStream 187 k 2023-01-24T14:29:24.460 INFO:teuthology.orchestra.run.smithi039.stdout: socat x86_64 1.7.4.1-1.el8 CentOS-AppStream 323 k 2023-01-24T14:29:24.461 INFO:teuthology.orchestra.run.smithi039.stdout: xmlstarlet x86_64 1.6.1-20.el8 CentOS-AppStream 71 k 2023-01-24T14:29:24.461 INFO:teuthology.orchestra.run.smithi039.stdout: yaml-cpp x86_64 0.6.2-3.el8 epel 171 k 2023-01-24T14:29:24.461 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-01-24T14:29:24.461 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction Summary 2023-01-24T14:29:24.461 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2023-01-24T14:29:24.462 INFO:teuthology.orchestra.run.smithi039.stdout:Install 11 Packages 2023-01-24T14:29:24.462 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-01-24T14:29:24.462 INFO:teuthology.orchestra.run.smithi039.stdout:Total download size: 152 M 2023-01-24T14:29:24.463 INFO:teuthology.orchestra.run.smithi039.stdout:Installed size: 1.7 G 2023-01-24T14:29:24.463 INFO:teuthology.orchestra.run.smithi039.stdout:Downloading Packages: 2023-01-24T14:29:24.529 INFO:teuthology.orchestra.run.smithi039.stdout:(1/11): jq-1.6-6.el8.x86_64.rpm 2.9 MB/s | 202 kB 00:00 2023-01-24T14:29:24.571 INFO:teuthology.orchestra.run.smithi039.stdout:(2/11): libasan6-11.2.1-9.1.el8.x86_64.rpm 12 MB/s | 402 kB 00:00 2023-01-24T14:29:24.596 INFO:teuthology.orchestra.run.smithi039.stdout:(3/11): oniguruma-6.8.2-2.el8.x86_64.rpm 7.3 MB/s | 187 kB 00:00 2023-01-24T14:29:24.630 INFO:teuthology.orchestra.run.smithi039.stdout:(4/11): socat-1.7.4.1-1.el8.x86_64.rpm 9.7 MB/s | 323 kB 00:00 2023-01-24T14:29:24.655 INFO:teuthology.orchestra.run.smithi039.stdout:(5/11): xmlstarlet-1.6.1-20.el8.x86_64.rpm 2.8 MB/s | 71 kB 00:00 2023-01-24T14:29:24.681 INFO:teuthology.orchestra.run.smithi039.stdout:(6/11): libatomic-8.5.0-18.el8.x86_64.rpm 1.0 MB/s | 25 kB 00:00 2023-01-24T14:29:24.707 INFO:teuthology.orchestra.run.smithi039.stdout:(7/11): libubsan-8.5.0-18.el8.x86_64.rpm 5.9 MB/s | 152 kB 00:00 2023-01-24T14:29:24.848 INFO:teuthology.orchestra.run.smithi039.stdout:(8/11): libcephsqlite-18.0.0-1954.g1a671c19.el8 348 kB/s | 134 kB 00:00 2023-01-24T14:29:25.082 INFO:teuthology.orchestra.run.smithi039.stdout:(9/11): yaml-cpp-0.6.2-3.el8.x86_64.rpm 732 kB/s | 171 kB 00:00 2023-01-24T14:29:25.591 INFO:teuthology.orchestra.run.smithi039.stdout:(10/11): cryptopp-8.6.0-1.el8.x86_64.rpm 1.4 MB/s | 1.3 MB 00:00 2023-01-24T14:29:27.235 INFO:teuthology.orchestra.run.smithi002.stdout:(11/11): ceph-test-18.0.0-1954.g1a671c19.el8.x8 18 MB/s | 149 MB 00:08 2023-01-24T14:29:27.237 INFO:teuthology.orchestra.run.smithi002.stdout:-------------------------------------------------------------------------------- 2023-01-24T14:29:27.237 INFO:teuthology.orchestra.run.smithi002.stdout:Total 18 MB/s | 152 MB 00:08 2023-01-24T14:29:27.311 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction check 2023-01-24T14:29:27.337 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction check succeeded. 2023-01-24T14:29:27.338 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction test 2023-01-24T14:29:28.063 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction test succeeded. 2023-01-24T14:29:28.067 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction 2023-01-24T14:29:29.129 INFO:teuthology.orchestra.run.smithi002.stdout: Preparing : 1/1 2023-01-24T14:29:29.444 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : yaml-cpp-0.6.2-3.el8.x86_64 1/11 2023-01-24T14:29:29.638 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : cryptopp-8.6.0-1.el8.x86_64 2/11 2023-01-24T14:29:29.767 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : libubsan-8.5.0-18.el8.x86_64 3/11 2023-01-24T14:29:29.832 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: libubsan-8.5.0-18.el8.x86_64 3/11 2023-01-24T14:29:29.934 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : libatomic-8.5.0-18.el8.x86_64 4/11 2023-01-24T14:29:30.014 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: libatomic-8.5.0-18.el8.x86_64 4/11 2023-01-24T14:29:30.240 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 5/11 2023-01-24T14:29:30.413 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 6/11 2023-01-24T14:29:30.592 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 7/11 2023-01-24T14:29:30.677 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 7/11 2023-01-24T14:29:30.888 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : jq-1.6-6.el8.x86_64 8/11 2023-01-24T14:29:31.059 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : libasan6-11.2.1-9.1.el8.x86_64 9/11 2023-01-24T14:29:31.139 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: libasan6-11.2.1-9.1.el8.x86_64 9/11 2023-01-24T14:29:31.326 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : libcephsqlite-2:18.0.0-1954.g1a671c19.el8.x86_64 10/11 2023-01-24T14:29:33.390 INFO:teuthology.orchestra.run.smithi039.stdout:(11/11): ceph-test-18.0.0-1954.g1a671c19.el8.x8 17 MB/s | 149 MB 00:08 2023-01-24T14:29:33.392 INFO:teuthology.orchestra.run.smithi039.stdout:-------------------------------------------------------------------------------- 2023-01-24T14:29:33.393 INFO:teuthology.orchestra.run.smithi039.stdout:Total 17 MB/s | 152 MB 00:08 2023-01-24T14:29:33.468 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction check 2023-01-24T14:29:33.494 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction check succeeded. 2023-01-24T14:29:33.494 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction test 2023-01-24T14:29:34.228 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction test succeeded. 2023-01-24T14:29:34.230 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction 2023-01-24T14:29:35.393 INFO:teuthology.orchestra.run.smithi039.stdout: Preparing : 1/1 2023-01-24T14:29:35.731 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : yaml-cpp-0.6.2-3.el8.x86_64 1/11 2023-01-24T14:29:35.909 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : cryptopp-8.6.0-1.el8.x86_64 2/11 2023-01-24T14:29:36.063 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : libubsan-8.5.0-18.el8.x86_64 3/11 2023-01-24T14:29:36.128 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: libubsan-8.5.0-18.el8.x86_64 3/11 2023-01-24T14:29:36.255 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : libatomic-8.5.0-18.el8.x86_64 4/11 2023-01-24T14:29:36.331 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: libatomic-8.5.0-18.el8.x86_64 4/11 2023-01-24T14:29:36.594 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 5/11 2023-01-24T14:29:36.776 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 6/11 2023-01-24T14:29:37.055 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 7/11 2023-01-24T14:29:37.165 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 7/11 2023-01-24T14:29:37.400 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : jq-1.6-6.el8.x86_64 8/11 2023-01-24T14:29:37.555 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : libasan6-11.2.1-9.1.el8.x86_64 9/11 2023-01-24T14:29:37.636 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: libasan6-11.2.1-9.1.el8.x86_64 9/11 2023-01-24T14:29:37.821 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : libcephsqlite-2:18.0.0-1954.g1a671c19.el8.x86_64 10/11 2023-01-24T14:29:55.732 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: libcephsqlite-2:18.0.0-1954.g1a671c19.el8.x86_64 10/11 2023-01-24T14:29:55.995 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : ceph-test-2:18.0.0-1954.g1a671c19.el8.x86_64 11/11 2023-01-24T14:29:56.305 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: ceph-test-2:18.0.0-1954.g1a671c19.el8.x86_64 11/11 2023-01-24T14:29:56.306 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : ceph-test-2:18.0.0-1954.g1a671c19.el8.x86_64 1/11 2023-01-24T14:29:56.306 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : libcephsqlite-2:18.0.0-1954.g1a671c19.el8.x86_64 2/11 2023-01-24T14:29:56.306 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : jq-1.6-6.el8.x86_64 3/11 2023-01-24T14:29:56.307 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : libasan6-11.2.1-9.1.el8.x86_64 4/11 2023-01-24T14:29:56.307 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 5/11 2023-01-24T14:29:56.307 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 6/11 2023-01-24T14:29:56.307 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 7/11 2023-01-24T14:29:56.307 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : libatomic-8.5.0-18.el8.x86_64 8/11 2023-01-24T14:29:56.308 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : libubsan-8.5.0-18.el8.x86_64 9/11 2023-01-24T14:29:56.308 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : cryptopp-8.6.0-1.el8.x86_64 10/11 2023-01-24T14:29:57.019 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : yaml-cpp-0.6.2-3.el8.x86_64 11/11 2023-01-24T14:29:57.020 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-01-24T14:29:57.020 INFO:teuthology.orchestra.run.smithi002.stdout:Installed: 2023-01-24T14:29:57.020 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-test-2:18.0.0-1954.g1a671c19.el8.x86_64 2023-01-24T14:29:57.020 INFO:teuthology.orchestra.run.smithi002.stdout: cryptopp-8.6.0-1.el8.x86_64 2023-01-24T14:29:57.020 INFO:teuthology.orchestra.run.smithi002.stdout: jq-1.6-6.el8.x86_64 2023-01-24T14:29:57.021 INFO:teuthology.orchestra.run.smithi002.stdout: libasan6-11.2.1-9.1.el8.x86_64 2023-01-24T14:29:57.021 INFO:teuthology.orchestra.run.smithi002.stdout: libatomic-8.5.0-18.el8.x86_64 2023-01-24T14:29:57.022 INFO:teuthology.orchestra.run.smithi002.stdout: libcephsqlite-2:18.0.0-1954.g1a671c19.el8.x86_64 2023-01-24T14:29:57.022 INFO:teuthology.orchestra.run.smithi002.stdout: libubsan-8.5.0-18.el8.x86_64 2023-01-24T14:29:57.022 INFO:teuthology.orchestra.run.smithi002.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-01-24T14:29:57.023 INFO:teuthology.orchestra.run.smithi002.stdout: socat-1.7.4.1-1.el8.x86_64 2023-01-24T14:29:57.023 INFO:teuthology.orchestra.run.smithi002.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-01-24T14:29:57.023 INFO:teuthology.orchestra.run.smithi002.stdout: yaml-cpp-0.6.2-3.el8.x86_64 2023-01-24T14:29:57.023 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-01-24T14:29:57.024 INFO:teuthology.orchestra.run.smithi002.stdout:Complete! 2023-01-24T14:29:57.242 DEBUG:teuthology.orchestra.run.smithi002:> sudo yum -y install ceph 2023-01-24T14:29:57.999 INFO:teuthology.orchestra.run.smithi002.stdout:Last metadata expiration check: 0:01:27 ago on Tue 24 Jan 2023 02:28:30 PM UTC. 2023-01-24T14:29:59.232 INFO:teuthology.orchestra.run.smithi002.stdout:Dependencies resolved. 2023-01-24T14:29:59.236 INFO:teuthology.orchestra.run.smithi002.stdout:======================================================================================= 2023-01-24T14:29:59.236 INFO:teuthology.orchestra.run.smithi002.stdout: Package Arch Version Repository Size 2023-01-24T14:29:59.237 INFO:teuthology.orchestra.run.smithi002.stdout:======================================================================================= 2023-01-24T14:29:59.237 INFO:teuthology.orchestra.run.smithi002.stdout:Installing: 2023-01-24T14:29:59.237 INFO:teuthology.orchestra.run.smithi002.stdout: ceph x86_64 2:18.0.0-1954.g1a671c19.el8 ceph 6.4 k 2023-01-24T14:29:59.237 INFO:teuthology.orchestra.run.smithi002.stdout:Installing dependencies: 2023-01-24T14:29:59.237 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-mds x86_64 2:18.0.0-1954.g1a671c19.el8 ceph 2.2 M 2023-01-24T14:29:59.238 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-mgr x86_64 2:18.0.0-1954.g1a671c19.el8 ceph 1.5 M 2023-01-24T14:29:59.238 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-mgr-modules-core noarch 2:18.0.0-1954.g1a671c19.el8 ceph-noarch 236 k 2023-01-24T14:29:59.238 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-mon x86_64 2:18.0.0-1954.g1a671c19.el8 ceph 7.4 M 2023-01-24T14:29:59.238 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-osd x86_64 2:18.0.0-1954.g1a671c19.el8 ceph 151 M 2023-01-24T14:29:59.238 INFO:teuthology.orchestra.run.smithi002.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-01-24T14:29:59.239 INFO:teuthology.orchestra.run.smithi002.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-01-24T14:29:59.239 INFO:teuthology.orchestra.run.smithi002.stdout: python3-cffi x86_64 1.11.5-5.el8 CentOS-BaseOS 237 k 2023-01-24T14:29:59.239 INFO:teuthology.orchestra.run.smithi002.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-01-24T14:29:59.239 INFO:teuthology.orchestra.run.smithi002.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-01-24T14:29:59.239 INFO:teuthology.orchestra.run.smithi002.stdout: python3-cryptography x86_64 3.2.1-5.el8 CentOS-BaseOS 559 k 2023-01-24T14:29:59.240 INFO:teuthology.orchestra.run.smithi002.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-01-24T14:29:59.240 INFO:teuthology.orchestra.run.smithi002.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-01-24T14:29:59.240 INFO:teuthology.orchestra.run.smithi002.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-01-24T14:29:59.240 INFO:teuthology.orchestra.run.smithi002.stdout: python3-mako noarch 1.0.6-13.el8 CentOS-AppStream 157 k 2023-01-24T14:29:59.240 INFO:teuthology.orchestra.run.smithi002.stdout: python3-markupsafe x86_64 0.23-19.el8 CentOS-AppStream 39 k 2023-01-24T14:29:59.241 INFO:teuthology.orchestra.run.smithi002.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-01-24T14:29:59.241 INFO:teuthology.orchestra.run.smithi002.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-01-24T14:29:59.241 INFO:teuthology.orchestra.run.smithi002.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-01-24T14:29:59.241 INFO:teuthology.orchestra.run.smithi002.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 CentOS-AppStream 103 k 2023-01-24T14:29:59.241 INFO:teuthology.orchestra.run.smithi002.stdout: python3-pycparser noarch 2.14-14.el8 CentOS-BaseOS 109 k 2023-01-24T14:29:59.242 INFO:teuthology.orchestra.run.smithi002.stdout: python3-pytz noarch 2017.2-9.el8 CentOS-AppStream 54 k 2023-01-24T14:29:59.242 INFO:teuthology.orchestra.run.smithi002.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-01-24T14:29:59.242 INFO:teuthology.orchestra.run.smithi002.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-01-24T14:29:59.242 INFO:teuthology.orchestra.run.smithi002.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-01-24T14:29:59.242 INFO:teuthology.orchestra.run.smithi002.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-01-24T14:29:59.243 INFO:teuthology.orchestra.run.smithi002.stdout: python3-waitress noarch 1.2.1-2.el8.1 epel 241 k 2023-01-24T14:29:59.243 INFO:teuthology.orchestra.run.smithi002.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-01-24T14:29:59.243 INFO:teuthology.orchestra.run.smithi002.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-01-24T14:29:59.243 INFO:teuthology.orchestra.run.smithi002.stdout: python3-werkzeug noarch 0.12.2-4.el8 CentOS-AppStream 457 k 2023-01-24T14:29:59.243 INFO:teuthology.orchestra.run.smithi002.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-01-24T14:29:59.244 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-01-24T14:29:59.244 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction Summary 2023-01-24T14:29:59.244 INFO:teuthology.orchestra.run.smithi002.stdout:======================================================================================= 2023-01-24T14:29:59.244 INFO:teuthology.orchestra.run.smithi002.stdout:Install 32 Packages 2023-01-24T14:29:59.244 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-01-24T14:29:59.245 INFO:teuthology.orchestra.run.smithi002.stdout:Total download size: 165 M 2023-01-24T14:29:59.245 INFO:teuthology.orchestra.run.smithi002.stdout:Installed size: 2.2 G 2023-01-24T14:29:59.246 INFO:teuthology.orchestra.run.smithi002.stdout:Downloading Packages: 2023-01-24T14:29:59.389 INFO:teuthology.orchestra.run.smithi002.stdout:(1/32): ceph-18.0.0-1954.g1a671c19.el8.x86_64.r 44 kB/s | 6.4 kB 00:00 2023-01-24T14:29:59.855 INFO:teuthology.orchestra.run.smithi002.stdout:(2/32): ceph-mgr-18.0.0-1954.g1a671c19.el8.x86_ 2.4 MB/s | 1.5 MB 00:00 2023-01-24T14:29:59.956 INFO:teuthology.orchestra.run.smithi002.stdout:(3/32): ceph-mds-18.0.0-1954.g1a671c19.el8.x86_ 3.1 MB/s | 2.2 MB 00:00 2023-01-24T14:30:00.031 INFO:teuthology.orchestra.run.smithi002.stdout:(4/32): ceph-mgr-modules-core-18.0.0-1954.g1a67 3.1 MB/s | 236 kB 00:00 2023-01-24T14:30:00.140 INFO:teuthology.orchestra.run.smithi002.stdout:(5/32): python3-mako-1.0.6-13.el8.noarch.rpm 1.4 MB/s | 157 kB 00:00 2023-01-24T14:30:00.173 INFO:teuthology.orchestra.run.smithi002.stdout:(6/32): python3-markupsafe-0.23-19.el8.x86_64.r 1.1 MB/s | 39 kB 00:00 2023-01-24T14:30:00.207 INFO:teuthology.orchestra.run.smithi002.stdout:(7/32): python3-pyOpenSSL-19.0.0-1.el8.noarch.r 3.0 MB/s | 103 kB 00:00 2023-01-24T14:30:00.232 INFO:teuthology.orchestra.run.smithi002.stdout:(8/32): python3-pytz-2017.2-9.el8.noarch.rpm 2.1 MB/s | 54 kB 00:00 2023-01-24T14:30:00.266 INFO:teuthology.orchestra.run.smithi002.stdout:(9/32): python3-werkzeug-0.12.2-4.el8.noarch.rp 13 MB/s | 457 kB 00:00 2023-01-24T14:30:00.291 INFO:teuthology.orchestra.run.smithi002.stdout:(10/32): python3-cffi-1.11.5-5.el8.x86_64.rpm 9.3 MB/s | 237 kB 00:00 2023-01-24T14:30:00.325 INFO:teuthology.orchestra.run.smithi002.stdout:(11/32): python3-cryptography-3.2.1-5.el8.x86_6 16 MB/s | 559 kB 00:00 2023-01-24T14:30:00.350 INFO:teuthology.orchestra.run.smithi002.stdout:(12/32): python3-pycparser-2.14-14.el8.noarch.r 4.3 MB/s | 109 kB 00:00 2023-01-24T14:30:00.467 INFO:teuthology.orchestra.run.smithi002.stdout:(13/32): ceph-mon-18.0.0-1954.g1a671c19.el8.x86 6.9 MB/s | 7.4 MB 00:01 2023-01-24T14:30:00.584 INFO:teuthology.orchestra.run.smithi002.stdout:(14/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 186 kB/s | 44 kB 00:00 2023-01-24T14:30:00.651 INFO:teuthology.orchestra.run.smithi002.stdout:(15/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 1.0 MB/s | 185 kB 00:00 2023-01-24T14:30:00.710 INFO:teuthology.orchestra.run.smithi002.stdout:(16/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 1.3 MB/s | 173 kB 00:00 2023-01-24T14:30:00.777 INFO:teuthology.orchestra.run.smithi002.stdout:(17/32): python3-jaraco-6.2-6.el8.noarch.rpm 167 kB/s | 11 kB 00:00 2023-01-24T14:30:00.827 INFO:teuthology.orchestra.run.smithi002.stdout:(18/32): python3-cherrypy-18.4.0-1.el8.noarch.r 2.1 MB/s | 384 kB 00:00 2023-01-24T14:30:00.844 INFO:teuthology.orchestra.run.smithi002.stdout:(19/32): python3-jaraco-functools-2.0-4.el8.noa 262 kB/s | 18 kB 00:00 2023-01-24T14:30:00.878 INFO:teuthology.orchestra.run.smithi002.stdout:(20/32): python3-logutils-0.3.5-11.el8.noarch.r 967 kB/s | 49 kB 00:00 2023-01-24T14:30:00.903 INFO:teuthology.orchestra.run.smithi002.stdout:(21/32): python3-more-itertools-7.2.0-3.el8.noa 1.0 MB/s | 59 kB 00:00 2023-01-24T14:30:00.962 INFO:teuthology.orchestra.run.smithi002.stdout:(22/32): python3-portend-2.6-1.el8.noarch.rpm 266 kB/s | 16 kB 00:00 2023-01-24T14:30:01.020 INFO:teuthology.orchestra.run.smithi002.stdout:(23/32): python3-pecan-1.3.2-9.el8.noarch.rpm 1.9 MB/s | 283 kB 00:00 2023-01-24T14:30:01.046 INFO:teuthology.orchestra.run.smithi002.stdout:(24/32): python3-simplegeneric-0.8.1-17.el8.noa 223 kB/s | 19 kB 00:00 2023-01-24T14:30:01.079 INFO:teuthology.orchestra.run.smithi002.stdout:(25/32): python3-singledispatch-3.4.0.3-18.el8. 407 kB/s | 24 kB 00:00 2023-01-24T14:30:01.105 INFO:teuthology.orchestra.run.smithi002.stdout:(26/32): python3-tempora-1.14.1-5.el8.noarch.rp 498 kB/s | 29 kB 00:00 2023-01-24T14:30:01.138 INFO:teuthology.orchestra.run.smithi002.stdout:(27/32): python3-trustme-0.6.0-4.el8.noarch.rpm 469 kB/s | 27 kB 00:00 2023-01-24T14:30:01.239 INFO:teuthology.orchestra.run.smithi002.stdout:(28/32): python3-waitress-1.2.1-2.el8.1.noarch. 1.8 MB/s | 241 kB 00:00 2023-01-24T14:30:01.297 INFO:teuthology.orchestra.run.smithi002.stdout:(29/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 1.5 MB/s | 251 kB 00:00 2023-01-24T14:30:01.322 INFO:teuthology.orchestra.run.smithi002.stdout:(30/32): python3-webtest-2.0.33-1.el8.noarch.rp 1.0 MB/s | 85 kB 00:00 2023-01-24T14:30:01.348 INFO:teuthology.orchestra.run.smithi002.stdout:(31/32): python3-zc-lockfile-2.0-2.el8.noarch.r 455 kB/s | 23 kB 00:00 2023-01-24T14:30:02.557 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: libcephsqlite-2:18.0.0-1954.g1a671c19.el8.x86_64 10/11 2023-01-24T14:30:02.849 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-test-2:18.0.0-1954.g1a671c19.el8.x86_64 11/11 2023-01-24T14:30:03.157 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-test-2:18.0.0-1954.g1a671c19.el8.x86_64 11/11 2023-01-24T14:30:03.157 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-test-2:18.0.0-1954.g1a671c19.el8.x86_64 1/11 2023-01-24T14:30:03.157 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libcephsqlite-2:18.0.0-1954.g1a671c19.el8.x86_64 2/11 2023-01-24T14:30:03.158 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : jq-1.6-6.el8.x86_64 3/11 2023-01-24T14:30:03.158 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libasan6-11.2.1-9.1.el8.x86_64 4/11 2023-01-24T14:30:03.158 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 5/11 2023-01-24T14:30:03.162 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 6/11 2023-01-24T14:30:03.162 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 7/11 2023-01-24T14:30:03.162 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libatomic-8.5.0-18.el8.x86_64 8/11 2023-01-24T14:30:03.163 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libubsan-8.5.0-18.el8.x86_64 9/11 2023-01-24T14:30:03.163 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : cryptopp-8.6.0-1.el8.x86_64 10/11 2023-01-24T14:30:03.748 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : yaml-cpp-0.6.2-3.el8.x86_64 11/11 2023-01-24T14:30:03.749 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-01-24T14:30:03.749 INFO:teuthology.orchestra.run.smithi039.stdout:Installed: 2023-01-24T14:30:03.749 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-test-2:18.0.0-1954.g1a671c19.el8.x86_64 2023-01-24T14:30:03.749 INFO:teuthology.orchestra.run.smithi039.stdout: cryptopp-8.6.0-1.el8.x86_64 2023-01-24T14:30:03.750 INFO:teuthology.orchestra.run.smithi039.stdout: jq-1.6-6.el8.x86_64 2023-01-24T14:30:03.750 INFO:teuthology.orchestra.run.smithi039.stdout: libasan6-11.2.1-9.1.el8.x86_64 2023-01-24T14:30:03.750 INFO:teuthology.orchestra.run.smithi039.stdout: libatomic-8.5.0-18.el8.x86_64 2023-01-24T14:30:03.750 INFO:teuthology.orchestra.run.smithi039.stdout: libcephsqlite-2:18.0.0-1954.g1a671c19.el8.x86_64 2023-01-24T14:30:03.750 INFO:teuthology.orchestra.run.smithi039.stdout: libubsan-8.5.0-18.el8.x86_64 2023-01-24T14:30:03.751 INFO:teuthology.orchestra.run.smithi039.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-01-24T14:30:03.751 INFO:teuthology.orchestra.run.smithi039.stdout: socat-1.7.4.1-1.el8.x86_64 2023-01-24T14:30:03.751 INFO:teuthology.orchestra.run.smithi039.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-01-24T14:30:03.751 INFO:teuthology.orchestra.run.smithi039.stdout: yaml-cpp-0.6.2-3.el8.x86_64 2023-01-24T14:30:03.751 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-01-24T14:30:03.752 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2023-01-24T14:30:03.986 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum -y install ceph 2023-01-24T14:30:04.747 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:01:34 ago on Tue 24 Jan 2023 02:28:30 PM UTC. 2023-01-24T14:30:05.977 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2023-01-24T14:30:05.979 INFO:teuthology.orchestra.run.smithi039.stdout:======================================================================================= 2023-01-24T14:30:05.979 INFO:teuthology.orchestra.run.smithi039.stdout: Package Arch Version Repository Size 2023-01-24T14:30:05.979 INFO:teuthology.orchestra.run.smithi039.stdout:======================================================================================= 2023-01-24T14:30:05.980 INFO:teuthology.orchestra.run.smithi039.stdout:Installing: 2023-01-24T14:30:05.980 INFO:teuthology.orchestra.run.smithi039.stdout: ceph x86_64 2:18.0.0-1954.g1a671c19.el8 ceph 6.4 k 2023-01-24T14:30:05.980 INFO:teuthology.orchestra.run.smithi039.stdout:Installing dependencies: 2023-01-24T14:30:05.980 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mds x86_64 2:18.0.0-1954.g1a671c19.el8 ceph 2.2 M 2023-01-24T14:30:05.981 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mgr x86_64 2:18.0.0-1954.g1a671c19.el8 ceph 1.5 M 2023-01-24T14:30:05.981 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mgr-modules-core noarch 2:18.0.0-1954.g1a671c19.el8 ceph-noarch 236 k 2023-01-24T14:30:05.981 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mon x86_64 2:18.0.0-1954.g1a671c19.el8 ceph 7.4 M 2023-01-24T14:30:05.981 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-osd x86_64 2:18.0.0-1954.g1a671c19.el8 ceph 151 M 2023-01-24T14:30:05.981 INFO:teuthology.orchestra.run.smithi039.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-01-24T14:30:05.982 INFO:teuthology.orchestra.run.smithi039.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-01-24T14:30:05.982 INFO:teuthology.orchestra.run.smithi039.stdout: python3-cffi x86_64 1.11.5-5.el8 CentOS-BaseOS 237 k 2023-01-24T14:30:05.982 INFO:teuthology.orchestra.run.smithi039.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-01-24T14:30:05.982 INFO:teuthology.orchestra.run.smithi039.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-01-24T14:30:05.982 INFO:teuthology.orchestra.run.smithi039.stdout: python3-cryptography x86_64 3.2.1-5.el8 CentOS-BaseOS 559 k 2023-01-24T14:30:05.983 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-01-24T14:30:05.983 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-01-24T14:30:05.983 INFO:teuthology.orchestra.run.smithi039.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-01-24T14:30:05.983 INFO:teuthology.orchestra.run.smithi039.stdout: python3-mako noarch 1.0.6-13.el8 CentOS-AppStream 157 k 2023-01-24T14:30:05.983 INFO:teuthology.orchestra.run.smithi039.stdout: python3-markupsafe x86_64 0.23-19.el8 CentOS-AppStream 39 k 2023-01-24T14:30:05.984 INFO:teuthology.orchestra.run.smithi039.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-01-24T14:30:05.984 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-01-24T14:30:05.984 INFO:teuthology.orchestra.run.smithi039.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-01-24T14:30:05.984 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 CentOS-AppStream 103 k 2023-01-24T14:30:05.984 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pycparser noarch 2.14-14.el8 CentOS-BaseOS 109 k 2023-01-24T14:30:05.985 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pytz noarch 2017.2-9.el8 CentOS-AppStream 54 k 2023-01-24T14:30:05.985 INFO:teuthology.orchestra.run.smithi039.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-01-24T14:30:05.986 INFO:teuthology.orchestra.run.smithi039.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-01-24T14:30:05.986 INFO:teuthology.orchestra.run.smithi039.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-01-24T14:30:05.986 INFO:teuthology.orchestra.run.smithi039.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-01-24T14:30:05.986 INFO:teuthology.orchestra.run.smithi039.stdout: python3-waitress noarch 1.2.1-2.el8.1 epel 241 k 2023-01-24T14:30:05.987 INFO:teuthology.orchestra.run.smithi039.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-01-24T14:30:05.987 INFO:teuthology.orchestra.run.smithi039.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-01-24T14:30:05.987 INFO:teuthology.orchestra.run.smithi039.stdout: python3-werkzeug noarch 0.12.2-4.el8 CentOS-AppStream 457 k 2023-01-24T14:30:05.987 INFO:teuthology.orchestra.run.smithi039.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-01-24T14:30:05.987 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-01-24T14:30:05.988 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction Summary 2023-01-24T14:30:05.988 INFO:teuthology.orchestra.run.smithi039.stdout:======================================================================================= 2023-01-24T14:30:05.988 INFO:teuthology.orchestra.run.smithi039.stdout:Install 32 Packages 2023-01-24T14:30:05.988 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-01-24T14:30:05.989 INFO:teuthology.orchestra.run.smithi039.stdout:Total download size: 165 M 2023-01-24T14:30:05.989 INFO:teuthology.orchestra.run.smithi039.stdout:Installed size: 2.2 G 2023-01-24T14:30:05.990 INFO:teuthology.orchestra.run.smithi039.stdout:Downloading Packages: 2023-01-24T14:30:06.211 INFO:teuthology.orchestra.run.smithi039.stdout:(1/32): ceph-18.0.0-1954.g1a671c19.el8.x86_64.r 28 kB/s | 6.4 kB 00:00 2023-01-24T14:30:06.636 INFO:teuthology.orchestra.run.smithi039.stdout:(2/32): ceph-mgr-18.0.0-1954.g1a671c19.el8.x86_ 2.3 MB/s | 1.5 MB 00:00 2023-01-24T14:30:07.478 INFO:teuthology.orchestra.run.smithi039.stdout:(3/32): ceph-mon-18.0.0-1954.g1a671c19.el8.x86_ 5.8 MB/s | 7.4 MB 00:01 2023-01-24T14:30:07.554 INFO:teuthology.orchestra.run.smithi039.stdout:(4/32): ceph-mgr-modules-core-18.0.0-1954.g1a67 3.1 MB/s | 236 kB 00:00 2023-01-24T14:30:07.654 INFO:teuthology.orchestra.run.smithi039.stdout:(5/32): python3-mako-1.0.6-13.el8.noarch.rpm 1.5 MB/s | 157 kB 00:00 2023-01-24T14:30:07.704 INFO:teuthology.orchestra.run.smithi039.stdout:(6/32): python3-markupsafe-0.23-19.el8.x86_64.r 784 kB/s | 39 kB 00:00 2023-01-24T14:30:07.729 INFO:teuthology.orchestra.run.smithi039.stdout:(7/32): python3-pyOpenSSL-19.0.0-1.el8.noarch.r 4.0 MB/s | 103 kB 00:00 2023-01-24T14:30:07.755 INFO:teuthology.orchestra.run.smithi039.stdout:(8/32): python3-pytz-2017.2-9.el8.noarch.rpm 2.1 MB/s | 54 kB 00:00 2023-01-24T14:30:07.789 INFO:teuthology.orchestra.run.smithi039.stdout:(9/32): python3-werkzeug-0.12.2-4.el8.noarch.rp 13 MB/s | 457 kB 00:00 2023-01-24T14:30:07.814 INFO:teuthology.orchestra.run.smithi039.stdout:(10/32): python3-cffi-1.11.5-5.el8.x86_64.rpm 9.3 MB/s | 237 kB 00:00 2023-01-24T14:30:07.847 INFO:teuthology.orchestra.run.smithi039.stdout:(11/32): python3-cryptography-3.2.1-5.el8.x86_6 16 MB/s | 559 kB 00:00 2023-01-24T14:30:07.873 INFO:teuthology.orchestra.run.smithi039.stdout:(12/32): python3-pycparser-2.14-14.el8.noarch.r 4.2 MB/s | 109 kB 00:00 2023-01-24T14:30:08.056 INFO:teuthology.orchestra.run.smithi039.stdout:(13/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 238 kB/s | 44 kB 00:00 2023-01-24T14:30:08.298 INFO:teuthology.orchestra.run.smithi039.stdout:(14/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 766 kB/s | 185 kB 00:00 2023-01-24T14:30:08.557 INFO:teuthology.orchestra.run.smithi039.stdout:(15/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 669 kB/s | 173 kB 00:00 2023-01-24T14:30:08.774 INFO:teuthology.orchestra.run.smithi039.stdout:(16/32): python3-cherrypy-18.4.0-1.el8.noarch.r 1.7 MB/s | 384 kB 00:00 2023-01-24T14:30:08.858 INFO:teuthology.orchestra.run.smithi039.stdout:(17/32): python3-jaraco-6.2-6.el8.noarch.rpm 133 kB/s | 11 kB 00:00 2023-01-24T14:30:08.908 INFO:teuthology.orchestra.run.smithi039.stdout:(18/32): python3-jaraco-functools-2.0-4.el8.noa 352 kB/s | 18 kB 00:00 2023-01-24T14:30:08.959 INFO:teuthology.orchestra.run.smithi039.stdout:(19/32): python3-logutils-0.3.5-11.el8.noarch.r 967 kB/s | 49 kB 00:00 2023-01-24T14:30:09.009 INFO:teuthology.orchestra.run.smithi039.stdout:(20/32): python3-more-itertools-7.2.0-3.el8.noa 1.2 MB/s | 59 kB 00:00 2023-01-24T14:30:09.386 INFO:teuthology.orchestra.run.smithi039.stdout:(21/32): python3-pecan-1.3.2-9.el8.noarch.rpm 750 kB/s | 283 kB 00:00 2023-01-24T14:30:09.711 INFO:teuthology.orchestra.run.smithi039.stdout:(22/32): python3-portend-2.6-1.el8.noarch.rpm 48 kB/s | 16 kB 00:00 2023-01-24T14:30:10.244 INFO:teuthology.orchestra.run.smithi039.stdout:(23/32): python3-simplegeneric-0.8.1-17.el8.noa 35 kB/s | 19 kB 00:00 2023-01-24T14:30:10.502 INFO:teuthology.orchestra.run.smithi039.stdout:(24/32): python3-singledispatch-3.4.0.3-18.el8. 93 kB/s | 24 kB 00:00 2023-01-24T14:30:10.619 INFO:teuthology.orchestra.run.smithi039.stdout:(25/32): python3-tempora-1.14.1-5.el8.noarch.rp 251 kB/s | 29 kB 00:00 2023-01-24T14:30:10.670 INFO:teuthology.orchestra.run.smithi039.stdout:(26/32): python3-trustme-0.6.0-4.el8.noarch.rpm 553 kB/s | 27 kB 00:00 2023-01-24T14:30:10.836 INFO:teuthology.orchestra.run.smithi039.stdout:(27/32): python3-waitress-1.2.1-2.el8.1.noarch. 1.4 MB/s | 241 kB 00:00 2023-01-24T14:30:11.070 INFO:teuthology.orchestra.run.smithi039.stdout:(28/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 1.1 MB/s | 251 kB 00:00 2023-01-24T14:30:11.162 INFO:teuthology.orchestra.run.smithi039.stdout:(29/32): python3-webtest-2.0.33-1.el8.noarch.rp 933 kB/s | 85 kB 00:00 2023-01-24T14:30:11.221 INFO:teuthology.orchestra.run.smithi039.stdout:(30/32): python3-zc-lockfile-2.0-2.el8.noarch.r 394 kB/s | 23 kB 00:00 2023-01-24T14:35:12.642 INFO:teuthology.orchestra.run.smithi039.stdout:[MIRROR] ceph-mds-18.0.0-1954.g1a671c19.el8.x86_64.rpm: Curl error (28): Timeout was reached for https://1.chacra.ceph.com/r/ceph/main/1a671c198559f54389cd5afa22085a0c7b29a1ef/centos/8/flavors/crimson/x86_64/ceph-mds-18.0.0-1954.g1a671c19.el8.x86_64.rpm [Operation too slow. Less than 1000 bytes/sec transferred the last 300 seconds] 2023-01-24T14:38:15.859 INFO:teuthology.orchestra.run.smithi002.stdout:[MIRROR] ceph-osd-18.0.0-1954.g1a671c19.el8.x86_64.rpm: Curl error (18): Transferred a partial file for https://1.chacra.ceph.com/r/ceph/main/1a671c198559f54389cd5afa22085a0c7b29a1ef/centos/8/flavors/crimson/x86_64/ceph-osd-18.0.0-1954.g1a671c19.el8.x86_64.rpm [transfer closed with 8768621 bytes remaining to read] 2023-01-24T14:38:21.542 INFO:teuthology.orchestra.run.smithi039.stdout:(31/32): ceph-osd-18.0.0-1954.g1a671c19.el8.x86 312 kB/s | 151 MB 08:14 2023-01-24T14:38:23.100 INFO:teuthology.orchestra.run.smithi039.stdout:(32/32): ceph-mds-18.0.0-1954.g1a671c19.el8.x86 4.5 kB/s | 2.2 MB 08:17 2023-01-24T14:38:23.101 INFO:teuthology.orchestra.run.smithi039.stdout:-------------------------------------------------------------------------------- 2023-01-24T14:38:23.101 INFO:teuthology.orchestra.run.smithi039.stdout:Total 341 kB/s | 165 MB 08:17 2023-01-24T14:38:23.180 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction check 2023-01-24T14:38:23.223 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction check succeeded. 2023-01-24T14:38:23.223 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction test 2023-01-24T14:38:24.146 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction test succeeded. 2023-01-24T14:38:24.149 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction 2023-01-24T14:38:25.969 INFO:teuthology.orchestra.run.smithi039.stdout: Preparing : 1/1 2023-01-24T14:38:26.218 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2023-01-24T14:38:26.443 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2023-01-24T14:38:26.720 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2023-01-24T14:38:26.967 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-waitress-1.2.1-2.el8.1.noarch 4/32 2023-01-24T14:38:27.172 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2023-01-24T14:38:27.345 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2023-01-24T14:38:27.554 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2023-01-24T14:38:27.766 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2023-01-24T14:38:27.976 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2023-01-24T14:38:28.361 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2023-01-24T14:38:28.627 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2023-01-24T14:38:28.895 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2023-01-24T14:38:29.303 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 13/32 2023-01-24T14:38:29.572 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-cryptography-3.2.1-5.el8.x86_64 14/32 2023-01-24T14:38:29.751 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2023-01-24T14:38:29.988 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2023-01-24T14:38:30.219 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2023-01-24T14:38:30.533 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2023-01-24T14:38:30.757 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2023-01-24T14:38:30.962 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 20/32 2023-01-24T14:38:31.206 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2023-01-24T14:38:31.482 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2023-01-24T14:38:31.789 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2023-01-24T14:38:32.007 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2023-01-24T14:38:32.409 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 25/32 2023-01-24T14:38:32.794 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2023-01-24T14:38:33.251 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-1954.g1a671c19.el8. 27/32 2023-01-24T14:38:33.533 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-mgr-2:18.0.0-1954.g1a671c19.el8.x86_64 28/32 2023-01-24T14:39:02.303 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-mgr-2:18.0.0-1954.g1a671c19.el8.x86_64 28/32 2023-01-24T14:39:02.488 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-osd-2:18.0.0-1954.g1a671c19.el8.x86_64 29/32 2023-01-24T14:39:03.291 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-osd-2:18.0.0-1954.g1a671c19.el8.x86_64 29/32 2023-01-24T14:39:03.604 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-mon-2:18.0.0-1954.g1a671c19.el8.x86_64 30/32 2023-01-24T14:39:03.834 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-mon-2:18.0.0-1954.g1a671c19.el8.x86_64 30/32 2023-01-24T14:39:04.177 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-mds-2:18.0.0-1954.g1a671c19.el8.x86_64 31/32 2023-01-24T14:39:04.196 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-mds-2:18.0.0-1954.g1a671c19.el8.x86_64 31/32 2023-01-24T14:39:05.660 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-2:18.0.0-1954.g1a671c19.el8.x86_64 32/32 2023-01-24T14:39:06.726 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-2:18.0.0-1954.g1a671c19.el8.x86_64 32/32 2023-01-24T14:39:06.727 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-2:18.0.0-1954.g1a671c19.el8.x86_64 1/32 2023-01-24T14:39:06.727 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-mds-2:18.0.0-1954.g1a671c19.el8.x86_64 2/32 2023-01-24T14:39:06.727 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-mgr-2:18.0.0-1954.g1a671c19.el8.x86_64 3/32 2023-01-24T14:39:06.727 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-mon-2:18.0.0-1954.g1a671c19.el8.x86_64 4/32 2023-01-24T14:39:06.727 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-osd-2:18.0.0-1954.g1a671c19.el8.x86_64 5/32 2023-01-24T14:39:06.728 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-1954.g1a671c19.el8. 6/32 2023-01-24T14:39:06.728 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 7/32 2023-01-24T14:39:06.728 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2023-01-24T14:39:06.728 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2023-01-24T14:39:06.728 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 10/32 2023-01-24T14:39:06.729 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2023-01-24T14:39:06.729 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 12/32 2023-01-24T14:39:06.729 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-cryptography-3.2.1-5.el8.x86_64 13/32 2023-01-24T14:39:06.729 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2023-01-24T14:39:06.729 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2023-01-24T14:39:06.730 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2023-01-24T14:39:06.730 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2023-01-24T14:39:06.730 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2023-01-24T14:39:06.730 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2023-01-24T14:39:06.731 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2023-01-24T14:39:06.731 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2023-01-24T14:39:06.731 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2023-01-24T14:39:06.731 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2023-01-24T14:39:06.731 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2023-01-24T14:39:06.732 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2023-01-24T14:39:06.732 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2023-01-24T14:39:06.732 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2023-01-24T14:39:06.732 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2023-01-24T14:39:06.732 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-waitress-1.2.1-2.el8.1.noarch 29/32 2023-01-24T14:39:06.733 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2023-01-24T14:39:06.733 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2023-01-24T14:39:12.070 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2023-01-24T14:39:12.070 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-01-24T14:39:12.071 INFO:teuthology.orchestra.run.smithi039.stdout:Installed: 2023-01-24T14:39:12.071 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-2:18.0.0-1954.g1a671c19.el8.x86_64 2023-01-24T14:39:12.071 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mds-2:18.0.0-1954.g1a671c19.el8.x86_64 2023-01-24T14:39:12.071 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mgr-2:18.0.0-1954.g1a671c19.el8.x86_64 2023-01-24T14:39:12.071 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mgr-modules-core-2:18.0.0-1954.g1a671c19.el8.noarch 2023-01-24T14:39:12.072 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mon-2:18.0.0-1954.g1a671c19.el8.x86_64 2023-01-24T14:39:12.072 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-osd-2:18.0.0-1954.g1a671c19.el8.x86_64 2023-01-24T14:39:12.072 INFO:teuthology.orchestra.run.smithi039.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-01-24T14:39:12.072 INFO:teuthology.orchestra.run.smithi039.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-01-24T14:39:12.073 INFO:teuthology.orchestra.run.smithi039.stdout: python3-cffi-1.11.5-5.el8.x86_64 2023-01-24T14:39:12.073 INFO:teuthology.orchestra.run.smithi039.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-01-24T14:39:12.073 INFO:teuthology.orchestra.run.smithi039.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-01-24T14:39:12.074 INFO:teuthology.orchestra.run.smithi039.stdout: python3-cryptography-3.2.1-5.el8.x86_64 2023-01-24T14:39:12.074 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jaraco-6.2-6.el8.noarch 2023-01-24T14:39:12.074 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-01-24T14:39:12.075 INFO:teuthology.orchestra.run.smithi039.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-01-24T14:39:12.075 INFO:teuthology.orchestra.run.smithi039.stdout: python3-mako-1.0.6-13.el8.noarch 2023-01-24T14:39:12.075 INFO:teuthology.orchestra.run.smithi039.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-01-24T14:39:12.075 INFO:teuthology.orchestra.run.smithi039.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-01-24T14:39:12.075 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-01-24T14:39:12.076 INFO:teuthology.orchestra.run.smithi039.stdout: python3-portend-2.6-1.el8.noarch 2023-01-24T14:39:12.076 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-01-24T14:39:12.076 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pycparser-2.14-14.el8.noarch 2023-01-24T14:39:12.076 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pytz-2017.2-9.el8.noarch 2023-01-24T14:39:12.076 INFO:teuthology.orchestra.run.smithi039.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-01-24T14:39:12.077 INFO:teuthology.orchestra.run.smithi039.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-01-24T14:39:12.077 INFO:teuthology.orchestra.run.smithi039.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-01-24T14:39:12.077 INFO:teuthology.orchestra.run.smithi039.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-01-24T14:39:12.077 INFO:teuthology.orchestra.run.smithi039.stdout: python3-waitress-1.2.1-2.el8.1.noarch 2023-01-24T14:39:12.077 INFO:teuthology.orchestra.run.smithi039.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-01-24T14:39:12.078 INFO:teuthology.orchestra.run.smithi039.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-01-24T14:39:12.078 INFO:teuthology.orchestra.run.smithi039.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-01-24T14:39:12.078 INFO:teuthology.orchestra.run.smithi039.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-01-24T14:39:12.078 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-01-24T14:39:12.078 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2023-01-24T14:39:14.078 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum -y install ceph-base 2023-01-24T14:39:14.840 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:10:44 ago on Tue 24 Jan 2023 02:28:30 PM UTC. 2023-01-24T14:39:15.964 INFO:teuthology.orchestra.run.smithi039.stdout:Package ceph-base-2:18.0.0-1954.g1a671c19.el8.x86_64 is already installed. 2023-01-24T14:39:16.077 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2023-01-24T14:39:16.078 INFO:teuthology.orchestra.run.smithi039.stdout:Nothing to do. 2023-01-24T14:39:16.078 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2023-01-24T14:39:16.378 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum -y install cephadm 2023-01-24T14:39:17.139 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:10:47 ago on Tue 24 Jan 2023 02:28:30 PM UTC. 2023-01-24T14:39:18.356 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2023-01-24T14:39:18.357 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2023-01-24T14:39:18.357 INFO:teuthology.orchestra.run.smithi039.stdout: Package Arch Version Repository Size 2023-01-24T14:39:18.357 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2023-01-24T14:39:18.357 INFO:teuthology.orchestra.run.smithi039.stdout:Installing: 2023-01-24T14:39:18.358 INFO:teuthology.orchestra.run.smithi039.stdout: cephadm noarch 2:18.0.0-1954.g1a671c19.el8 ceph-noarch 82 k 2023-01-24T14:39:18.358 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-01-24T14:39:18.358 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction Summary 2023-01-24T14:39:18.358 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2023-01-24T14:39:18.359 INFO:teuthology.orchestra.run.smithi039.stdout:Install 1 Package 2023-01-24T14:39:18.359 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-01-24T14:39:18.359 INFO:teuthology.orchestra.run.smithi039.stdout:Total download size: 82 k 2023-01-24T14:39:18.359 INFO:teuthology.orchestra.run.smithi039.stdout:Installed size: 374 k 2023-01-24T14:39:18.359 INFO:teuthology.orchestra.run.smithi039.stdout:Downloading Packages: 2023-01-24T14:39:19.244 INFO:teuthology.orchestra.run.smithi039.stdout:cephadm-18.0.0-1954.g1a671c19.el8.noarch.rpm 93 kB/s | 82 kB 00:00 2023-01-24T14:39:19.268 INFO:teuthology.orchestra.run.smithi039.stdout:-------------------------------------------------------------------------------- 2023-01-24T14:39:19.268 INFO:teuthology.orchestra.run.smithi039.stdout:Total 90 kB/s | 82 kB 00:00 2023-01-24T14:39:19.269 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction check 2023-01-24T14:39:19.278 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction check succeeded. 2023-01-24T14:39:19.278 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction test 2023-01-24T14:39:19.281 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction test succeeded. 2023-01-24T14:39:19.285 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction 2023-01-24T14:39:19.914 INFO:teuthology.orchestra.run.smithi039.stdout: Preparing : 1/1 2023-01-24T14:39:22.473 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: cephadm-2:18.0.0-1954.g1a671c19.el8.noarch 1/1 2023-01-24T14:39:22.644 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : cephadm-2:18.0.0-1954.g1a671c19.el8.noarch 1/1 2023-01-24T14:39:22.895 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: cephadm-2:18.0.0-1954.g1a671c19.el8.noarch 1/1 2023-01-24T14:39:23.372 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : cephadm-2:18.0.0-1954.g1a671c19.el8.noarch 1/1 2023-01-24T14:39:23.372 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-01-24T14:39:23.372 INFO:teuthology.orchestra.run.smithi039.stdout:Installed: 2023-01-24T14:39:23.373 INFO:teuthology.orchestra.run.smithi039.stdout: cephadm-2:18.0.0-1954.g1a671c19.el8.noarch 2023-01-24T14:39:23.373 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-01-24T14:39:23.373 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2023-01-24T14:39:23.641 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum -y install ceph-immutable-object-cache 2023-01-24T14:39:24.403 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:10:54 ago on Tue 24 Jan 2023 02:28:30 PM UTC. 2023-01-24T14:39:25.624 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2023-01-24T14:39:25.625 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2023-01-24T14:39:25.625 INFO:teuthology.orchestra.run.smithi039.stdout: Package Arch Version Repo Size 2023-01-24T14:39:25.625 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2023-01-24T14:39:25.626 INFO:teuthology.orchestra.run.smithi039.stdout:Installing: 2023-01-24T14:39:25.626 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-1954.g1a671c19.el8 ceph 157 k 2023-01-24T14:39:25.626 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-01-24T14:39:25.626 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction Summary 2023-01-24T14:39:25.626 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2023-01-24T14:39:25.627 INFO:teuthology.orchestra.run.smithi039.stdout:Install 1 Package 2023-01-24T14:39:25.627 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-01-24T14:39:25.627 INFO:teuthology.orchestra.run.smithi039.stdout:Total download size: 157 k 2023-01-24T14:39:25.627 INFO:teuthology.orchestra.run.smithi039.stdout:Installed size: 540 k 2023-01-24T14:39:25.627 INFO:teuthology.orchestra.run.smithi039.stdout:Downloading Packages: 2023-01-24T14:39:26.398 INFO:teuthology.orchestra.run.smithi039.stdout:ceph-immutable-object-cache-18.0.0-1954.g1a671c 203 kB/s | 157 kB 00:00 2023-01-24T14:39:26.399 INFO:teuthology.orchestra.run.smithi039.stdout:-------------------------------------------------------------------------------- 2023-01-24T14:39:26.399 INFO:teuthology.orchestra.run.smithi039.stdout:Total 203 kB/s | 157 kB 00:00 2023-01-24T14:39:26.399 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction check 2023-01-24T14:39:26.411 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction check succeeded. 2023-01-24T14:39:26.411 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction test 2023-01-24T14:39:26.465 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction test succeeded. 2023-01-24T14:39:26.469 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction 2023-01-24T14:39:26.605 INFO:teuthology.orchestra.run.smithi039.stdout: Preparing : 1/1 2023-01-24T14:39:26.801 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-1954.g1a671c19. 1/1 2023-01-24T14:39:27.248 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-1954.g1a671c19. 1/1 2023-01-24T14:39:27.726 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-1954.g1a671c19. 1/1 2023-01-24T14:39:27.727 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-01-24T14:39:27.727 INFO:teuthology.orchestra.run.smithi039.stdout:Installed: 2023-01-24T14:39:27.728 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-immutable-object-cache-2:18.0.0-1954.g1a671c19.el8.x86_64 2023-01-24T14:39:27.728 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-01-24T14:39:27.729 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2023-01-24T14:39:27.912 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum -y install ceph-mgr 2023-01-24T14:39:28.675 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:10:58 ago on Tue 24 Jan 2023 02:28:30 PM UTC. 2023-01-24T14:39:29.784 INFO:teuthology.orchestra.run.smithi039.stdout:Package ceph-mgr-2:18.0.0-1954.g1a671c19.el8.x86_64 is already installed. 2023-01-24T14:39:29.896 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2023-01-24T14:39:29.898 INFO:teuthology.orchestra.run.smithi039.stdout:Nothing to do. 2023-01-24T14:39:29.898 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2023-01-24T14:39:29.971 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum -y install ceph-mgr-dashboard 2023-01-24T14:39:30.719 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:11:00 ago on Tue 24 Jan 2023 02:28:30 PM UTC. 2023-01-24T14:39:31.941 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2023-01-24T14:39:31.942 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2023-01-24T14:39:31.943 INFO:teuthology.orchestra.run.smithi039.stdout: Package Arch Version Repository Size 2023-01-24T14:39:31.943 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2023-01-24T14:39:31.943 INFO:teuthology.orchestra.run.smithi039.stdout:Installing: 2023-01-24T14:39:31.943 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mgr-dashboard noarch 2:18.0.0-1954.g1a671c19.el8 ceph-noarch 1.7 M 2023-01-24T14:39:31.943 INFO:teuthology.orchestra.run.smithi039.stdout:Installing dependencies: 2023-01-24T14:39:31.944 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-grafana-dashboards noarch 2:18.0.0-1954.g1a671c19.el8 ceph-noarch 23 k 2023-01-24T14:39:31.944 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-prometheus-alerts noarch 2:18.0.0-1954.g1a671c19.el8 ceph-noarch 15 k 2023-01-24T14:39:31.944 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jwt noarch 1.6.1-2.el8 CentOS-BaseOS 43 k 2023-01-24T14:39:31.944 INFO:teuthology.orchestra.run.smithi039.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-01-24T14:39:31.944 INFO:teuthology.orchestra.run.smithi039.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-01-24T14:39:31.945 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-01-24T14:39:31.945 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction Summary 2023-01-24T14:39:31.945 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2023-01-24T14:39:31.945 INFO:teuthology.orchestra.run.smithi039.stdout:Install 6 Packages 2023-01-24T14:39:31.945 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-01-24T14:39:31.946 INFO:teuthology.orchestra.run.smithi039.stdout:Total download size: 2.0 M 2023-01-24T14:39:31.946 INFO:teuthology.orchestra.run.smithi039.stdout:Installed size: 8.2 M 2023-01-24T14:39:31.946 INFO:teuthology.orchestra.run.smithi039.stdout:Downloading Packages: 2023-01-24T14:39:32.269 INFO:teuthology.orchestra.run.smithi039.stdout:(1/6): ceph-grafana-dashboards-18.0.0-1954.g1a6 71 kB/s | 23 kB 00:00 2023-01-24T14:39:32.293 INFO:teuthology.orchestra.run.smithi039.stdout:(2/6): ceph-prometheus-alerts-18.0.0-1954.g1a67 42 kB/s | 15 kB 00:00 2023-01-24T14:39:32.352 INFO:teuthology.orchestra.run.smithi039.stdout:(3/6): python3-jwt-1.6.1-2.el8.noarch.rpm 509 kB/s | 43 kB 00:00 2023-01-24T14:39:32.452 INFO:teuthology.orchestra.run.smithi039.stdout:(4/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 212 kB/s | 34 kB 00:00 2023-01-24T14:39:32.585 INFO:teuthology.orchestra.run.smithi039.stdout:(5/6): python3-routes-2.4.1-12.el8.noarch.rpm 841 kB/s | 196 kB 00:00 2023-01-24T14:39:32.869 INFO:teuthology.orchestra.run.smithi039.stdout:(6/6): ceph-mgr-dashboard-18.0.0-1954.g1a671c19 1.9 MB/s | 1.7 MB 00:00 2023-01-24T14:39:32.872 INFO:teuthology.orchestra.run.smithi039.stdout:-------------------------------------------------------------------------------- 2023-01-24T14:39:32.872 INFO:teuthology.orchestra.run.smithi039.stdout:Total 2.2 MB/s | 2.0 MB 00:00 2023-01-24T14:39:32.883 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction check 2023-01-24T14:39:32.893 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction check succeeded. 2023-01-24T14:39:32.893 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction test 2023-01-24T14:39:32.984 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction test succeeded. 2023-01-24T14:39:32.988 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction 2023-01-24T14:39:33.265 INFO:teuthology.orchestra.run.smithi039.stdout: Preparing : 1/1 2023-01-24T14:39:33.541 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2023-01-24T14:39:33.760 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2023-01-24T14:39:33.986 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2023-01-24T14:39:34.115 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-1954.g1a671c19.el8.n 4/6 2023-01-24T14:39:34.534 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-1954.g1a671c19.el8. 5/6 2023-01-24T14:39:34.792 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-1954.g1a671c19.el8.noarc 6/6 2023-01-24T14:39:35.108 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-1954.g1a671c19.el8.noarc 6/6 2023-01-24T14:39:35.108 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-1954.g1a671c19.el8. 1/6 2023-01-24T14:39:35.109 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-1954.g1a671c19.el8.noarc 2/6 2023-01-24T14:39:35.109 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-1954.g1a671c19.el8.n 3/6 2023-01-24T14:39:35.109 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2023-01-24T14:39:35.109 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2023-01-24T14:39:35.624 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2023-01-24T14:39:35.625 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-01-24T14:39:35.625 INFO:teuthology.orchestra.run.smithi039.stdout:Installed: 2023-01-24T14:39:35.625 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-grafana-dashboards-2:18.0.0-1954.g1a671c19.el8.noarch 2023-01-24T14:39:35.625 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mgr-dashboard-2:18.0.0-1954.g1a671c19.el8.noarch 2023-01-24T14:39:35.625 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-prometheus-alerts-2:18.0.0-1954.g1a671c19.el8.noarch 2023-01-24T14:39:35.626 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-01-24T14:39:35.626 INFO:teuthology.orchestra.run.smithi039.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-01-24T14:39:35.626 INFO:teuthology.orchestra.run.smithi039.stdout: python3-routes-2.4.1-12.el8.noarch 2023-01-24T14:39:35.626 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-01-24T14:39:35.626 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2023-01-24T14:39:35.759 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum -y install ceph-mgr-diskprediction-local 2023-01-24T14:39:36.516 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:11:06 ago on Tue 24 Jan 2023 02:28:30 PM UTC. 2023-01-24T14:39:37.246 INFO:teuthology.orchestra.run.smithi002.stdout:(32/32): ceph-osd-18.0.0-1954.g1a671c19.el8.x86 267 kB/s | 151 MB 09:37 2023-01-24T14:39:37.248 INFO:teuthology.orchestra.run.smithi002.stdout:-------------------------------------------------------------------------------- 2023-01-24T14:39:37.248 INFO:teuthology.orchestra.run.smithi002.stdout:Total 293 kB/s | 165 MB 09:38 2023-01-24T14:39:37.328 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction check 2023-01-24T14:39:37.371 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction check succeeded. 2023-01-24T14:39:37.371 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction test 2023-01-24T14:39:37.758 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2023-01-24T14:39:37.759 INFO:teuthology.orchestra.run.smithi039.stdout:===================================================================================================== 2023-01-24T14:39:37.759 INFO:teuthology.orchestra.run.smithi039.stdout: Package Arch Version Repository Size 2023-01-24T14:39:37.760 INFO:teuthology.orchestra.run.smithi039.stdout:===================================================================================================== 2023-01-24T14:39:37.760 INFO:teuthology.orchestra.run.smithi039.stdout:Installing: 2023-01-24T14:39:37.760 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-1954.g1a671c19.el8 ceph-noarch 7.4 M 2023-01-24T14:39:37.760 INFO:teuthology.orchestra.run.smithi039.stdout:Installing dependencies: 2023-01-24T14:39:37.760 INFO:teuthology.orchestra.run.smithi039.stdout: libgfortran x86_64 8.5.0-18.el8 CentOS-BaseOS 645 k 2023-01-24T14:39:37.761 INFO:teuthology.orchestra.run.smithi039.stdout: libquadmath x86_64 8.5.0-18.el8 CentOS-BaseOS 172 k 2023-01-24T14:39:37.761 INFO:teuthology.orchestra.run.smithi039.stdout: openblas x86_64 0.3.15-4.el8 CentOS-AppStream 4.7 M 2023-01-24T14:39:37.761 INFO:teuthology.orchestra.run.smithi039.stdout: openblas-threads x86_64 0.3.15-4.el8 CentOS-AppStream 4.9 M 2023-01-24T14:39:37.761 INFO:teuthology.orchestra.run.smithi039.stdout: python3-numpy x86_64 1:1.14.3-10.el8 CentOS-AppStream 3.7 M 2023-01-24T14:39:37.761 INFO:teuthology.orchestra.run.smithi039.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 CentOS-AppStream 225 k 2023-01-24T14:39:37.762 INFO:teuthology.orchestra.run.smithi039.stdout: python3-scipy x86_64 1.0.0-21.module_el8.5.0+771+e5d9a225 CentOS-AppStream 14 M 2023-01-24T14:39:37.762 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-01-24T14:39:37.762 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction Summary 2023-01-24T14:39:37.762 INFO:teuthology.orchestra.run.smithi039.stdout:===================================================================================================== 2023-01-24T14:39:37.762 INFO:teuthology.orchestra.run.smithi039.stdout:Install 8 Packages 2023-01-24T14:39:37.763 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-01-24T14:39:37.763 INFO:teuthology.orchestra.run.smithi039.stdout:Total download size: 35 M 2023-01-24T14:39:37.764 INFO:teuthology.orchestra.run.smithi039.stdout:Installed size: 210 M 2023-01-24T14:39:37.764 INFO:teuthology.orchestra.run.smithi039.stdout:Downloading Packages: 2023-01-24T14:39:37.934 INFO:teuthology.orchestra.run.smithi039.stdout:(1/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 28 MB/s | 4.9 MB 00:00 2023-01-24T14:39:38.051 INFO:teuthology.orchestra.run.smithi039.stdout:(2/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 32 MB/s | 3.7 MB 00:00 2023-01-24T14:39:38.076 INFO:teuthology.orchestra.run.smithi039.stdout:(3/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 8.8 MB/s | 225 kB 00:00 2023-01-24T14:39:38.143 INFO:teuthology.orchestra.run.smithi039.stdout:(4/8): openblas-0.3.15-4.el8.x86_64.rpm 12 MB/s | 4.7 MB 00:00 2023-01-24T14:39:38.202 INFO:teuthology.orchestra.run.smithi039.stdout:(5/8): libgfortran-8.5.0-18.el8.x86_64.rpm 11 MB/s | 645 kB 00:00 2023-01-24T14:39:38.235 INFO:teuthology.orchestra.run.smithi039.stdout:(6/8): libquadmath-8.5.0-18.el8.x86_64.rpm 5.0 MB/s | 172 kB 00:00 2023-01-24T14:39:38.301 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction test succeeded. 2023-01-24T14:39:38.304 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction 2023-01-24T14:39:38.461 INFO:teuthology.orchestra.run.smithi039.stdout:(7/8): python3-scipy-1.0.0-21.module_el8.5.0+77 36 MB/s | 14 MB 00:00 2023-01-24T14:39:39.561 INFO:teuthology.orchestra.run.smithi039.stdout:(8/8): ceph-mgr-diskprediction-local-18.0.0-195 4.1 MB/s | 7.4 MB 00:01 2023-01-24T14:39:39.563 INFO:teuthology.orchestra.run.smithi039.stdout:-------------------------------------------------------------------------------- 2023-01-24T14:39:39.564 INFO:teuthology.orchestra.run.smithi039.stdout:Total 20 MB/s | 35 MB 00:01 2023-01-24T14:39:39.805 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction check 2023-01-24T14:39:39.824 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction check succeeded. 2023-01-24T14:39:39.825 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction test 2023-01-24T14:39:40.132 INFO:teuthology.orchestra.run.smithi002.stdout: Preparing : 1/1 2023-01-24T14:39:40.252 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction test succeeded. 2023-01-24T14:39:40.256 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction 2023-01-24T14:39:40.407 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2023-01-24T14:39:40.623 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2023-01-24T14:39:40.848 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2023-01-24T14:39:40.886 INFO:teuthology.orchestra.run.smithi039.stdout: Preparing : 1/1 2023-01-24T14:39:41.040 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : libquadmath-8.5.0-18.el8.x86_64 1/8 2023-01-24T14:39:41.113 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-waitress-1.2.1-2.el8.1.noarch 4/32 2023-01-24T14:39:41.187 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: libquadmath-8.5.0-18.el8.x86_64 1/8 2023-01-24T14:39:41.302 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2023-01-24T14:39:41.348 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : libgfortran-8.5.0-18.el8.x86_64 2/8 2023-01-24T14:39:41.492 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2023-01-24T14:39:41.709 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2023-01-24T14:39:41.887 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2023-01-24T14:39:41.970 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: libgfortran-8.5.0-18.el8.x86_64 2/8 2023-01-24T14:39:42.088 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2023-01-24T14:39:42.165 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2023-01-24T14:39:42.408 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2023-01-24T14:39:42.664 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2023-01-24T14:39:42.798 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2023-01-24T14:39:42.925 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2023-01-24T14:39:42.932 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2023-01-24T14:39:43.346 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 13/32 2023-01-24T14:39:43.652 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-cryptography-3.2.1-5.el8.x86_64 14/32 2023-01-24T14:39:43.666 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2023-01-24T14:39:43.864 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2023-01-24T14:39:44.056 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2023-01-24T14:39:44.161 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2023-01-24T14:39:44.306 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-01-24T14:39:44.383 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2023-01-24T14:39:44.707 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2023-01-24T14:39:44.921 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2023-01-24T14:39:45.126 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 20/32 2023-01-24T14:39:45.412 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2023-01-24T14:39:45.698 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2023-01-24T14:39:45.952 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2023-01-24T14:39:46.192 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-01-24T14:39:46.233 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2023-01-24T14:39:46.605 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 25/32 2023-01-24T14:39:46.991 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2023-01-24T14:39:47.382 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-1954.g1a671c19.el8. 27/32 2023-01-24T14:39:47.495 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 7/8 2023-01-24T14:39:47.681 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-1954.g1a671c1 8/8 2023-01-24T14:39:47.720 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : ceph-mgr-2:18.0.0-1954.g1a671c19.el8.x86_64 28/32 2023-01-24T14:39:48.060 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-1954.g1a671c1 8/8 2023-01-24T14:39:48.060 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-1954.g1a671c1 1/8 2023-01-24T14:39:48.061 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2023-01-24T14:39:48.061 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2023-01-24T14:39:48.061 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2023-01-24T14:39:48.062 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2023-01-24T14:39:48.062 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 6/8 2023-01-24T14:39:48.062 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libgfortran-8.5.0-18.el8.x86_64 7/8 2023-01-24T14:39:48.599 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libquadmath-8.5.0-18.el8.x86_64 8/8 2023-01-24T14:39:48.599 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-01-24T14:39:48.599 INFO:teuthology.orchestra.run.smithi039.stdout:Installed: 2023-01-24T14:39:48.599 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mgr-diskprediction-local-2:18.0.0-1954.g1a671c19.el8.noarch 2023-01-24T14:39:48.600 INFO:teuthology.orchestra.run.smithi039.stdout: libgfortran-8.5.0-18.el8.x86_64 2023-01-24T14:39:48.600 INFO:teuthology.orchestra.run.smithi039.stdout: libquadmath-8.5.0-18.el8.x86_64 2023-01-24T14:39:48.600 INFO:teuthology.orchestra.run.smithi039.stdout: openblas-0.3.15-4.el8.x86_64 2023-01-24T14:39:48.600 INFO:teuthology.orchestra.run.smithi039.stdout: openblas-threads-0.3.15-4.el8.x86_64 2023-01-24T14:39:48.600 INFO:teuthology.orchestra.run.smithi039.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-01-24T14:39:48.601 INFO:teuthology.orchestra.run.smithi039.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-01-24T14:39:48.601 INFO:teuthology.orchestra.run.smithi039.stdout: python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x86_64 2023-01-24T14:39:48.601 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-01-24T14:39:48.601 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2023-01-24T14:39:48.722 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum -y install ceph-mgr-rook 2023-01-24T14:39:49.491 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:11:19 ago on Tue 24 Jan 2023 02:28:30 PM UTC. 2023-01-24T14:39:50.731 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2023-01-24T14:39:50.733 INFO:teuthology.orchestra.run.smithi039.stdout:======================================================================================== 2023-01-24T14:39:50.733 INFO:teuthology.orchestra.run.smithi039.stdout: Package Arch Version Repository Size 2023-01-24T14:39:50.734 INFO:teuthology.orchestra.run.smithi039.stdout:======================================================================================== 2023-01-24T14:39:50.734 INFO:teuthology.orchestra.run.smithi039.stdout:Installing: 2023-01-24T14:39:50.734 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mgr-rook noarch 2:18.0.0-1954.g1a671c19.el8 ceph-noarch 47 k 2023-01-24T14:39:50.734 INFO:teuthology.orchestra.run.smithi039.stdout:Installing dependencies: 2023-01-24T14:39:50.734 INFO:teuthology.orchestra.run.smithi039.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-01-24T14:39:50.735 INFO:teuthology.orchestra.run.smithi039.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-01-24T14:39:50.735 INFO:teuthology.orchestra.run.smithi039.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-01-24T14:39:50.735 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jsonpatch noarch 1.21-2.el8 CentOS-AppStream 27 k 2023-01-24T14:39:50.735 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jsonpointer noarch 1.10-11.el8 CentOS-AppStream 20 k 2023-01-24T14:39:50.735 INFO:teuthology.orchestra.run.smithi039.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-01-24T14:39:50.736 INFO:teuthology.orchestra.run.smithi039.stdout: python3-oauthlib noarch 2.1.0-1.el8 CentOS-BaseOS 155 k 2023-01-24T14:39:50.736 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pyasn1 noarch 0.3.7-6.el8 CentOS-AppStream 126 k 2023-01-24T14:39:50.736 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 CentOS-AppStream 110 k 2023-01-24T14:39:50.736 INFO:teuthology.orchestra.run.smithi039.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 CentOS-BaseOS 43 k 2023-01-24T14:39:50.736 INFO:teuthology.orchestra.run.smithi039.stdout: python3-rsa noarch 4.8-1.el8 epel 61 k 2023-01-24T14:39:50.737 INFO:teuthology.orchestra.run.smithi039.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-01-24T14:39:50.737 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-01-24T14:39:50.737 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction Summary 2023-01-24T14:39:50.737 INFO:teuthology.orchestra.run.smithi039.stdout:======================================================================================== 2023-01-24T14:39:50.737 INFO:teuthology.orchestra.run.smithi039.stdout:Install 13 Packages 2023-01-24T14:39:50.738 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-01-24T14:39:50.738 INFO:teuthology.orchestra.run.smithi039.stdout:Total download size: 1.9 M 2023-01-24T14:39:50.739 INFO:teuthology.orchestra.run.smithi039.stdout:Installed size: 25 M 2023-01-24T14:39:50.739 INFO:teuthology.orchestra.run.smithi039.stdout:Downloading Packages: 2023-01-24T14:39:50.803 INFO:teuthology.orchestra.run.smithi039.stdout:(1/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 404 kB/s | 27 kB 00:00 2023-01-24T14:39:50.837 INFO:teuthology.orchestra.run.smithi039.stdout:(2/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 3.9 MB/s | 126 kB 00:00 2023-01-24T14:39:50.862 INFO:teuthology.orchestra.run.smithi039.stdout:(3/13): python3-jsonpointer-1.10-11.el8.noarch. 160 kB/s | 20 kB 00:00 2023-01-24T14:39:50.888 INFO:teuthology.orchestra.run.smithi039.stdout:(4/13): python3-pyasn1-modules-0.3.7-6.el8.noar 2.1 MB/s | 110 kB 00:00 2023-01-24T14:39:50.913 INFO:teuthology.orchestra.run.smithi039.stdout:(5/13): python3-requests-oauthlib-1.0.0-1.el8.n 1.7 MB/s | 43 kB 00:00 2023-01-24T14:39:50.938 INFO:teuthology.orchestra.run.smithi039.stdout:(6/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 2.0 MB/s | 155 kB 00:00 2023-01-24T14:39:51.138 INFO:teuthology.orchestra.run.smithi039.stdout:(7/13): python3-cachetools-3.1.1-4.el8.noarch.r 146 kB/s | 33 kB 00:00 2023-01-24T14:39:51.164 INFO:teuthology.orchestra.run.smithi039.stdout:(8/13): python3-certifi-2018.10.15-7.el8.noarch 72 kB/s | 16 kB 00:00 2023-01-24T14:39:51.206 INFO:teuthology.orchestra.run.smithi039.stdout:(9/13): ceph-mgr-rook-18.0.0-1954.g1a671c19.el8 101 kB/s | 47 kB 00:00 2023-01-24T14:39:51.262 INFO:teuthology.orchestra.run.smithi039.stdout:(10/13): python3-google-auth-1.1.1-10.el8.noarc 657 kB/s | 81 kB 00:00 2023-01-24T14:39:51.323 INFO:teuthology.orchestra.run.smithi039.stdout:(11/13): python3-websocket-client-0.56.0-5.el8. 1.0 MB/s | 61 kB 00:00 2023-01-24T14:39:51.381 INFO:teuthology.orchestra.run.smithi039.stdout:(12/13): python3-rsa-4.8-1.el8.noarch.rpm 350 kB/s | 61 kB 00:00 2023-01-24T14:39:51.474 INFO:teuthology.orchestra.run.smithi039.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 3.8 MB/s | 1.2 MB 00:00 2023-01-24T14:39:51.475 INFO:teuthology.orchestra.run.smithi039.stdout:-------------------------------------------------------------------------------- 2023-01-24T14:39:51.475 INFO:teuthology.orchestra.run.smithi039.stdout:Total 2.6 MB/s | 1.9 MB 00:00 2023-01-24T14:39:51.536 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction check 2023-01-24T14:39:51.551 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction check succeeded. 2023-01-24T14:39:51.551 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction test 2023-01-24T14:39:51.659 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction test succeeded. 2023-01-24T14:39:51.663 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction 2023-01-24T14:39:52.201 INFO:teuthology.orchestra.run.smithi039.stdout: Preparing : 1/1 2023-01-24T14:39:52.604 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2023-01-24T14:39:52.792 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2023-01-24T14:39:53.026 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-rsa-4.8-1.el8.noarch 3/13 2023-01-24T14:39:53.271 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2023-01-24T14:39:53.487 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2023-01-24T14:39:53.797 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2023-01-24T14:39:54.159 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2023-01-24T14:39:54.420 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2023-01-24T14:39:55.536 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2023-01-24T14:39:55.862 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2023-01-24T14:39:56.137 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2023-01-24T14:39:56.373 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2023-01-24T14:39:56.571 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-mgr-rook-2:18.0.0-1954.g1a671c19.el8.noarch 13/13 2023-01-24T14:39:56.905 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-1954.g1a671c19.el8.noarch 13/13 2023-01-24T14:39:56.906 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-mgr-rook-2:18.0.0-1954.g1a671c19.el8.noarch 1/13 2023-01-24T14:39:56.906 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2023-01-24T14:39:56.906 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2023-01-24T14:39:56.906 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2023-01-24T14:39:56.906 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2023-01-24T14:39:56.907 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2023-01-24T14:39:56.907 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2023-01-24T14:39:56.907 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2023-01-24T14:39:56.907 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2023-01-24T14:39:56.907 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2023-01-24T14:39:56.908 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2023-01-24T14:39:56.908 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-rsa-4.8-1.el8.noarch 12/13 2023-01-24T14:39:57.478 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2023-01-24T14:39:57.479 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-01-24T14:39:57.479 INFO:teuthology.orchestra.run.smithi039.stdout:Installed: 2023-01-24T14:39:57.479 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mgr-rook-2:18.0.0-1954.g1a671c19.el8.noarch 2023-01-24T14:39:57.479 INFO:teuthology.orchestra.run.smithi039.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-01-24T14:39:57.479 INFO:teuthology.orchestra.run.smithi039.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-01-24T14:39:57.480 INFO:teuthology.orchestra.run.smithi039.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-01-24T14:39:57.480 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-01-24T14:39:57.480 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-01-24T14:39:57.480 INFO:teuthology.orchestra.run.smithi039.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-01-24T14:39:57.480 INFO:teuthology.orchestra.run.smithi039.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-01-24T14:39:57.481 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-01-24T14:39:57.481 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-01-24T14:39:57.481 INFO:teuthology.orchestra.run.smithi039.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-01-24T14:39:57.481 INFO:teuthology.orchestra.run.smithi039.stdout: python3-rsa-4.8-1.el8.noarch 2023-01-24T14:39:57.481 INFO:teuthology.orchestra.run.smithi039.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-01-24T14:39:57.482 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-01-24T14:39:57.482 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2023-01-24T14:39:57.639 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum -y install ceph-mgr-cephadm 2023-01-24T14:39:58.405 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:11:28 ago on Tue 24 Jan 2023 02:28:30 PM UTC. 2023-01-24T14:39:59.628 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2023-01-24T14:39:59.629 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2023-01-24T14:39:59.629 INFO:teuthology.orchestra.run.smithi039.stdout: Package Arch Version Repository Size 2023-01-24T14:39:59.629 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2023-01-24T14:39:59.630 INFO:teuthology.orchestra.run.smithi039.stdout:Installing: 2023-01-24T14:39:59.630 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mgr-cephadm noarch 2:18.0.0-1954.g1a671c19.el8 ceph-noarch 120 k 2023-01-24T14:39:59.630 INFO:teuthology.orchestra.run.smithi039.stdout:Installing dependencies: 2023-01-24T14:39:59.630 INFO:teuthology.orchestra.run.smithi039.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-01-24T14:39:59.630 INFO:teuthology.orchestra.run.smithi039.stdout: python3-babel noarch 2.5.1-7.el8 CentOS-AppStream 4.8 M 2023-01-24T14:39:59.631 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jinja2 noarch 2.10.1-3.el8 CentOS-AppStream 538 k 2023-01-24T14:39:59.631 INFO:teuthology.orchestra.run.smithi039.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-01-24T14:39:59.631 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-01-24T14:39:59.631 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction Summary 2023-01-24T14:39:59.631 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2023-01-24T14:39:59.632 INFO:teuthology.orchestra.run.smithi039.stdout:Install 5 Packages 2023-01-24T14:39:59.632 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-01-24T14:39:59.632 INFO:teuthology.orchestra.run.smithi039.stdout:Total download size: 6.0 M 2023-01-24T14:39:59.633 INFO:teuthology.orchestra.run.smithi039.stdout:Installed size: 26 M 2023-01-24T14:39:59.633 INFO:teuthology.orchestra.run.smithi039.stdout:Downloading Packages: 2023-01-24T14:39:59.716 INFO:teuthology.orchestra.run.smithi039.stdout:(1/5): python3-jinja2-2.10.1-3.el8.noarch.rpm 6.3 MB/s | 538 kB 00:00 2023-01-24T14:39:59.808 INFO:teuthology.orchestra.run.smithi039.stdout:(2/5): python3-babel-2.5.1-7.el8.noarch.rpm 27 MB/s | 4.8 MB 00:00 2023-01-24T14:39:59.983 INFO:teuthology.orchestra.run.smithi039.stdout:(3/5): python3-natsort-7.1.1-2.el8.noarch.rpm 344 kB/s | 60 kB 00:00 2023-01-24T14:40:00.008 INFO:teuthology.orchestra.run.smithi039.stdout:(4/5): python3-asyncssh-2.7.0-2.el8.noarch.rpm 1.6 MB/s | 476 kB 00:00 2023-01-24T14:40:00.234 INFO:teuthology.orchestra.run.smithi039.stdout:(5/5): ceph-mgr-cephadm-18.0.0-1954.g1a671c19.e 199 kB/s | 120 kB 00:00 2023-01-24T14:40:00.235 INFO:teuthology.orchestra.run.smithi039.stdout:-------------------------------------------------------------------------------- 2023-01-24T14:40:00.235 INFO:teuthology.orchestra.run.smithi039.stdout:Total 9.8 MB/s | 6.0 MB 00:00 2023-01-24T14:40:00.289 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction check 2023-01-24T14:40:00.301 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction check succeeded. 2023-01-24T14:40:00.301 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction test 2023-01-24T14:40:00.440 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction test succeeded. 2023-01-24T14:40:00.444 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction 2023-01-24T14:40:00.784 INFO:teuthology.orchestra.run.smithi039.stdout: Preparing : 1/1 2023-01-24T14:40:01.152 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 1/5 2023-01-24T14:40:02.201 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 2/5 2023-01-24T14:40:02.641 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 3/5 2023-01-24T14:40:02.943 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 4/5 2023-01-24T14:40:03.145 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-1954.g1a671c19.el8.noarch 5/5 2023-01-24T14:40:03.447 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-1954.g1a671c19.el8.noarch 5/5 2023-01-24T14:40:03.447 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-1954.g1a671c19.el8.noarch 1/5 2023-01-24T14:40:03.447 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/5 2023-01-24T14:40:03.448 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 3/5 2023-01-24T14:40:03.448 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 4/5 2023-01-24T14:40:03.976 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 5/5 2023-01-24T14:40:03.977 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-01-24T14:40:03.977 INFO:teuthology.orchestra.run.smithi039.stdout:Installed: 2023-01-24T14:40:03.977 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mgr-cephadm-2:18.0.0-1954.g1a671c19.el8.noarch 2023-01-24T14:40:03.978 INFO:teuthology.orchestra.run.smithi039.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-01-24T14:40:03.978 INFO:teuthology.orchestra.run.smithi039.stdout: python3-babel-2.5.1-7.el8.noarch 2023-01-24T14:40:03.978 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jinja2-2.10.1-3.el8.noarch 2023-01-24T14:40:03.978 INFO:teuthology.orchestra.run.smithi039.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-01-24T14:40:03.978 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-01-24T14:40:03.979 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2023-01-24T14:40:04.093 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum -y install ceph-fuse 2023-01-24T14:40:04.855 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:11:34 ago on Tue 24 Jan 2023 02:28:30 PM UTC. 2023-01-24T14:40:06.095 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2023-01-24T14:40:06.095 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2023-01-24T14:40:06.096 INFO:teuthology.orchestra.run.smithi039.stdout: Package Arch Version Repository Size 2023-01-24T14:40:06.096 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2023-01-24T14:40:06.096 INFO:teuthology.orchestra.run.smithi039.stdout:Installing: 2023-01-24T14:40:06.096 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-fuse x86_64 2:18.0.0-1954.g1a671c19.el8 ceph 814 k 2023-01-24T14:40:06.096 INFO:teuthology.orchestra.run.smithi039.stdout:Installing dependencies: 2023-01-24T14:40:06.097 INFO:teuthology.orchestra.run.smithi039.stdout: fuse x86_64 2.9.7-16.el8 CentOS-BaseOS 83 k 2023-01-24T14:40:06.097 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-01-24T14:40:06.097 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction Summary 2023-01-24T14:40:06.097 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2023-01-24T14:40:06.097 INFO:teuthology.orchestra.run.smithi039.stdout:Install 2 Packages 2023-01-24T14:40:06.098 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-01-24T14:40:06.098 INFO:teuthology.orchestra.run.smithi039.stdout:Total download size: 897 k 2023-01-24T14:40:06.099 INFO:teuthology.orchestra.run.smithi039.stdout:Installed size: 3.3 M 2023-01-24T14:40:06.099 INFO:teuthology.orchestra.run.smithi039.stdout:Downloading Packages: 2023-01-24T14:40:06.175 INFO:teuthology.orchestra.run.smithi039.stdout:(1/2): fuse-2.9.7-16.el8.x86_64.rpm 1.0 MB/s | 83 kB 00:00 2023-01-24T14:40:07.242 INFO:teuthology.orchestra.run.smithi039.stdout:(2/2): ceph-fuse-18.0.0-1954.g1a671c19.el8.x86_ 711 kB/s | 814 kB 00:01 2023-01-24T14:40:07.243 INFO:teuthology.orchestra.run.smithi039.stdout:-------------------------------------------------------------------------------- 2023-01-24T14:40:07.244 INFO:teuthology.orchestra.run.smithi039.stdout:Total 782 kB/s | 897 kB 00:01 2023-01-24T14:40:07.256 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction check 2023-01-24T14:40:07.280 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction check succeeded. 2023-01-24T14:40:07.280 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction test 2023-01-24T14:40:07.388 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction test succeeded. 2023-01-24T14:40:07.392 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction 2023-01-24T14:40:07.624 INFO:teuthology.orchestra.run.smithi039.stdout: Preparing : 1/1 2023-01-24T14:40:07.933 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : fuse-2.9.7-16.el8.x86_64 1/2 2023-01-24T14:40:08.531 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-fuse-2:18.0.0-1954.g1a671c19.el8.x86_64 2/2 2023-01-24T14:40:09.131 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-fuse-2:18.0.0-1954.g1a671c19.el8.x86_64 2/2 2023-01-24T14:40:09.132 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-fuse-2:18.0.0-1954.g1a671c19.el8.x86_64 1/2 2023-01-24T14:40:09.670 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : fuse-2.9.7-16.el8.x86_64 2/2 2023-01-24T14:40:09.670 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-01-24T14:40:09.671 INFO:teuthology.orchestra.run.smithi039.stdout:Installed: 2023-01-24T14:40:09.671 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-fuse-2:18.0.0-1954.g1a671c19.el8.x86_64 fuse-2.9.7-16.el8.x86_64 2023-01-24T14:40:09.671 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-01-24T14:40:09.671 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2023-01-24T14:40:09.872 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum -y install ceph-volume 2023-01-24T14:40:10.644 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:11:40 ago on Tue 24 Jan 2023 02:28:30 PM UTC. 2023-01-24T14:40:11.868 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2023-01-24T14:40:11.868 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2023-01-24T14:40:11.869 INFO:teuthology.orchestra.run.smithi039.stdout: Package Arch Version Repository Size 2023-01-24T14:40:11.869 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2023-01-24T14:40:11.869 INFO:teuthology.orchestra.run.smithi039.stdout:Installing: 2023-01-24T14:40:11.869 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-volume noarch 2:18.0.0-1954.g1a671c19.el8 ceph-noarch 263 k 2023-01-24T14:40:11.870 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-01-24T14:40:11.870 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction Summary 2023-01-24T14:40:11.870 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2023-01-24T14:40:11.870 INFO:teuthology.orchestra.run.smithi039.stdout:Install 1 Package 2023-01-24T14:40:11.870 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-01-24T14:40:11.870 INFO:teuthology.orchestra.run.smithi039.stdout:Total download size: 263 k 2023-01-24T14:40:11.871 INFO:teuthology.orchestra.run.smithi039.stdout:Installed size: 1.3 M 2023-01-24T14:40:11.871 INFO:teuthology.orchestra.run.smithi039.stdout:Downloading Packages: 2023-01-24T14:40:12.681 INFO:teuthology.orchestra.run.smithi039.stdout:ceph-volume-18.0.0-1954.g1a671c19.el8.noarch.rp 324 kB/s | 263 kB 00:00 2023-01-24T14:40:12.681 INFO:teuthology.orchestra.run.smithi039.stdout:-------------------------------------------------------------------------------- 2023-01-24T14:40:12.682 INFO:teuthology.orchestra.run.smithi039.stdout:Total 324 kB/s | 263 kB 00:00 2023-01-24T14:40:12.682 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction check 2023-01-24T14:40:12.693 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction check succeeded. 2023-01-24T14:40:12.694 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction test 2023-01-24T14:40:12.784 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction test succeeded. 2023-01-24T14:40:12.788 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction 2023-01-24T14:40:13.008 INFO:teuthology.orchestra.run.smithi039.stdout: Preparing : 1/1 2023-01-24T14:40:13.303 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-volume-2:18.0.0-1954.g1a671c19.el8.noarch 1/1 2023-01-24T14:40:13.781 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-volume-2:18.0.0-1954.g1a671c19.el8.noarch 1/1 2023-01-24T14:40:15.044 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-volume-2:18.0.0-1954.g1a671c19.el8.noarch 1/1 2023-01-24T14:40:15.044 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-01-24T14:40:15.045 INFO:teuthology.orchestra.run.smithi039.stdout:Installed: 2023-01-24T14:40:15.045 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-volume-2:18.0.0-1954.g1a671c19.el8.noarch 2023-01-24T14:40:15.045 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-01-24T14:40:15.046 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2023-01-24T14:40:15.292 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum -y install librados-devel 2023-01-24T14:40:16.076 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:11:46 ago on Tue 24 Jan 2023 02:28:30 PM UTC. 2023-01-24T14:40:16.709 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: ceph-mgr-2:18.0.0-1954.g1a671c19.el8.x86_64 28/32 2023-01-24T14:40:16.934 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : ceph-osd-2:18.0.0-1954.g1a671c19.el8.x86_64 29/32 2023-01-24T14:40:17.301 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2023-01-24T14:40:17.302 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2023-01-24T14:40:17.302 INFO:teuthology.orchestra.run.smithi039.stdout: Package Arch Version Repo Size 2023-01-24T14:40:17.302 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2023-01-24T14:40:17.303 INFO:teuthology.orchestra.run.smithi039.stdout:Installing: 2023-01-24T14:40:17.303 INFO:teuthology.orchestra.run.smithi039.stdout: librados-devel x86_64 2:18.0.0-1954.g1a671c19.el8 ceph 139 k 2023-01-24T14:40:17.303 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-01-24T14:40:17.303 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction Summary 2023-01-24T14:40:17.303 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2023-01-24T14:40:17.304 INFO:teuthology.orchestra.run.smithi039.stdout:Install 1 Package 2023-01-24T14:40:17.304 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-01-24T14:40:17.304 INFO:teuthology.orchestra.run.smithi039.stdout:Total download size: 139 k 2023-01-24T14:40:17.304 INFO:teuthology.orchestra.run.smithi039.stdout:Installed size: 512 k 2023-01-24T14:40:17.305 INFO:teuthology.orchestra.run.smithi039.stdout:Downloading Packages: 2023-01-24T14:40:17.739 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: ceph-osd-2:18.0.0-1954.g1a671c19.el8.x86_64 29/32 2023-01-24T14:40:17.806 INFO:teuthology.orchestra.run.smithi039.stdout:librados-devel-18.0.0-1954.g1a671c19.el8.x86_64 276 kB/s | 139 kB 00:00 2023-01-24T14:40:17.807 INFO:teuthology.orchestra.run.smithi039.stdout:-------------------------------------------------------------------------------- 2023-01-24T14:40:17.807 INFO:teuthology.orchestra.run.smithi039.stdout:Total 275 kB/s | 139 kB 00:00 2023-01-24T14:40:17.807 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction check 2023-01-24T14:40:17.813 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction check succeeded. 2023-01-24T14:40:17.813 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction test 2023-01-24T14:40:17.869 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction test succeeded. 2023-01-24T14:40:17.873 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction 2023-01-24T14:40:17.990 INFO:teuthology.orchestra.run.smithi039.stdout: Preparing : 1/1 2023-01-24T14:40:18.084 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : ceph-mon-2:18.0.0-1954.g1a671c19.el8.x86_64 30/32 2023-01-24T14:40:18.258 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : librados-devel-2:18.0.0-1954.g1a671c19.el8.x86_64 1/1 2023-01-24T14:40:18.314 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: ceph-mon-2:18.0.0-1954.g1a671c19.el8.x86_64 30/32 2023-01-24T14:40:18.484 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : ceph-mds-2:18.0.0-1954.g1a671c19.el8.x86_64 31/32 2023-01-24T14:40:18.502 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: ceph-mds-2:18.0.0-1954.g1a671c19.el8.x86_64 31/32 2023-01-24T14:40:18.576 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: librados-devel-2:18.0.0-1954.g1a671c19.el8.x86_64 1/1 2023-01-24T14:40:18.658 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : ceph-2:18.0.0-1954.g1a671c19.el8.x86_64 32/32 2023-01-24T14:40:19.072 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : librados-devel-2:18.0.0-1954.g1a671c19.el8.x86_64 1/1 2023-01-24T14:40:19.072 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-01-24T14:40:19.073 INFO:teuthology.orchestra.run.smithi039.stdout:Installed: 2023-01-24T14:40:19.073 INFO:teuthology.orchestra.run.smithi039.stdout: librados-devel-2:18.0.0-1954.g1a671c19.el8.x86_64 2023-01-24T14:40:19.073 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-01-24T14:40:19.073 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2023-01-24T14:40:19.134 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: ceph-2:18.0.0-1954.g1a671c19.el8.x86_64 32/32 2023-01-24T14:40:19.135 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : ceph-2:18.0.0-1954.g1a671c19.el8.x86_64 1/32 2023-01-24T14:40:19.135 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : ceph-mds-2:18.0.0-1954.g1a671c19.el8.x86_64 2/32 2023-01-24T14:40:19.135 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : ceph-mgr-2:18.0.0-1954.g1a671c19.el8.x86_64 3/32 2023-01-24T14:40:19.135 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : ceph-mon-2:18.0.0-1954.g1a671c19.el8.x86_64 4/32 2023-01-24T14:40:19.136 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : ceph-osd-2:18.0.0-1954.g1a671c19.el8.x86_64 5/32 2023-01-24T14:40:19.136 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-1954.g1a671c19.el8. 6/32 2023-01-24T14:40:19.136 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 7/32 2023-01-24T14:40:19.136 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2023-01-24T14:40:19.136 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2023-01-24T14:40:19.137 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 10/32 2023-01-24T14:40:19.137 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2023-01-24T14:40:19.137 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 12/32 2023-01-24T14:40:19.137 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-cryptography-3.2.1-5.el8.x86_64 13/32 2023-01-24T14:40:19.137 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2023-01-24T14:40:19.138 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2023-01-24T14:40:19.138 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2023-01-24T14:40:19.138 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2023-01-24T14:40:19.138 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2023-01-24T14:40:19.138 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2023-01-24T14:40:19.139 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2023-01-24T14:40:19.139 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2023-01-24T14:40:19.139 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2023-01-24T14:40:19.139 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2023-01-24T14:40:19.139 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2023-01-24T14:40:19.140 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2023-01-24T14:40:19.140 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2023-01-24T14:40:19.140 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2023-01-24T14:40:19.140 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2023-01-24T14:40:19.140 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-waitress-1.2.1-2.el8.1.noarch 29/32 2023-01-24T14:40:19.141 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2023-01-24T14:40:19.141 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2023-01-24T14:40:19.322 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum -y install libcephfs2 2023-01-24T14:40:20.058 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2023-01-24T14:40:20.058 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-01-24T14:40:20.058 INFO:teuthology.orchestra.run.smithi002.stdout:Installed: 2023-01-24T14:40:20.058 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-2:18.0.0-1954.g1a671c19.el8.x86_64 2023-01-24T14:40:20.059 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-mds-2:18.0.0-1954.g1a671c19.el8.x86_64 2023-01-24T14:40:20.059 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-mgr-2:18.0.0-1954.g1a671c19.el8.x86_64 2023-01-24T14:40:20.059 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-mgr-modules-core-2:18.0.0-1954.g1a671c19.el8.noarch 2023-01-24T14:40:20.059 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-mon-2:18.0.0-1954.g1a671c19.el8.x86_64 2023-01-24T14:40:20.059 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-osd-2:18.0.0-1954.g1a671c19.el8.x86_64 2023-01-24T14:40:20.060 INFO:teuthology.orchestra.run.smithi002.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-01-24T14:40:20.060 INFO:teuthology.orchestra.run.smithi002.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-01-24T14:40:20.060 INFO:teuthology.orchestra.run.smithi002.stdout: python3-cffi-1.11.5-5.el8.x86_64 2023-01-24T14:40:20.060 INFO:teuthology.orchestra.run.smithi002.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-01-24T14:40:20.060 INFO:teuthology.orchestra.run.smithi002.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-01-24T14:40:20.061 INFO:teuthology.orchestra.run.smithi002.stdout: python3-cryptography-3.2.1-5.el8.x86_64 2023-01-24T14:40:20.061 INFO:teuthology.orchestra.run.smithi002.stdout: python3-jaraco-6.2-6.el8.noarch 2023-01-24T14:40:20.061 INFO:teuthology.orchestra.run.smithi002.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-01-24T14:40:20.061 INFO:teuthology.orchestra.run.smithi002.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-01-24T14:40:20.061 INFO:teuthology.orchestra.run.smithi002.stdout: python3-mako-1.0.6-13.el8.noarch 2023-01-24T14:40:20.062 INFO:teuthology.orchestra.run.smithi002.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-01-24T14:40:20.062 INFO:teuthology.orchestra.run.smithi002.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-01-24T14:40:20.062 INFO:teuthology.orchestra.run.smithi002.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-01-24T14:40:20.062 INFO:teuthology.orchestra.run.smithi002.stdout: python3-portend-2.6-1.el8.noarch 2023-01-24T14:40:20.062 INFO:teuthology.orchestra.run.smithi002.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-01-24T14:40:20.063 INFO:teuthology.orchestra.run.smithi002.stdout: python3-pycparser-2.14-14.el8.noarch 2023-01-24T14:40:20.063 INFO:teuthology.orchestra.run.smithi002.stdout: python3-pytz-2017.2-9.el8.noarch 2023-01-24T14:40:20.063 INFO:teuthology.orchestra.run.smithi002.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-01-24T14:40:20.063 INFO:teuthology.orchestra.run.smithi002.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-01-24T14:40:20.063 INFO:teuthology.orchestra.run.smithi002.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-01-24T14:40:20.064 INFO:teuthology.orchestra.run.smithi002.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-01-24T14:40:20.064 INFO:teuthology.orchestra.run.smithi002.stdout: python3-waitress-1.2.1-2.el8.1.noarch 2023-01-24T14:40:20.064 INFO:teuthology.orchestra.run.smithi002.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-01-24T14:40:20.064 INFO:teuthology.orchestra.run.smithi002.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-01-24T14:40:20.064 INFO:teuthology.orchestra.run.smithi002.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-01-24T14:40:20.065 INFO:teuthology.orchestra.run.smithi002.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-01-24T14:40:20.065 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-01-24T14:40:20.065 INFO:teuthology.orchestra.run.smithi002.stdout:Complete! 2023-01-24T14:40:20.087 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:11:50 ago on Tue 24 Jan 2023 02:28:30 PM UTC. 2023-01-24T14:40:20.254 DEBUG:teuthology.orchestra.run.smithi002:> sudo yum -y install ceph-base 2023-01-24T14:40:21.011 INFO:teuthology.orchestra.run.smithi002.stdout:Last metadata expiration check: 0:11:51 ago on Tue 24 Jan 2023 02:28:30 PM UTC. 2023-01-24T14:40:21.193 INFO:teuthology.orchestra.run.smithi039.stdout:Package libcephfs2-2:18.0.0-1954.g1a671c19.el8.x86_64 is already installed. 2023-01-24T14:40:21.306 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2023-01-24T14:40:21.308 INFO:teuthology.orchestra.run.smithi039.stdout:Nothing to do. 2023-01-24T14:40:21.308 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2023-01-24T14:40:21.399 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum -y install libcephfs-devel 2023-01-24T14:40:22.115 INFO:teuthology.orchestra.run.smithi002.stdout:Package ceph-base-2:18.0.0-1954.g1a671c19.el8.x86_64 is already installed. 2023-01-24T14:40:22.168 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:11:52 ago on Tue 24 Jan 2023 02:28:30 PM UTC. 2023-01-24T14:40:22.226 INFO:teuthology.orchestra.run.smithi002.stdout:Dependencies resolved. 2023-01-24T14:40:22.227 INFO:teuthology.orchestra.run.smithi002.stdout:Nothing to do. 2023-01-24T14:40:22.227 INFO:teuthology.orchestra.run.smithi002.stdout:Complete! 2023-01-24T14:40:22.314 DEBUG:teuthology.orchestra.run.smithi002:> sudo yum -y install cephadm 2023-01-24T14:40:23.069 INFO:teuthology.orchestra.run.smithi002.stdout:Last metadata expiration check: 0:11:53 ago on Tue 24 Jan 2023 02:28:30 PM UTC. 2023-01-24T14:40:23.394 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2023-01-24T14:40:23.395 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2023-01-24T14:40:23.396 INFO:teuthology.orchestra.run.smithi039.stdout: Package Arch Version Repo Size 2023-01-24T14:40:23.396 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2023-01-24T14:40:23.396 INFO:teuthology.orchestra.run.smithi039.stdout:Installing: 2023-01-24T14:40:23.396 INFO:teuthology.orchestra.run.smithi039.stdout: libcephfs-devel x86_64 2:18.0.0-1954.g1a671c19.el8 ceph 30 k 2023-01-24T14:40:23.397 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-01-24T14:40:23.397 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction Summary 2023-01-24T14:40:23.397 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2023-01-24T14:40:23.397 INFO:teuthology.orchestra.run.smithi039.stdout:Install 1 Package 2023-01-24T14:40:23.397 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-01-24T14:40:23.398 INFO:teuthology.orchestra.run.smithi039.stdout:Total download size: 30 k 2023-01-24T14:40:23.398 INFO:teuthology.orchestra.run.smithi039.stdout:Installed size: 137 k 2023-01-24T14:40:23.398 INFO:teuthology.orchestra.run.smithi039.stdout:Downloading Packages: 2023-01-24T14:40:23.554 INFO:teuthology.orchestra.run.smithi039.stdout:libcephfs-devel-18.0.0-1954.g1a671c19.el8.x86_6 194 kB/s | 30 kB 00:00 2023-01-24T14:40:23.554 INFO:teuthology.orchestra.run.smithi039.stdout:-------------------------------------------------------------------------------- 2023-01-24T14:40:23.555 INFO:teuthology.orchestra.run.smithi039.stdout:Total 192 kB/s | 30 kB 00:00 2023-01-24T14:40:23.555 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction check 2023-01-24T14:40:23.559 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction check succeeded. 2023-01-24T14:40:23.559 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction test 2023-01-24T14:40:23.584 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction test succeeded. 2023-01-24T14:40:23.587 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction 2023-01-24T14:40:23.649 INFO:teuthology.orchestra.run.smithi039.stdout: Preparing : 1/1 2023-01-24T14:40:23.907 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : libcephfs-devel-2:18.0.0-1954.g1a671c19.el8.x86_64 1/1 2023-01-24T14:40:24.235 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: libcephfs-devel-2:18.0.0-1954.g1a671c19.el8.x86_64 1/1 2023-01-24T14:40:24.293 INFO:teuthology.orchestra.run.smithi002.stdout:Dependencies resolved. 2023-01-24T14:40:24.294 INFO:teuthology.orchestra.run.smithi002.stdout:================================================================================ 2023-01-24T14:40:24.295 INFO:teuthology.orchestra.run.smithi002.stdout: Package Arch Version Repository Size 2023-01-24T14:40:24.295 INFO:teuthology.orchestra.run.smithi002.stdout:================================================================================ 2023-01-24T14:40:24.295 INFO:teuthology.orchestra.run.smithi002.stdout:Installing: 2023-01-24T14:40:24.295 INFO:teuthology.orchestra.run.smithi002.stdout: cephadm noarch 2:18.0.0-1954.g1a671c19.el8 ceph-noarch 82 k 2023-01-24T14:40:24.295 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-01-24T14:40:24.296 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction Summary 2023-01-24T14:40:24.296 INFO:teuthology.orchestra.run.smithi002.stdout:================================================================================ 2023-01-24T14:40:24.296 INFO:teuthology.orchestra.run.smithi002.stdout:Install 1 Package 2023-01-24T14:40:24.296 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-01-24T14:40:24.296 INFO:teuthology.orchestra.run.smithi002.stdout:Total download size: 82 k 2023-01-24T14:40:24.297 INFO:teuthology.orchestra.run.smithi002.stdout:Installed size: 374 k 2023-01-24T14:40:24.297 INFO:teuthology.orchestra.run.smithi002.stdout:Downloading Packages: 2023-01-24T14:40:24.709 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libcephfs-devel-2:18.0.0-1954.g1a671c19.el8.x86_64 1/1 2023-01-24T14:40:24.709 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-01-24T14:40:24.709 INFO:teuthology.orchestra.run.smithi039.stdout:Installed: 2023-01-24T14:40:24.709 INFO:teuthology.orchestra.run.smithi039.stdout: libcephfs-devel-2:18.0.0-1954.g1a671c19.el8.x86_64 2023-01-24T14:40:24.710 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-01-24T14:40:24.710 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2023-01-24T14:40:24.711 INFO:teuthology.orchestra.run.smithi002.stdout:cephadm-18.0.0-1954.g1a671c19.el8.noarch.rpm 199 kB/s | 82 kB 00:00 2023-01-24T14:40:24.711 INFO:teuthology.orchestra.run.smithi002.stdout:-------------------------------------------------------------------------------- 2023-01-24T14:40:24.711 INFO:teuthology.orchestra.run.smithi002.stdout:Total 198 kB/s | 82 kB 00:00 2023-01-24T14:40:24.712 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction check 2023-01-24T14:40:24.721 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction check succeeded. 2023-01-24T14:40:24.721 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction test 2023-01-24T14:40:24.723 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction test succeeded. 2023-01-24T14:40:24.727 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction 2023-01-24T14:40:24.815 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum -y install librados2 2023-01-24T14:40:25.301 INFO:teuthology.orchestra.run.smithi002.stdout: Preparing : 1/1 2023-01-24T14:40:25.577 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:11:55 ago on Tue 24 Jan 2023 02:28:30 PM UTC. 2023-01-24T14:40:26.684 INFO:teuthology.orchestra.run.smithi039.stdout:Package librados2-2:18.0.0-1954.g1a671c19.el8.x86_64 is already installed. 2023-01-24T14:40:26.797 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2023-01-24T14:40:26.798 INFO:teuthology.orchestra.run.smithi039.stdout:Nothing to do. 2023-01-24T14:40:26.798 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2023-01-24T14:40:26.875 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum -y install librbd1 2023-01-24T14:40:27.646 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:11:57 ago on Tue 24 Jan 2023 02:28:30 PM UTC. 2023-01-24T14:40:28.759 INFO:teuthology.orchestra.run.smithi039.stdout:Package librbd1-2:18.0.0-1954.g1a671c19.el8.x86_64 is already installed. 2023-01-24T14:40:28.873 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2023-01-24T14:40:28.874 INFO:teuthology.orchestra.run.smithi039.stdout:Nothing to do. 2023-01-24T14:40:28.874 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2023-01-24T14:40:28.974 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum -y install python3-rados 2023-01-24T14:40:29.741 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:11:59 ago on Tue 24 Jan 2023 02:28:30 PM UTC. 2023-01-24T14:40:30.854 INFO:teuthology.orchestra.run.smithi039.stdout:Package python3-rados-2:18.0.0-1954.g1a671c19.el8.x86_64 is already installed. 2023-01-24T14:40:30.968 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2023-01-24T14:40:30.970 INFO:teuthology.orchestra.run.smithi039.stdout:Nothing to do. 2023-01-24T14:40:30.970 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2023-01-24T14:40:31.058 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum -y install python3-rgw 2023-01-24T14:40:31.810 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:12:01 ago on Tue 24 Jan 2023 02:28:30 PM UTC. 2023-01-24T14:40:32.925 INFO:teuthology.orchestra.run.smithi039.stdout:Package python3-rgw-2:18.0.0-1954.g1a671c19.el8.x86_64 is already installed. 2023-01-24T14:40:33.038 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2023-01-24T14:40:33.039 INFO:teuthology.orchestra.run.smithi039.stdout:Nothing to do. 2023-01-24T14:40:33.039 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2023-01-24T14:40:33.126 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum -y install python3-cephfs 2023-01-24T14:40:33.895 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:12:03 ago on Tue 24 Jan 2023 02:28:30 PM UTC. 2023-01-24T14:40:35.005 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: cephadm-2:18.0.0-1954.g1a671c19.el8.noarch 1/1 2023-01-24T14:40:35.015 INFO:teuthology.orchestra.run.smithi039.stdout:Package python3-cephfs-2:18.0.0-1954.g1a671c19.el8.x86_64 is already installed. 2023-01-24T14:40:35.128 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2023-01-24T14:40:35.130 INFO:teuthology.orchestra.run.smithi039.stdout:Nothing to do. 2023-01-24T14:40:35.130 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2023-01-24T14:40:35.220 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum -y install python3-rbd 2023-01-24T14:40:35.978 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:12:05 ago on Tue 24 Jan 2023 02:28:30 PM UTC. 2023-01-24T14:40:37.048 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : cephadm-2:18.0.0-1954.g1a671c19.el8.noarch 1/1 2023-01-24T14:40:37.089 INFO:teuthology.orchestra.run.smithi039.stdout:Package python3-rbd-2:18.0.0-1954.g1a671c19.el8.x86_64 is already installed. 2023-01-24T14:40:37.202 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2023-01-24T14:40:37.203 INFO:teuthology.orchestra.run.smithi039.stdout:Nothing to do. 2023-01-24T14:40:37.204 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2023-01-24T14:40:37.287 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum -y install rbd-fuse 2023-01-24T14:40:37.306 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: cephadm-2:18.0.0-1954.g1a671c19.el8.noarch 1/1 2023-01-24T14:40:38.038 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:12:08 ago on Tue 24 Jan 2023 02:28:30 PM UTC. 2023-01-24T14:40:38.139 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : cephadm-2:18.0.0-1954.g1a671c19.el8.noarch 1/1 2023-01-24T14:40:38.139 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-01-24T14:40:38.139 INFO:teuthology.orchestra.run.smithi002.stdout:Installed: 2023-01-24T14:40:38.139 INFO:teuthology.orchestra.run.smithi002.stdout: cephadm-2:18.0.0-1954.g1a671c19.el8.noarch 2023-01-24T14:40:38.140 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-01-24T14:40:38.140 INFO:teuthology.orchestra.run.smithi002.stdout:Complete! 2023-01-24T14:40:38.337 DEBUG:teuthology.orchestra.run.smithi002:> sudo yum -y install ceph-immutable-object-cache 2023-01-24T14:40:39.101 INFO:teuthology.orchestra.run.smithi002.stdout:Last metadata expiration check: 0:12:09 ago on Tue 24 Jan 2023 02:28:30 PM UTC. 2023-01-24T14:40:39.263 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2023-01-24T14:40:39.264 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2023-01-24T14:40:39.264 INFO:teuthology.orchestra.run.smithi039.stdout: Package Architecture Version Repository Size 2023-01-24T14:40:39.264 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2023-01-24T14:40:39.265 INFO:teuthology.orchestra.run.smithi039.stdout:Installing: 2023-01-24T14:40:39.265 INFO:teuthology.orchestra.run.smithi039.stdout: rbd-fuse x86_64 2:18.0.0-1954.g1a671c19.el8 ceph 91 k 2023-01-24T14:40:39.265 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-01-24T14:40:39.265 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction Summary 2023-01-24T14:40:39.266 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2023-01-24T14:40:39.266 INFO:teuthology.orchestra.run.smithi039.stdout:Install 1 Package 2023-01-24T14:40:39.266 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-01-24T14:40:39.266 INFO:teuthology.orchestra.run.smithi039.stdout:Total download size: 91 k 2023-01-24T14:40:39.266 INFO:teuthology.orchestra.run.smithi039.stdout:Installed size: 303 k 2023-01-24T14:40:39.267 INFO:teuthology.orchestra.run.smithi039.stdout:Downloading Packages: 2023-01-24T14:40:39.636 INFO:teuthology.orchestra.run.smithi039.stdout:rbd-fuse-18.0.0-1954.g1a671c19.el8.x86_64.rpm 245 kB/s | 91 kB 00:00 2023-01-24T14:40:39.637 INFO:teuthology.orchestra.run.smithi039.stdout:-------------------------------------------------------------------------------- 2023-01-24T14:40:39.637 INFO:teuthology.orchestra.run.smithi039.stdout:Total 245 kB/s | 91 kB 00:00 2023-01-24T14:40:39.638 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction check 2023-01-24T14:40:39.648 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction check succeeded. 2023-01-24T14:40:39.648 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction test 2023-01-24T14:40:39.706 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction test succeeded. 2023-01-24T14:40:39.710 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction 2023-01-24T14:40:39.828 INFO:teuthology.orchestra.run.smithi039.stdout: Preparing : 1/1 2023-01-24T14:40:40.097 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : rbd-fuse-2:18.0.0-1954.g1a671c19.el8.x86_64 1/1 2023-01-24T14:40:40.326 INFO:teuthology.orchestra.run.smithi002.stdout:Dependencies resolved. 2023-01-24T14:40:40.327 INFO:teuthology.orchestra.run.smithi002.stdout:================================================================================ 2023-01-24T14:40:40.327 INFO:teuthology.orchestra.run.smithi002.stdout: Package Arch Version Repo Size 2023-01-24T14:40:40.328 INFO:teuthology.orchestra.run.smithi002.stdout:================================================================================ 2023-01-24T14:40:40.328 INFO:teuthology.orchestra.run.smithi002.stdout:Installing: 2023-01-24T14:40:40.328 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-1954.g1a671c19.el8 ceph 157 k 2023-01-24T14:40:40.328 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-01-24T14:40:40.329 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction Summary 2023-01-24T14:40:40.329 INFO:teuthology.orchestra.run.smithi002.stdout:================================================================================ 2023-01-24T14:40:40.329 INFO:teuthology.orchestra.run.smithi002.stdout:Install 1 Package 2023-01-24T14:40:40.329 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-01-24T14:40:40.329 INFO:teuthology.orchestra.run.smithi002.stdout:Total download size: 157 k 2023-01-24T14:40:40.330 INFO:teuthology.orchestra.run.smithi002.stdout:Installed size: 540 k 2023-01-24T14:40:40.330 INFO:teuthology.orchestra.run.smithi002.stdout:Downloading Packages: 2023-01-24T14:40:40.414 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: rbd-fuse-2:18.0.0-1954.g1a671c19.el8.x86_64 1/1 2023-01-24T14:40:40.775 INFO:teuthology.orchestra.run.smithi002.stdout:ceph-immutable-object-cache-18.0.0-1954.g1a671c 351 kB/s | 157 kB 00:00 2023-01-24T14:40:40.776 INFO:teuthology.orchestra.run.smithi002.stdout:-------------------------------------------------------------------------------- 2023-01-24T14:40:40.776 INFO:teuthology.orchestra.run.smithi002.stdout:Total 350 kB/s | 157 kB 00:00 2023-01-24T14:40:40.776 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction check 2023-01-24T14:40:40.788 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction check succeeded. 2023-01-24T14:40:40.789 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction test 2023-01-24T14:40:40.842 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction test succeeded. 2023-01-24T14:40:40.846 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction 2023-01-24T14:40:40.958 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : rbd-fuse-2:18.0.0-1954.g1a671c19.el8.x86_64 1/1 2023-01-24T14:40:40.958 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-01-24T14:40:40.959 INFO:teuthology.orchestra.run.smithi039.stdout:Installed: 2023-01-24T14:40:40.959 INFO:teuthology.orchestra.run.smithi039.stdout: rbd-fuse-2:18.0.0-1954.g1a671c19.el8.x86_64 2023-01-24T14:40:40.959 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-01-24T14:40:40.959 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2023-01-24T14:40:40.978 INFO:teuthology.orchestra.run.smithi002.stdout: Preparing : 1/1 2023-01-24T14:40:41.147 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-1954.g1a671c19. 1/1 2023-01-24T14:40:41.185 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum -y install rbd-mirror 2023-01-24T14:40:41.587 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-1954.g1a671c19. 1/1 2023-01-24T14:40:41.949 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:12:11 ago on Tue 24 Jan 2023 02:28:30 PM UTC. 2023-01-24T14:40:42.083 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-1954.g1a671c19. 1/1 2023-01-24T14:40:42.083 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-01-24T14:40:42.083 INFO:teuthology.orchestra.run.smithi002.stdout:Installed: 2023-01-24T14:40:42.084 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-immutable-object-cache-2:18.0.0-1954.g1a671c19.el8.x86_64 2023-01-24T14:40:42.084 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-01-24T14:40:42.084 INFO:teuthology.orchestra.run.smithi002.stdout:Complete! 2023-01-24T14:40:42.278 DEBUG:teuthology.orchestra.run.smithi002:> sudo yum -y install ceph-mgr 2023-01-24T14:40:43.044 INFO:teuthology.orchestra.run.smithi002.stdout:Last metadata expiration check: 0:12:13 ago on Tue 24 Jan 2023 02:28:30 PM UTC. 2023-01-24T14:40:43.168 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2023-01-24T14:40:43.169 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2023-01-24T14:40:43.169 INFO:teuthology.orchestra.run.smithi039.stdout: Package Arch Version Repo Size 2023-01-24T14:40:43.169 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2023-01-24T14:40:43.169 INFO:teuthology.orchestra.run.smithi039.stdout:Installing: 2023-01-24T14:40:43.170 INFO:teuthology.orchestra.run.smithi039.stdout: rbd-mirror x86_64 2:18.0.0-1954.g1a671c19.el8 ceph 3.3 M 2023-01-24T14:40:43.170 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-01-24T14:40:43.170 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction Summary 2023-01-24T14:40:43.170 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2023-01-24T14:40:43.170 INFO:teuthology.orchestra.run.smithi039.stdout:Install 1 Package 2023-01-24T14:40:43.170 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-01-24T14:40:43.171 INFO:teuthology.orchestra.run.smithi039.stdout:Total download size: 3.3 M 2023-01-24T14:40:43.171 INFO:teuthology.orchestra.run.smithi039.stdout:Installed size: 18 M 2023-01-24T14:40:43.171 INFO:teuthology.orchestra.run.smithi039.stdout:Downloading Packages: 2023-01-24T14:40:44.160 INFO:teuthology.orchestra.run.smithi002.stdout:Package ceph-mgr-2:18.0.0-1954.g1a671c19.el8.x86_64 is already installed. 2023-01-24T14:40:44.271 INFO:teuthology.orchestra.run.smithi002.stdout:Dependencies resolved. 2023-01-24T14:40:44.273 INFO:teuthology.orchestra.run.smithi002.stdout:Nothing to do. 2023-01-24T14:40:44.273 INFO:teuthology.orchestra.run.smithi002.stdout:Complete! 2023-01-24T14:40:44.363 DEBUG:teuthology.orchestra.run.smithi002:> sudo yum -y install ceph-mgr-dashboard 2023-01-24T14:40:45.118 INFO:teuthology.orchestra.run.smithi002.stdout:Last metadata expiration check: 0:12:15 ago on Tue 24 Jan 2023 02:28:30 PM UTC. 2023-01-24T14:40:45.948 INFO:teuthology.orchestra.run.smithi039.stdout:rbd-mirror-18.0.0-1954.g1a671c19.el8.x86_64.rpm 1.2 MB/s | 3.3 MB 00:02 2023-01-24T14:40:45.949 INFO:teuthology.orchestra.run.smithi039.stdout:-------------------------------------------------------------------------------- 2023-01-24T14:40:45.949 INFO:teuthology.orchestra.run.smithi039.stdout:Total 1.2 MB/s | 3.3 MB 00:02 2023-01-24T14:40:45.950 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction check 2023-01-24T14:40:45.964 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction check succeeded. 2023-01-24T14:40:45.964 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction test 2023-01-24T14:40:46.036 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction test succeeded. 2023-01-24T14:40:46.040 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction 2023-01-24T14:40:46.353 INFO:teuthology.orchestra.run.smithi002.stdout:Dependencies resolved. 2023-01-24T14:40:46.354 INFO:teuthology.orchestra.run.smithi002.stdout:================================================================================ 2023-01-24T14:40:46.355 INFO:teuthology.orchestra.run.smithi002.stdout: Package Arch Version Repository Size 2023-01-24T14:40:46.355 INFO:teuthology.orchestra.run.smithi002.stdout:================================================================================ 2023-01-24T14:40:46.355 INFO:teuthology.orchestra.run.smithi002.stdout:Installing: 2023-01-24T14:40:46.355 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-mgr-dashboard noarch 2:18.0.0-1954.g1a671c19.el8 ceph-noarch 1.7 M 2023-01-24T14:40:46.355 INFO:teuthology.orchestra.run.smithi002.stdout:Installing dependencies: 2023-01-24T14:40:46.356 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-grafana-dashboards noarch 2:18.0.0-1954.g1a671c19.el8 ceph-noarch 23 k 2023-01-24T14:40:46.356 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-prometheus-alerts noarch 2:18.0.0-1954.g1a671c19.el8 ceph-noarch 15 k 2023-01-24T14:40:46.356 INFO:teuthology.orchestra.run.smithi002.stdout: python3-jwt noarch 1.6.1-2.el8 CentOS-BaseOS 43 k 2023-01-24T14:40:46.356 INFO:teuthology.orchestra.run.smithi002.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-01-24T14:40:46.356 INFO:teuthology.orchestra.run.smithi002.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-01-24T14:40:46.357 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-01-24T14:40:46.357 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction Summary 2023-01-24T14:40:46.357 INFO:teuthology.orchestra.run.smithi002.stdout:================================================================================ 2023-01-24T14:40:46.357 INFO:teuthology.orchestra.run.smithi002.stdout:Install 6 Packages 2023-01-24T14:40:46.357 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-01-24T14:40:46.358 INFO:teuthology.orchestra.run.smithi002.stdout:Total download size: 2.0 M 2023-01-24T14:40:46.358 INFO:teuthology.orchestra.run.smithi002.stdout:Installed size: 8.2 M 2023-01-24T14:40:46.359 INFO:teuthology.orchestra.run.smithi002.stdout:Downloading Packages: 2023-01-24T14:40:46.511 INFO:teuthology.orchestra.run.smithi002.stdout:(1/6): ceph-prometheus-alerts-18.0.0-1954.g1a67 95 kB/s | 15 kB 00:00 2023-01-24T14:40:46.514 INFO:teuthology.orchestra.run.smithi039.stdout: Preparing : 1/1 2023-01-24T14:40:46.528 INFO:teuthology.orchestra.run.smithi002.stdout:(2/6): ceph-grafana-dashboards-18.0.0-1954.g1a6 135 kB/s | 23 kB 00:00 2023-01-24T14:40:46.603 INFO:teuthology.orchestra.run.smithi002.stdout:(3/6): python3-jwt-1.6.1-2.el8.noarch.rpm 464 kB/s | 43 kB 00:00 2023-01-24T14:40:46.645 INFO:teuthology.orchestra.run.smithi002.stdout:(4/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 287 kB/s | 34 kB 00:00 2023-01-24T14:40:46.698 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : rbd-mirror-2:18.0.0-1954.g1a671c19.el8.x86_64 1/1 2023-01-24T14:40:46.762 INFO:teuthology.orchestra.run.smithi002.stdout:(5/6): ceph-mgr-dashboard-18.0.0-1954.g1a671c19 4.3 MB/s | 1.7 MB 00:00 2023-01-24T14:40:46.821 INFO:teuthology.orchestra.run.smithi002.stdout:(6/6): python3-routes-2.4.1-12.el8.noarch.rpm 906 kB/s | 196 kB 00:00 2023-01-24T14:40:46.822 INFO:teuthology.orchestra.run.smithi002.stdout:-------------------------------------------------------------------------------- 2023-01-24T14:40:46.823 INFO:teuthology.orchestra.run.smithi002.stdout:Total 4.3 MB/s | 2.0 MB 00:00 2023-01-24T14:40:46.833 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction check 2023-01-24T14:40:46.843 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction check succeeded. 2023-01-24T14:40:46.843 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction test 2023-01-24T14:40:46.931 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction test succeeded. 2023-01-24T14:40:46.935 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction 2023-01-24T14:40:47.151 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: rbd-mirror-2:18.0.0-1954.g1a671c19.el8.x86_64 1/1 2023-01-24T14:40:47.246 INFO:teuthology.orchestra.run.smithi002.stdout: Preparing : 1/1 2023-01-24T14:40:47.495 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2023-01-24T14:40:47.698 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : rbd-mirror-2:18.0.0-1954.g1a671c19.el8.x86_64 1/1 2023-01-24T14:40:47.698 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-01-24T14:40:47.698 INFO:teuthology.orchestra.run.smithi039.stdout:Installed: 2023-01-24T14:40:47.698 INFO:teuthology.orchestra.run.smithi039.stdout: rbd-mirror-2:18.0.0-1954.g1a671c19.el8.x86_64 2023-01-24T14:40:47.699 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-01-24T14:40:47.699 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2023-01-24T14:40:47.723 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2023-01-24T14:40:47.901 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum -y install rbd-nbd 2023-01-24T14:40:47.965 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2023-01-24T14:40:48.094 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-1954.g1a671c19.el8.n 4/6 2023-01-24T14:40:48.485 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-1954.g1a671c19.el8. 5/6 2023-01-24T14:40:48.666 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:12:18 ago on Tue 24 Jan 2023 02:28:30 PM UTC. 2023-01-24T14:40:49.053 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-1954.g1a671c19.el8.noarc 6/6 2023-01-24T14:40:49.374 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-1954.g1a671c19.el8.noarc 6/6 2023-01-24T14:40:49.374 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-1954.g1a671c19.el8. 1/6 2023-01-24T14:40:49.374 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-1954.g1a671c19.el8.noarc 2/6 2023-01-24T14:40:49.374 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-1954.g1a671c19.el8.n 3/6 2023-01-24T14:40:49.375 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2023-01-24T14:40:49.375 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2023-01-24T14:40:49.886 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2023-01-24T14:40:49.887 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2023-01-24T14:40:49.887 INFO:teuthology.orchestra.run.smithi039.stdout: Package Architecture Version Repository Size 2023-01-24T14:40:49.887 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2023-01-24T14:40:49.887 INFO:teuthology.orchestra.run.smithi039.stdout:Installing: 2023-01-24T14:40:49.888 INFO:teuthology.orchestra.run.smithi039.stdout: rbd-nbd x86_64 2:18.0.0-1954.g1a671c19.el8 ceph 177 k 2023-01-24T14:40:49.888 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-01-24T14:40:49.888 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction Summary 2023-01-24T14:40:49.888 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2023-01-24T14:40:49.888 INFO:teuthology.orchestra.run.smithi039.stdout:Install 1 Package 2023-01-24T14:40:49.889 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-01-24T14:40:49.889 INFO:teuthology.orchestra.run.smithi039.stdout:Total download size: 177 k 2023-01-24T14:40:49.890 INFO:teuthology.orchestra.run.smithi039.stdout:Installed size: 626 k 2023-01-24T14:40:49.890 INFO:teuthology.orchestra.run.smithi039.stdout:Downloading Packages: 2023-01-24T14:40:49.899 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2023-01-24T14:40:49.899 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-01-24T14:40:49.899 INFO:teuthology.orchestra.run.smithi002.stdout:Installed: 2023-01-24T14:40:49.899 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-grafana-dashboards-2:18.0.0-1954.g1a671c19.el8.noarch 2023-01-24T14:40:49.900 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-mgr-dashboard-2:18.0.0-1954.g1a671c19.el8.noarch 2023-01-24T14:40:49.900 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-prometheus-alerts-2:18.0.0-1954.g1a671c19.el8.noarch 2023-01-24T14:40:49.900 INFO:teuthology.orchestra.run.smithi002.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-01-24T14:40:49.900 INFO:teuthology.orchestra.run.smithi002.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-01-24T14:40:49.901 INFO:teuthology.orchestra.run.smithi002.stdout: python3-routes-2.4.1-12.el8.noarch 2023-01-24T14:40:49.901 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-01-24T14:40:49.901 INFO:teuthology.orchestra.run.smithi002.stdout:Complete! 2023-01-24T14:40:50.015 DEBUG:teuthology.orchestra.run.smithi002:> sudo yum -y install ceph-mgr-diskprediction-local 2023-01-24T14:40:50.380 INFO:teuthology.orchestra.run.smithi039.stdout:rbd-nbd-18.0.0-1954.g1a671c19.el8.x86_64.rpm 360 kB/s | 177 kB 00:00 2023-01-24T14:40:50.382 INFO:teuthology.orchestra.run.smithi039.stdout:-------------------------------------------------------------------------------- 2023-01-24T14:40:50.382 INFO:teuthology.orchestra.run.smithi039.stdout:Total 359 kB/s | 177 kB 00:00 2023-01-24T14:40:50.382 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction check 2023-01-24T14:40:50.392 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction check succeeded. 2023-01-24T14:40:50.393 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction test 2023-01-24T14:40:50.449 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction test succeeded. 2023-01-24T14:40:50.453 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction 2023-01-24T14:40:50.573 INFO:teuthology.orchestra.run.smithi039.stdout: Preparing : 1/1 2023-01-24T14:40:50.766 INFO:teuthology.orchestra.run.smithi002.stdout:Last metadata expiration check: 0:12:20 ago on Tue 24 Jan 2023 02:28:30 PM UTC. 2023-01-24T14:40:50.770 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : rbd-nbd-2:18.0.0-1954.g1a671c19.el8.x86_64 1/1 2023-01-24T14:40:51.080 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: rbd-nbd-2:18.0.0-1954.g1a671c19.el8.x86_64 1/1 2023-01-24T14:40:51.562 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : rbd-nbd-2:18.0.0-1954.g1a671c19.el8.x86_64 1/1 2023-01-24T14:40:51.562 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-01-24T14:40:51.563 INFO:teuthology.orchestra.run.smithi039.stdout:Installed: 2023-01-24T14:40:51.563 INFO:teuthology.orchestra.run.smithi039.stdout: rbd-nbd-2:18.0.0-1954.g1a671c19.el8.x86_64 2023-01-24T14:40:51.563 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-01-24T14:40:51.563 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2023-01-24T14:40:51.757 DEBUG:teuthology.parallel:result is None 2023-01-24T14:40:52.022 INFO:teuthology.orchestra.run.smithi002.stdout:Dependencies resolved. 2023-01-24T14:40:52.023 INFO:teuthology.orchestra.run.smithi002.stdout:===================================================================================================== 2023-01-24T14:40:52.023 INFO:teuthology.orchestra.run.smithi002.stdout: Package Arch Version Repository Size 2023-01-24T14:40:52.023 INFO:teuthology.orchestra.run.smithi002.stdout:===================================================================================================== 2023-01-24T14:40:52.023 INFO:teuthology.orchestra.run.smithi002.stdout:Installing: 2023-01-24T14:40:52.024 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-1954.g1a671c19.el8 ceph-noarch 7.4 M 2023-01-24T14:40:52.024 INFO:teuthology.orchestra.run.smithi002.stdout:Installing dependencies: 2023-01-24T14:40:52.024 INFO:teuthology.orchestra.run.smithi002.stdout: libgfortran x86_64 8.5.0-18.el8 CentOS-BaseOS 645 k 2023-01-24T14:40:52.024 INFO:teuthology.orchestra.run.smithi002.stdout: libquadmath x86_64 8.5.0-18.el8 CentOS-BaseOS 172 k 2023-01-24T14:40:52.025 INFO:teuthology.orchestra.run.smithi002.stdout: openblas x86_64 0.3.15-4.el8 CentOS-AppStream 4.7 M 2023-01-24T14:40:52.025 INFO:teuthology.orchestra.run.smithi002.stdout: openblas-threads x86_64 0.3.15-4.el8 CentOS-AppStream 4.9 M 2023-01-24T14:40:52.025 INFO:teuthology.orchestra.run.smithi002.stdout: python3-numpy x86_64 1:1.14.3-10.el8 CentOS-AppStream 3.7 M 2023-01-24T14:40:52.025 INFO:teuthology.orchestra.run.smithi002.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 CentOS-AppStream 225 k 2023-01-24T14:40:52.025 INFO:teuthology.orchestra.run.smithi002.stdout: python3-scipy x86_64 1.0.0-21.module_el8.5.0+771+e5d9a225 CentOS-AppStream 14 M 2023-01-24T14:40:52.026 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-01-24T14:40:52.026 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction Summary 2023-01-24T14:40:52.026 INFO:teuthology.orchestra.run.smithi002.stdout:===================================================================================================== 2023-01-24T14:40:52.026 INFO:teuthology.orchestra.run.smithi002.stdout:Install 8 Packages 2023-01-24T14:40:52.026 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-01-24T14:40:52.027 INFO:teuthology.orchestra.run.smithi002.stdout:Total download size: 35 M 2023-01-24T14:40:52.027 INFO:teuthology.orchestra.run.smithi002.stdout:Installed size: 210 M 2023-01-24T14:40:52.027 INFO:teuthology.orchestra.run.smithi002.stdout:Downloading Packages: 2023-01-24T14:40:52.252 INFO:teuthology.orchestra.run.smithi002.stdout:(1/8): openblas-0.3.15-4.el8.x86_64.rpm 21 MB/s | 4.7 MB 00:00 2023-01-24T14:40:52.319 INFO:teuthology.orchestra.run.smithi002.stdout:(2/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 17 MB/s | 4.9 MB 00:00 2023-01-24T14:40:52.352 INFO:teuthology.orchestra.run.smithi002.stdout:(3/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 6.6 MB/s | 225 kB 00:00 2023-01-24T14:40:52.444 INFO:teuthology.orchestra.run.smithi002.stdout:(4/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 19 MB/s | 3.7 MB 00:00 2023-01-24T14:40:52.486 INFO:teuthology.orchestra.run.smithi002.stdout:(5/8): libgfortran-8.5.0-18.el8.x86_64.rpm 15 MB/s | 645 kB 00:00 2023-01-24T14:40:52.520 INFO:teuthology.orchestra.run.smithi002.stdout:(6/8): libquadmath-8.5.0-18.el8.x86_64.rpm 5.0 MB/s | 172 kB 00:00 2023-01-24T14:40:52.754 INFO:teuthology.orchestra.run.smithi002.stdout:(7/8): python3-scipy-1.0.0-21.module_el8.5.0+77 34 MB/s | 14 MB 00:00 2023-01-24T14:40:54.470 INFO:teuthology.orchestra.run.smithi002.stdout:(8/8): ceph-mgr-diskprediction-local-18.0.0-195 3.0 MB/s | 7.4 MB 00:02 2023-01-24T14:40:54.473 INFO:teuthology.orchestra.run.smithi002.stdout:-------------------------------------------------------------------------------- 2023-01-24T14:40:54.474 INFO:teuthology.orchestra.run.smithi002.stdout:Total 14 MB/s | 35 MB 00:02 2023-01-24T14:40:54.715 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction check 2023-01-24T14:40:54.735 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction check succeeded. 2023-01-24T14:40:54.735 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction test 2023-01-24T14:40:55.156 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction test succeeded. 2023-01-24T14:40:55.159 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction 2023-01-24T14:40:55.739 INFO:teuthology.orchestra.run.smithi002.stdout: Preparing : 1/1 2023-01-24T14:40:55.894 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : libquadmath-8.5.0-18.el8.x86_64 1/8 2023-01-24T14:40:56.043 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: libquadmath-8.5.0-18.el8.x86_64 1/8 2023-01-24T14:40:56.219 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : libgfortran-8.5.0-18.el8.x86_64 2/8 2023-01-24T14:40:56.855 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: libgfortran-8.5.0-18.el8.x86_64 2/8 2023-01-24T14:40:57.036 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2023-01-24T14:40:57.691 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2023-01-24T14:40:57.844 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2023-01-24T14:40:58.580 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2023-01-24T14:40:58.993 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2023-01-24T14:40:59.402 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-01-24T14:41:01.280 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-01-24T14:41:02.590 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 7/8 2023-01-24T14:41:02.802 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-1954.g1a671c1 8/8 2023-01-24T14:41:03.148 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-1954.g1a671c1 8/8 2023-01-24T14:41:03.149 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-1954.g1a671c1 1/8 2023-01-24T14:41:03.149 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2023-01-24T14:41:03.149 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2023-01-24T14:41:03.149 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2023-01-24T14:41:03.150 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2023-01-24T14:41:03.150 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 6/8 2023-01-24T14:41:03.150 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : libgfortran-8.5.0-18.el8.x86_64 7/8 2023-01-24T14:41:03.684 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : libquadmath-8.5.0-18.el8.x86_64 8/8 2023-01-24T14:41:03.684 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-01-24T14:41:03.685 INFO:teuthology.orchestra.run.smithi002.stdout:Installed: 2023-01-24T14:41:03.685 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-mgr-diskprediction-local-2:18.0.0-1954.g1a671c19.el8.noarch 2023-01-24T14:41:03.685 INFO:teuthology.orchestra.run.smithi002.stdout: libgfortran-8.5.0-18.el8.x86_64 2023-01-24T14:41:03.685 INFO:teuthology.orchestra.run.smithi002.stdout: libquadmath-8.5.0-18.el8.x86_64 2023-01-24T14:41:03.685 INFO:teuthology.orchestra.run.smithi002.stdout: openblas-0.3.15-4.el8.x86_64 2023-01-24T14:41:03.686 INFO:teuthology.orchestra.run.smithi002.stdout: openblas-threads-0.3.15-4.el8.x86_64 2023-01-24T14:41:03.686 INFO:teuthology.orchestra.run.smithi002.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-01-24T14:41:03.686 INFO:teuthology.orchestra.run.smithi002.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-01-24T14:41:03.686 INFO:teuthology.orchestra.run.smithi002.stdout: python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x86_64 2023-01-24T14:41:03.686 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-01-24T14:41:03.687 INFO:teuthology.orchestra.run.smithi002.stdout:Complete! 2023-01-24T14:41:03.853 DEBUG:teuthology.orchestra.run.smithi002:> sudo yum -y install ceph-mgr-rook 2023-01-24T14:41:04.622 INFO:teuthology.orchestra.run.smithi002.stdout:Last metadata expiration check: 0:12:34 ago on Tue 24 Jan 2023 02:28:30 PM UTC. 2023-01-24T14:41:05.851 INFO:teuthology.orchestra.run.smithi002.stdout:Dependencies resolved. 2023-01-24T14:41:05.853 INFO:teuthology.orchestra.run.smithi002.stdout:======================================================================================== 2023-01-24T14:41:05.853 INFO:teuthology.orchestra.run.smithi002.stdout: Package Arch Version Repository Size 2023-01-24T14:41:05.853 INFO:teuthology.orchestra.run.smithi002.stdout:======================================================================================== 2023-01-24T14:41:05.853 INFO:teuthology.orchestra.run.smithi002.stdout:Installing: 2023-01-24T14:41:05.854 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-mgr-rook noarch 2:18.0.0-1954.g1a671c19.el8 ceph-noarch 47 k 2023-01-24T14:41:05.854 INFO:teuthology.orchestra.run.smithi002.stdout:Installing dependencies: 2023-01-24T14:41:05.854 INFO:teuthology.orchestra.run.smithi002.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-01-24T14:41:05.854 INFO:teuthology.orchestra.run.smithi002.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-01-24T14:41:05.854 INFO:teuthology.orchestra.run.smithi002.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-01-24T14:41:05.855 INFO:teuthology.orchestra.run.smithi002.stdout: python3-jsonpatch noarch 1.21-2.el8 CentOS-AppStream 27 k 2023-01-24T14:41:05.855 INFO:teuthology.orchestra.run.smithi002.stdout: python3-jsonpointer noarch 1.10-11.el8 CentOS-AppStream 20 k 2023-01-24T14:41:05.855 INFO:teuthology.orchestra.run.smithi002.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-01-24T14:41:05.855 INFO:teuthology.orchestra.run.smithi002.stdout: python3-oauthlib noarch 2.1.0-1.el8 CentOS-BaseOS 155 k 2023-01-24T14:41:05.855 INFO:teuthology.orchestra.run.smithi002.stdout: python3-pyasn1 noarch 0.3.7-6.el8 CentOS-AppStream 126 k 2023-01-24T14:41:05.856 INFO:teuthology.orchestra.run.smithi002.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 CentOS-AppStream 110 k 2023-01-24T14:41:05.856 INFO:teuthology.orchestra.run.smithi002.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 CentOS-BaseOS 43 k 2023-01-24T14:41:05.856 INFO:teuthology.orchestra.run.smithi002.stdout: python3-rsa noarch 4.8-1.el8 epel 61 k 2023-01-24T14:41:05.856 INFO:teuthology.orchestra.run.smithi002.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-01-24T14:41:05.856 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-01-24T14:41:05.857 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction Summary 2023-01-24T14:41:05.857 INFO:teuthology.orchestra.run.smithi002.stdout:======================================================================================== 2023-01-24T14:41:05.857 INFO:teuthology.orchestra.run.smithi002.stdout:Install 13 Packages 2023-01-24T14:41:05.857 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-01-24T14:41:05.858 INFO:teuthology.orchestra.run.smithi002.stdout:Total download size: 1.9 M 2023-01-24T14:41:05.858 INFO:teuthology.orchestra.run.smithi002.stdout:Installed size: 25 M 2023-01-24T14:41:05.858 INFO:teuthology.orchestra.run.smithi002.stdout:Downloading Packages: 2023-01-24T14:41:05.919 INFO:teuthology.orchestra.run.smithi002.stdout:(1/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 428 kB/s | 27 kB 00:00 2023-01-24T14:41:05.936 INFO:teuthology.orchestra.run.smithi002.stdout:(2/13): python3-jsonpointer-1.10-11.el8.noarch. 254 kB/s | 20 kB 00:00 2023-01-24T14:41:05.961 INFO:teuthology.orchestra.run.smithi002.stdout:(3/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 3.0 MB/s | 126 kB 00:00 2023-01-24T14:41:05.986 INFO:teuthology.orchestra.run.smithi002.stdout:(4/13): python3-pyasn1-modules-0.3.7-6.el8.noar 2.1 MB/s | 110 kB 00:00 2023-01-24T14:41:06.012 INFO:teuthology.orchestra.run.smithi002.stdout:(5/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 3.0 MB/s | 155 kB 00:00 2023-01-24T14:41:06.037 INFO:teuthology.orchestra.run.smithi002.stdout:(6/13): python3-requests-oauthlib-1.0.0-1.el8.n 846 kB/s | 43 kB 00:00 2023-01-24T14:41:06.063 INFO:teuthology.orchestra.run.smithi002.stdout:(7/13): ceph-mgr-rook-18.0.0-1954.g1a671c19.el8 230 kB/s | 47 kB 00:00 2023-01-24T14:41:06.171 INFO:teuthology.orchestra.run.smithi002.stdout:(8/13): python3-certifi-2018.10.15-7.el8.noarch 122 kB/s | 16 kB 00:00 2023-01-24T14:41:06.196 INFO:teuthology.orchestra.run.smithi002.stdout:(9/13): python3-cachetools-3.1.1-4.el8.noarch.r 179 kB/s | 33 kB 00:00 2023-01-24T14:41:06.222 INFO:teuthology.orchestra.run.smithi002.stdout:(10/13): python3-google-auth-1.1.1-10.el8.noarc 511 kB/s | 81 kB 00:00 2023-01-24T14:41:06.314 INFO:teuthology.orchestra.run.smithi002.stdout:(11/13): python3-rsa-4.8-1.el8.noarch.rpm 523 kB/s | 61 kB 00:00 2023-01-24T14:41:06.347 INFO:teuthology.orchestra.run.smithi002.stdout:(12/13): python3-websocket-client-0.56.0-5.el8. 489 kB/s | 61 kB 00:00 2023-01-24T14:41:06.731 INFO:teuthology.orchestra.run.smithi002.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 2.1 MB/s | 1.2 MB 00:00 2023-01-24T14:41:06.732 INFO:teuthology.orchestra.run.smithi002.stdout:-------------------------------------------------------------------------------- 2023-01-24T14:41:06.732 INFO:teuthology.orchestra.run.smithi002.stdout:Total 2.2 MB/s | 1.9 MB 00:00 2023-01-24T14:41:06.787 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction check 2023-01-24T14:41:06.802 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction check succeeded. 2023-01-24T14:41:06.803 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction test 2023-01-24T14:41:06.907 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction test succeeded. 2023-01-24T14:41:06.923 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction 2023-01-24T14:41:07.448 INFO:teuthology.orchestra.run.smithi002.stdout: Preparing : 1/1 2023-01-24T14:41:07.731 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2023-01-24T14:41:07.937 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2023-01-24T14:41:08.188 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-rsa-4.8-1.el8.noarch 3/13 2023-01-24T14:41:08.450 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2023-01-24T14:41:08.658 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2023-01-24T14:41:08.950 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2023-01-24T14:41:09.311 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2023-01-24T14:41:09.768 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2023-01-24T14:41:10.902 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2023-01-24T14:41:11.199 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2023-01-24T14:41:11.450 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2023-01-24T14:41:11.661 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2023-01-24T14:41:11.833 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : ceph-mgr-rook-2:18.0.0-1954.g1a671c19.el8.noarch 13/13 2023-01-24T14:41:12.170 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-1954.g1a671c19.el8.noarch 13/13 2023-01-24T14:41:12.170 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : ceph-mgr-rook-2:18.0.0-1954.g1a671c19.el8.noarch 1/13 2023-01-24T14:41:12.170 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2023-01-24T14:41:12.170 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2023-01-24T14:41:12.171 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2023-01-24T14:41:12.171 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2023-01-24T14:41:12.171 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2023-01-24T14:41:12.171 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2023-01-24T14:41:12.171 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2023-01-24T14:41:12.172 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2023-01-24T14:41:12.172 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2023-01-24T14:41:12.172 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2023-01-24T14:41:12.172 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-rsa-4.8-1.el8.noarch 12/13 2023-01-24T14:41:12.749 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2023-01-24T14:41:12.750 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-01-24T14:41:12.750 INFO:teuthology.orchestra.run.smithi002.stdout:Installed: 2023-01-24T14:41:12.750 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-mgr-rook-2:18.0.0-1954.g1a671c19.el8.noarch 2023-01-24T14:41:12.750 INFO:teuthology.orchestra.run.smithi002.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-01-24T14:41:12.751 INFO:teuthology.orchestra.run.smithi002.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-01-24T14:41:12.751 INFO:teuthology.orchestra.run.smithi002.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-01-24T14:41:12.751 INFO:teuthology.orchestra.run.smithi002.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-01-24T14:41:12.751 INFO:teuthology.orchestra.run.smithi002.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-01-24T14:41:12.751 INFO:teuthology.orchestra.run.smithi002.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-01-24T14:41:12.751 INFO:teuthology.orchestra.run.smithi002.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-01-24T14:41:12.752 INFO:teuthology.orchestra.run.smithi002.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-01-24T14:41:12.752 INFO:teuthology.orchestra.run.smithi002.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-01-24T14:41:12.752 INFO:teuthology.orchestra.run.smithi002.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-01-24T14:41:12.752 INFO:teuthology.orchestra.run.smithi002.stdout: python3-rsa-4.8-1.el8.noarch 2023-01-24T14:41:12.753 INFO:teuthology.orchestra.run.smithi002.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-01-24T14:41:12.753 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-01-24T14:41:12.753 INFO:teuthology.orchestra.run.smithi002.stdout:Complete! 2023-01-24T14:41:12.894 DEBUG:teuthology.orchestra.run.smithi002:> sudo yum -y install ceph-mgr-cephadm 2023-01-24T14:41:13.659 INFO:teuthology.orchestra.run.smithi002.stdout:Last metadata expiration check: 0:12:43 ago on Tue 24 Jan 2023 02:28:30 PM UTC. 2023-01-24T14:41:14.902 INFO:teuthology.orchestra.run.smithi002.stdout:Dependencies resolved. 2023-01-24T14:41:14.903 INFO:teuthology.orchestra.run.smithi002.stdout:================================================================================ 2023-01-24T14:41:14.903 INFO:teuthology.orchestra.run.smithi002.stdout: Package Arch Version Repository Size 2023-01-24T14:41:14.903 INFO:teuthology.orchestra.run.smithi002.stdout:================================================================================ 2023-01-24T14:41:14.903 INFO:teuthology.orchestra.run.smithi002.stdout:Installing: 2023-01-24T14:41:14.904 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-mgr-cephadm noarch 2:18.0.0-1954.g1a671c19.el8 ceph-noarch 120 k 2023-01-24T14:41:14.904 INFO:teuthology.orchestra.run.smithi002.stdout:Installing dependencies: 2023-01-24T14:41:14.904 INFO:teuthology.orchestra.run.smithi002.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-01-24T14:41:14.904 INFO:teuthology.orchestra.run.smithi002.stdout: python3-babel noarch 2.5.1-7.el8 CentOS-AppStream 4.8 M 2023-01-24T14:41:14.904 INFO:teuthology.orchestra.run.smithi002.stdout: python3-jinja2 noarch 2.10.1-3.el8 CentOS-AppStream 538 k 2023-01-24T14:41:14.905 INFO:teuthology.orchestra.run.smithi002.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-01-24T14:41:14.905 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-01-24T14:41:14.905 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction Summary 2023-01-24T14:41:14.905 INFO:teuthology.orchestra.run.smithi002.stdout:================================================================================ 2023-01-24T14:41:14.905 INFO:teuthology.orchestra.run.smithi002.stdout:Install 5 Packages 2023-01-24T14:41:14.906 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-01-24T14:41:14.906 INFO:teuthology.orchestra.run.smithi002.stdout:Total download size: 6.0 M 2023-01-24T14:41:14.907 INFO:teuthology.orchestra.run.smithi002.stdout:Installed size: 26 M 2023-01-24T14:41:14.907 INFO:teuthology.orchestra.run.smithi002.stdout:Downloading Packages: 2023-01-24T14:41:14.989 INFO:teuthology.orchestra.run.smithi002.stdout:(1/5): python3-jinja2-2.10.1-3.el8.noarch.rpm 6.3 MB/s | 538 kB 00:00 2023-01-24T14:41:15.089 INFO:teuthology.orchestra.run.smithi002.stdout:(2/5): python3-babel-2.5.1-7.el8.noarch.rpm 26 MB/s | 4.8 MB 00:00 2023-01-24T14:41:15.281 INFO:teuthology.orchestra.run.smithi002.stdout:(3/5): python3-natsort-7.1.1-2.el8.noarch.rpm 315 kB/s | 60 kB 00:00 2023-01-24T14:41:15.314 INFO:teuthology.orchestra.run.smithi002.stdout:(4/5): ceph-mgr-cephadm-18.0.0-1954.g1a671c19.e 294 kB/s | 120 kB 00:00 2023-01-24T14:41:15.565 INFO:teuthology.orchestra.run.smithi002.stdout:(5/5): python3-asyncssh-2.7.0-2.el8.noarch.rpm 833 kB/s | 476 kB 00:00 2023-01-24T14:41:15.567 INFO:teuthology.orchestra.run.smithi002.stdout:-------------------------------------------------------------------------------- 2023-01-24T14:41:15.567 INFO:teuthology.orchestra.run.smithi002.stdout:Total 9.0 MB/s | 6.0 MB 00:00 2023-01-24T14:41:15.621 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction check 2023-01-24T14:41:15.633 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction check succeeded. 2023-01-24T14:41:15.633 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction test 2023-01-24T14:41:15.776 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction test succeeded. 2023-01-24T14:41:15.780 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction 2023-01-24T14:41:16.101 INFO:teuthology.orchestra.run.smithi002.stdout: Preparing : 1/1 2023-01-24T14:41:16.440 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 1/5 2023-01-24T14:41:17.487 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 2/5 2023-01-24T14:41:17.903 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 3/5 2023-01-24T14:41:18.163 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 4/5 2023-01-24T14:41:18.366 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-1954.g1a671c19.el8.noarch 5/5 2023-01-24T14:41:18.691 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-1954.g1a671c19.el8.noarch 5/5 2023-01-24T14:41:18.691 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-1954.g1a671c19.el8.noarch 1/5 2023-01-24T14:41:18.692 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/5 2023-01-24T14:41:18.692 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 3/5 2023-01-24T14:41:18.692 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 4/5 2023-01-24T14:41:19.217 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 5/5 2023-01-24T14:41:19.217 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-01-24T14:41:19.217 INFO:teuthology.orchestra.run.smithi002.stdout:Installed: 2023-01-24T14:41:19.218 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-mgr-cephadm-2:18.0.0-1954.g1a671c19.el8.noarch 2023-01-24T14:41:19.218 INFO:teuthology.orchestra.run.smithi002.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-01-24T14:41:19.218 INFO:teuthology.orchestra.run.smithi002.stdout: python3-babel-2.5.1-7.el8.noarch 2023-01-24T14:41:19.218 INFO:teuthology.orchestra.run.smithi002.stdout: python3-jinja2-2.10.1-3.el8.noarch 2023-01-24T14:41:19.218 INFO:teuthology.orchestra.run.smithi002.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-01-24T14:41:19.219 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-01-24T14:41:19.219 INFO:teuthology.orchestra.run.smithi002.stdout:Complete! 2023-01-24T14:41:19.338 DEBUG:teuthology.orchestra.run.smithi002:> sudo yum -y install ceph-fuse 2023-01-24T14:41:20.099 INFO:teuthology.orchestra.run.smithi002.stdout:Last metadata expiration check: 0:12:50 ago on Tue 24 Jan 2023 02:28:30 PM UTC. 2023-01-24T14:41:21.330 INFO:teuthology.orchestra.run.smithi002.stdout:Dependencies resolved. 2023-01-24T14:41:21.330 INFO:teuthology.orchestra.run.smithi002.stdout:================================================================================ 2023-01-24T14:41:21.330 INFO:teuthology.orchestra.run.smithi002.stdout: Package Arch Version Repository Size 2023-01-24T14:41:21.331 INFO:teuthology.orchestra.run.smithi002.stdout:================================================================================ 2023-01-24T14:41:21.331 INFO:teuthology.orchestra.run.smithi002.stdout:Installing: 2023-01-24T14:41:21.331 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-fuse x86_64 2:18.0.0-1954.g1a671c19.el8 ceph 814 k 2023-01-24T14:41:21.331 INFO:teuthology.orchestra.run.smithi002.stdout:Installing dependencies: 2023-01-24T14:41:21.332 INFO:teuthology.orchestra.run.smithi002.stdout: fuse x86_64 2.9.7-16.el8 CentOS-BaseOS 83 k 2023-01-24T14:41:21.332 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-01-24T14:41:21.332 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction Summary 2023-01-24T14:41:21.332 INFO:teuthology.orchestra.run.smithi002.stdout:================================================================================ 2023-01-24T14:41:21.332 INFO:teuthology.orchestra.run.smithi002.stdout:Install 2 Packages 2023-01-24T14:41:21.332 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-01-24T14:41:21.333 INFO:teuthology.orchestra.run.smithi002.stdout:Total download size: 897 k 2023-01-24T14:41:21.333 INFO:teuthology.orchestra.run.smithi002.stdout:Installed size: 3.3 M 2023-01-24T14:41:21.333 INFO:teuthology.orchestra.run.smithi002.stdout:Downloading Packages: 2023-01-24T14:41:21.389 INFO:teuthology.orchestra.run.smithi002.stdout:(1/2): fuse-2.9.7-16.el8.x86_64.rpm 1.4 MB/s | 83 kB 00:00 2023-01-24T14:41:22.323 INFO:teuthology.orchestra.run.smithi002.stdout:(2/2): ceph-fuse-18.0.0-1954.g1a671c19.el8.x86_ 822 kB/s | 814 kB 00:00 2023-01-24T14:41:22.324 INFO:teuthology.orchestra.run.smithi002.stdout:-------------------------------------------------------------------------------- 2023-01-24T14:41:22.324 INFO:teuthology.orchestra.run.smithi002.stdout:Total 903 kB/s | 897 kB 00:00 2023-01-24T14:41:22.336 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction check 2023-01-24T14:41:22.360 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction check succeeded. 2023-01-24T14:41:22.360 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction test 2023-01-24T14:41:22.466 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction test succeeded. 2023-01-24T14:41:22.470 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction 2023-01-24T14:41:22.667 INFO:teuthology.orchestra.run.smithi002.stdout: Preparing : 1/1 2023-01-24T14:41:23.113 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : fuse-2.9.7-16.el8.x86_64 1/2 2023-01-24T14:41:23.298 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : ceph-fuse-2:18.0.0-1954.g1a671c19.el8.x86_64 2/2 2023-01-24T14:41:23.723 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: ceph-fuse-2:18.0.0-1954.g1a671c19.el8.x86_64 2/2 2023-01-24T14:41:23.724 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : ceph-fuse-2:18.0.0-1954.g1a671c19.el8.x86_64 1/2 2023-01-24T14:41:24.288 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : fuse-2.9.7-16.el8.x86_64 2/2 2023-01-24T14:41:24.289 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-01-24T14:41:24.289 INFO:teuthology.orchestra.run.smithi002.stdout:Installed: 2023-01-24T14:41:24.289 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-fuse-2:18.0.0-1954.g1a671c19.el8.x86_64 fuse-2.9.7-16.el8.x86_64 2023-01-24T14:41:24.289 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-01-24T14:41:24.290 INFO:teuthology.orchestra.run.smithi002.stdout:Complete! 2023-01-24T14:41:24.523 DEBUG:teuthology.orchestra.run.smithi002:> sudo yum -y install ceph-volume 2023-01-24T14:41:25.292 INFO:teuthology.orchestra.run.smithi002.stdout:Last metadata expiration check: 0:12:55 ago on Tue 24 Jan 2023 02:28:30 PM UTC. 2023-01-24T14:41:26.528 INFO:teuthology.orchestra.run.smithi002.stdout:Dependencies resolved. 2023-01-24T14:41:26.529 INFO:teuthology.orchestra.run.smithi002.stdout:================================================================================ 2023-01-24T14:41:26.530 INFO:teuthology.orchestra.run.smithi002.stdout: Package Arch Version Repository Size 2023-01-24T14:41:26.530 INFO:teuthology.orchestra.run.smithi002.stdout:================================================================================ 2023-01-24T14:41:26.530 INFO:teuthology.orchestra.run.smithi002.stdout:Installing: 2023-01-24T14:41:26.530 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-volume noarch 2:18.0.0-1954.g1a671c19.el8 ceph-noarch 263 k 2023-01-24T14:41:26.531 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-01-24T14:41:26.531 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction Summary 2023-01-24T14:41:26.531 INFO:teuthology.orchestra.run.smithi002.stdout:================================================================================ 2023-01-24T14:41:26.531 INFO:teuthology.orchestra.run.smithi002.stdout:Install 1 Package 2023-01-24T14:41:26.531 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-01-24T14:41:26.532 INFO:teuthology.orchestra.run.smithi002.stdout:Total download size: 263 k 2023-01-24T14:41:26.532 INFO:teuthology.orchestra.run.smithi002.stdout:Installed size: 1.3 M 2023-01-24T14:41:26.532 INFO:teuthology.orchestra.run.smithi002.stdout:Downloading Packages: 2023-01-24T14:41:27.152 INFO:teuthology.orchestra.run.smithi002.stdout:ceph-volume-18.0.0-1954.g1a671c19.el8.noarch.rp 423 kB/s | 263 kB 00:00 2023-01-24T14:41:27.153 INFO:teuthology.orchestra.run.smithi002.stdout:-------------------------------------------------------------------------------- 2023-01-24T14:41:27.153 INFO:teuthology.orchestra.run.smithi002.stdout:Total 422 kB/s | 263 kB 00:00 2023-01-24T14:41:27.153 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction check 2023-01-24T14:41:27.164 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction check succeeded. 2023-01-24T14:41:27.164 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction test 2023-01-24T14:41:27.256 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction test succeeded. 2023-01-24T14:41:27.260 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction 2023-01-24T14:41:27.457 INFO:teuthology.orchestra.run.smithi002.stdout: Preparing : 1/1 2023-01-24T14:41:27.748 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : ceph-volume-2:18.0.0-1954.g1a671c19.el8.noarch 1/1 2023-01-24T14:41:28.192 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: ceph-volume-2:18.0.0-1954.g1a671c19.el8.noarch 1/1 2023-01-24T14:41:28.691 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : ceph-volume-2:18.0.0-1954.g1a671c19.el8.noarch 1/1 2023-01-24T14:41:28.691 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-01-24T14:41:28.691 INFO:teuthology.orchestra.run.smithi002.stdout:Installed: 2023-01-24T14:41:28.692 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-volume-2:18.0.0-1954.g1a671c19.el8.noarch 2023-01-24T14:41:28.692 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-01-24T14:41:28.692 INFO:teuthology.orchestra.run.smithi002.stdout:Complete! 2023-01-24T14:41:28.891 DEBUG:teuthology.orchestra.run.smithi002:> sudo yum -y install librados-devel 2023-01-24T14:41:29.667 INFO:teuthology.orchestra.run.smithi002.stdout:Last metadata expiration check: 0:12:59 ago on Tue 24 Jan 2023 02:28:30 PM UTC. 2023-01-24T14:41:30.882 INFO:teuthology.orchestra.run.smithi002.stdout:Dependencies resolved. 2023-01-24T14:41:30.883 INFO:teuthology.orchestra.run.smithi002.stdout:================================================================================ 2023-01-24T14:41:30.884 INFO:teuthology.orchestra.run.smithi002.stdout: Package Arch Version Repo Size 2023-01-24T14:41:30.884 INFO:teuthology.orchestra.run.smithi002.stdout:================================================================================ 2023-01-24T14:41:30.884 INFO:teuthology.orchestra.run.smithi002.stdout:Installing: 2023-01-24T14:41:30.884 INFO:teuthology.orchestra.run.smithi002.stdout: librados-devel x86_64 2:18.0.0-1954.g1a671c19.el8 ceph 139 k 2023-01-24T14:41:30.884 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-01-24T14:41:30.885 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction Summary 2023-01-24T14:41:30.885 INFO:teuthology.orchestra.run.smithi002.stdout:================================================================================ 2023-01-24T14:41:30.885 INFO:teuthology.orchestra.run.smithi002.stdout:Install 1 Package 2023-01-24T14:41:30.885 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-01-24T14:41:30.885 INFO:teuthology.orchestra.run.smithi002.stdout:Total download size: 139 k 2023-01-24T14:41:30.886 INFO:teuthology.orchestra.run.smithi002.stdout:Installed size: 512 k 2023-01-24T14:41:30.886 INFO:teuthology.orchestra.run.smithi002.stdout:Downloading Packages: 2023-01-24T14:41:31.581 INFO:teuthology.orchestra.run.smithi002.stdout:librados-devel-18.0.0-1954.g1a671c19.el8.x86_64 199 kB/s | 139 kB 00:00 2023-01-24T14:41:31.582 INFO:teuthology.orchestra.run.smithi002.stdout:-------------------------------------------------------------------------------- 2023-01-24T14:41:31.582 INFO:teuthology.orchestra.run.smithi002.stdout:Total 198 kB/s | 139 kB 00:00 2023-01-24T14:41:31.583 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction check 2023-01-24T14:41:31.589 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction check succeeded. 2023-01-24T14:41:31.590 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction test 2023-01-24T14:41:31.648 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction test succeeded. 2023-01-24T14:41:31.652 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction 2023-01-24T14:41:31.774 INFO:teuthology.orchestra.run.smithi002.stdout: Preparing : 1/1 2023-01-24T14:41:31.979 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : librados-devel-2:18.0.0-1954.g1a671c19.el8.x86_64 1/1 2023-01-24T14:41:32.292 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: librados-devel-2:18.0.0-1954.g1a671c19.el8.x86_64 1/1 2023-01-24T14:41:32.784 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : librados-devel-2:18.0.0-1954.g1a671c19.el8.x86_64 1/1 2023-01-24T14:41:32.785 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-01-24T14:41:32.785 INFO:teuthology.orchestra.run.smithi002.stdout:Installed: 2023-01-24T14:41:32.785 INFO:teuthology.orchestra.run.smithi002.stdout: librados-devel-2:18.0.0-1954.g1a671c19.el8.x86_64 2023-01-24T14:41:32.785 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-01-24T14:41:32.786 INFO:teuthology.orchestra.run.smithi002.stdout:Complete! 2023-01-24T14:41:33.009 DEBUG:teuthology.orchestra.run.smithi002:> sudo yum -y install libcephfs2 2023-01-24T14:41:33.772 INFO:teuthology.orchestra.run.smithi002.stdout:Last metadata expiration check: 0:13:03 ago on Tue 24 Jan 2023 02:28:30 PM UTC. 2023-01-24T14:41:34.884 INFO:teuthology.orchestra.run.smithi002.stdout:Package libcephfs2-2:18.0.0-1954.g1a671c19.el8.x86_64 is already installed. 2023-01-24T14:41:34.996 INFO:teuthology.orchestra.run.smithi002.stdout:Dependencies resolved. 2023-01-24T14:41:34.997 INFO:teuthology.orchestra.run.smithi002.stdout:Nothing to do. 2023-01-24T14:41:34.997 INFO:teuthology.orchestra.run.smithi002.stdout:Complete! 2023-01-24T14:41:35.087 DEBUG:teuthology.orchestra.run.smithi002:> sudo yum -y install libcephfs-devel 2023-01-24T14:41:35.851 INFO:teuthology.orchestra.run.smithi002.stdout:Last metadata expiration check: 0:13:05 ago on Tue 24 Jan 2023 02:28:30 PM UTC. 2023-01-24T14:41:37.080 INFO:teuthology.orchestra.run.smithi002.stdout:Dependencies resolved. 2023-01-24T14:41:37.081 INFO:teuthology.orchestra.run.smithi002.stdout:================================================================================ 2023-01-24T14:41:37.082 INFO:teuthology.orchestra.run.smithi002.stdout: Package Arch Version Repo Size 2023-01-24T14:41:37.082 INFO:teuthology.orchestra.run.smithi002.stdout:================================================================================ 2023-01-24T14:41:37.082 INFO:teuthology.orchestra.run.smithi002.stdout:Installing: 2023-01-24T14:41:37.082 INFO:teuthology.orchestra.run.smithi002.stdout: libcephfs-devel x86_64 2:18.0.0-1954.g1a671c19.el8 ceph 30 k 2023-01-24T14:41:37.082 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-01-24T14:41:37.083 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction Summary 2023-01-24T14:41:37.083 INFO:teuthology.orchestra.run.smithi002.stdout:================================================================================ 2023-01-24T14:41:37.083 INFO:teuthology.orchestra.run.smithi002.stdout:Install 1 Package 2023-01-24T14:41:37.083 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-01-24T14:41:37.084 INFO:teuthology.orchestra.run.smithi002.stdout:Total download size: 30 k 2023-01-24T14:41:37.084 INFO:teuthology.orchestra.run.smithi002.stdout:Installed size: 137 k 2023-01-24T14:41:37.084 INFO:teuthology.orchestra.run.smithi002.stdout:Downloading Packages: 2023-01-24T14:41:37.279 INFO:teuthology.orchestra.run.smithi002.stdout:libcephfs-devel-18.0.0-1954.g1a671c19.el8.x86_6 155 kB/s | 30 kB 00:00 2023-01-24T14:41:37.280 INFO:teuthology.orchestra.run.smithi002.stdout:-------------------------------------------------------------------------------- 2023-01-24T14:41:37.280 INFO:teuthology.orchestra.run.smithi002.stdout:Total 154 kB/s | 30 kB 00:00 2023-01-24T14:41:37.281 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction check 2023-01-24T14:41:37.284 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction check succeeded. 2023-01-24T14:41:37.284 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction test 2023-01-24T14:41:37.312 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction test succeeded. 2023-01-24T14:41:37.315 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction 2023-01-24T14:41:37.381 INFO:teuthology.orchestra.run.smithi002.stdout: Preparing : 1/1 2023-01-24T14:41:37.628 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : libcephfs-devel-2:18.0.0-1954.g1a671c19.el8.x86_64 1/1 2023-01-24T14:41:37.907 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: libcephfs-devel-2:18.0.0-1954.g1a671c19.el8.x86_64 1/1 2023-01-24T14:41:38.393 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : libcephfs-devel-2:18.0.0-1954.g1a671c19.el8.x86_64 1/1 2023-01-24T14:41:38.393 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-01-24T14:41:38.394 INFO:teuthology.orchestra.run.smithi002.stdout:Installed: 2023-01-24T14:41:38.394 INFO:teuthology.orchestra.run.smithi002.stdout: libcephfs-devel-2:18.0.0-1954.g1a671c19.el8.x86_64 2023-01-24T14:41:38.394 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-01-24T14:41:38.394 INFO:teuthology.orchestra.run.smithi002.stdout:Complete! 2023-01-24T14:41:38.506 DEBUG:teuthology.orchestra.run.smithi002:> sudo yum -y install librados2 2023-01-24T14:41:39.260 INFO:teuthology.orchestra.run.smithi002.stdout:Last metadata expiration check: 0:13:09 ago on Tue 24 Jan 2023 02:28:30 PM UTC. 2023-01-24T14:41:40.371 INFO:teuthology.orchestra.run.smithi002.stdout:Package librados2-2:18.0.0-1954.g1a671c19.el8.x86_64 is already installed. 2023-01-24T14:41:40.488 INFO:teuthology.orchestra.run.smithi002.stdout:Dependencies resolved. 2023-01-24T14:41:40.488 INFO:teuthology.orchestra.run.smithi002.stdout:Nothing to do. 2023-01-24T14:41:40.489 INFO:teuthology.orchestra.run.smithi002.stdout:Complete! 2023-01-24T14:41:40.587 DEBUG:teuthology.orchestra.run.smithi002:> sudo yum -y install librbd1 2023-01-24T14:41:41.355 INFO:teuthology.orchestra.run.smithi002.stdout:Last metadata expiration check: 0:13:11 ago on Tue 24 Jan 2023 02:28:30 PM UTC. 2023-01-24T14:41:42.477 INFO:teuthology.orchestra.run.smithi002.stdout:Package librbd1-2:18.0.0-1954.g1a671c19.el8.x86_64 is already installed. 2023-01-24T14:41:42.591 INFO:teuthology.orchestra.run.smithi002.stdout:Dependencies resolved. 2023-01-24T14:41:42.592 INFO:teuthology.orchestra.run.smithi002.stdout:Nothing to do. 2023-01-24T14:41:42.593 INFO:teuthology.orchestra.run.smithi002.stdout:Complete! 2023-01-24T14:41:42.692 DEBUG:teuthology.orchestra.run.smithi002:> sudo yum -y install python3-rados 2023-01-24T14:41:43.461 INFO:teuthology.orchestra.run.smithi002.stdout:Last metadata expiration check: 0:13:13 ago on Tue 24 Jan 2023 02:28:30 PM UTC. 2023-01-24T14:41:44.572 INFO:teuthology.orchestra.run.smithi002.stdout:Package python3-rados-2:18.0.0-1954.g1a671c19.el8.x86_64 is already installed. 2023-01-24T14:41:44.685 INFO:teuthology.orchestra.run.smithi002.stdout:Dependencies resolved. 2023-01-24T14:41:44.686 INFO:teuthology.orchestra.run.smithi002.stdout:Nothing to do. 2023-01-24T14:41:44.686 INFO:teuthology.orchestra.run.smithi002.stdout:Complete! 2023-01-24T14:41:44.786 DEBUG:teuthology.orchestra.run.smithi002:> sudo yum -y install python3-rgw 2023-01-24T14:41:45.542 INFO:teuthology.orchestra.run.smithi002.stdout:Last metadata expiration check: 0:13:15 ago on Tue 24 Jan 2023 02:28:30 PM UTC. 2023-01-24T14:41:46.650 INFO:teuthology.orchestra.run.smithi002.stdout:Package python3-rgw-2:18.0.0-1954.g1a671c19.el8.x86_64 is already installed. 2023-01-24T14:41:46.762 INFO:teuthology.orchestra.run.smithi002.stdout:Dependencies resolved. 2023-01-24T14:41:46.763 INFO:teuthology.orchestra.run.smithi002.stdout:Nothing to do. 2023-01-24T14:41:46.763 INFO:teuthology.orchestra.run.smithi002.stdout:Complete! 2023-01-24T14:41:46.849 DEBUG:teuthology.orchestra.run.smithi002:> sudo yum -y install python3-cephfs 2023-01-24T14:41:47.618 INFO:teuthology.orchestra.run.smithi002.stdout:Last metadata expiration check: 0:13:17 ago on Tue 24 Jan 2023 02:28:30 PM UTC. 2023-01-24T14:41:48.737 INFO:teuthology.orchestra.run.smithi002.stdout:Package python3-cephfs-2:18.0.0-1954.g1a671c19.el8.x86_64 is already installed. 2023-01-24T14:41:48.848 INFO:teuthology.orchestra.run.smithi002.stdout:Dependencies resolved. 2023-01-24T14:41:48.849 INFO:teuthology.orchestra.run.smithi002.stdout:Nothing to do. 2023-01-24T14:41:48.849 INFO:teuthology.orchestra.run.smithi002.stdout:Complete! 2023-01-24T14:41:48.944 DEBUG:teuthology.orchestra.run.smithi002:> sudo yum -y install python3-rbd 2023-01-24T14:41:49.703 INFO:teuthology.orchestra.run.smithi002.stdout:Last metadata expiration check: 0:13:19 ago on Tue 24 Jan 2023 02:28:30 PM UTC. 2023-01-24T14:41:50.811 INFO:teuthology.orchestra.run.smithi002.stdout:Package python3-rbd-2:18.0.0-1954.g1a671c19.el8.x86_64 is already installed. 2023-01-24T14:41:50.924 INFO:teuthology.orchestra.run.smithi002.stdout:Dependencies resolved. 2023-01-24T14:41:50.925 INFO:teuthology.orchestra.run.smithi002.stdout:Nothing to do. 2023-01-24T14:41:50.925 INFO:teuthology.orchestra.run.smithi002.stdout:Complete! 2023-01-24T14:41:51.020 DEBUG:teuthology.orchestra.run.smithi002:> sudo yum -y install rbd-fuse 2023-01-24T14:41:51.778 INFO:teuthology.orchestra.run.smithi002.stdout:Last metadata expiration check: 0:13:21 ago on Tue 24 Jan 2023 02:28:30 PM UTC. 2023-01-24T14:41:53.018 INFO:teuthology.orchestra.run.smithi002.stdout:Dependencies resolved. 2023-01-24T14:41:53.019 INFO:teuthology.orchestra.run.smithi002.stdout:================================================================================ 2023-01-24T14:41:53.019 INFO:teuthology.orchestra.run.smithi002.stdout: Package Architecture Version Repository Size 2023-01-24T14:41:53.019 INFO:teuthology.orchestra.run.smithi002.stdout:================================================================================ 2023-01-24T14:41:53.020 INFO:teuthology.orchestra.run.smithi002.stdout:Installing: 2023-01-24T14:41:53.020 INFO:teuthology.orchestra.run.smithi002.stdout: rbd-fuse x86_64 2:18.0.0-1954.g1a671c19.el8 ceph 91 k 2023-01-24T14:41:53.020 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-01-24T14:41:53.020 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction Summary 2023-01-24T14:41:53.020 INFO:teuthology.orchestra.run.smithi002.stdout:================================================================================ 2023-01-24T14:41:53.021 INFO:teuthology.orchestra.run.smithi002.stdout:Install 1 Package 2023-01-24T14:41:53.021 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-01-24T14:41:53.021 INFO:teuthology.orchestra.run.smithi002.stdout:Total download size: 91 k 2023-01-24T14:41:53.021 INFO:teuthology.orchestra.run.smithi002.stdout:Installed size: 303 k 2023-01-24T14:41:53.021 INFO:teuthology.orchestra.run.smithi002.stdout:Downloading Packages: 2023-01-24T14:41:53.434 INFO:teuthology.orchestra.run.smithi002.stdout:rbd-fuse-18.0.0-1954.g1a671c19.el8.x86_64.rpm 220 kB/s | 91 kB 00:00 2023-01-24T14:41:53.435 INFO:teuthology.orchestra.run.smithi002.stdout:-------------------------------------------------------------------------------- 2023-01-24T14:41:53.435 INFO:teuthology.orchestra.run.smithi002.stdout:Total 219 kB/s | 91 kB 00:00 2023-01-24T14:41:53.436 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction check 2023-01-24T14:41:53.446 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction check succeeded. 2023-01-24T14:41:53.446 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction test 2023-01-24T14:41:53.505 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction test succeeded. 2023-01-24T14:41:53.509 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction 2023-01-24T14:41:53.633 INFO:teuthology.orchestra.run.smithi002.stdout: Preparing : 1/1 2023-01-24T14:41:53.893 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : rbd-fuse-2:18.0.0-1954.g1a671c19.el8.x86_64 1/1 2023-01-24T14:41:54.218 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: rbd-fuse-2:18.0.0-1954.g1a671c19.el8.x86_64 1/1 2023-01-24T14:41:54.763 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : rbd-fuse-2:18.0.0-1954.g1a671c19.el8.x86_64 1/1 2023-01-24T14:41:54.763 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-01-24T14:41:54.763 INFO:teuthology.orchestra.run.smithi002.stdout:Installed: 2023-01-24T14:41:54.764 INFO:teuthology.orchestra.run.smithi002.stdout: rbd-fuse-2:18.0.0-1954.g1a671c19.el8.x86_64 2023-01-24T14:41:54.764 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-01-24T14:41:54.764 INFO:teuthology.orchestra.run.smithi002.stdout:Complete! 2023-01-24T14:41:54.942 DEBUG:teuthology.orchestra.run.smithi002:> sudo yum -y install rbd-mirror 2023-01-24T14:41:55.705 INFO:teuthology.orchestra.run.smithi002.stdout:Last metadata expiration check: 0:13:25 ago on Tue 24 Jan 2023 02:28:30 PM UTC. 2023-01-24T14:41:56.939 INFO:teuthology.orchestra.run.smithi002.stdout:Dependencies resolved. 2023-01-24T14:41:56.940 INFO:teuthology.orchestra.run.smithi002.stdout:================================================================================ 2023-01-24T14:41:56.940 INFO:teuthology.orchestra.run.smithi002.stdout: Package Arch Version Repo Size 2023-01-24T14:41:56.940 INFO:teuthology.orchestra.run.smithi002.stdout:================================================================================ 2023-01-24T14:41:56.940 INFO:teuthology.orchestra.run.smithi002.stdout:Installing: 2023-01-24T14:41:56.941 INFO:teuthology.orchestra.run.smithi002.stdout: rbd-mirror x86_64 2:18.0.0-1954.g1a671c19.el8 ceph 3.3 M 2023-01-24T14:41:56.941 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-01-24T14:41:56.941 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction Summary 2023-01-24T14:41:56.941 INFO:teuthology.orchestra.run.smithi002.stdout:================================================================================ 2023-01-24T14:41:56.942 INFO:teuthology.orchestra.run.smithi002.stdout:Install 1 Package 2023-01-24T14:41:56.942 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-01-24T14:41:56.942 INFO:teuthology.orchestra.run.smithi002.stdout:Total download size: 3.3 M 2023-01-24T14:41:56.942 INFO:teuthology.orchestra.run.smithi002.stdout:Installed size: 18 M 2023-01-24T14:41:56.943 INFO:teuthology.orchestra.run.smithi002.stdout:Downloading Packages: 2023-01-24T14:41:59.063 INFO:teuthology.orchestra.run.smithi002.stdout:rbd-mirror-18.0.0-1954.g1a671c19.el8.x86_64.rpm 1.6 MB/s | 3.3 MB 00:02 2023-01-24T14:41:59.065 INFO:teuthology.orchestra.run.smithi002.stdout:-------------------------------------------------------------------------------- 2023-01-24T14:41:59.066 INFO:teuthology.orchestra.run.smithi002.stdout:Total 1.6 MB/s | 3.3 MB 00:02 2023-01-24T14:41:59.066 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction check 2023-01-24T14:41:59.078 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction check succeeded. 2023-01-24T14:41:59.079 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction test 2023-01-24T14:41:59.153 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction test succeeded. 2023-01-24T14:41:59.157 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction 2023-01-24T14:41:59.631 INFO:teuthology.orchestra.run.smithi002.stdout: Preparing : 1/1 2023-01-24T14:41:59.827 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : rbd-mirror-2:18.0.0-1954.g1a671c19.el8.x86_64 1/1 2023-01-24T14:42:00.268 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: rbd-mirror-2:18.0.0-1954.g1a671c19.el8.x86_64 1/1 2023-01-24T14:42:00.828 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : rbd-mirror-2:18.0.0-1954.g1a671c19.el8.x86_64 1/1 2023-01-24T14:42:00.828 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-01-24T14:42:00.828 INFO:teuthology.orchestra.run.smithi002.stdout:Installed: 2023-01-24T14:42:00.828 INFO:teuthology.orchestra.run.smithi002.stdout: rbd-mirror-2:18.0.0-1954.g1a671c19.el8.x86_64 2023-01-24T14:42:00.829 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-01-24T14:42:00.829 INFO:teuthology.orchestra.run.smithi002.stdout:Complete! 2023-01-24T14:42:01.057 DEBUG:teuthology.orchestra.run.smithi002:> sudo yum -y install rbd-nbd 2023-01-24T14:42:01.821 INFO:teuthology.orchestra.run.smithi002.stdout:Last metadata expiration check: 0:13:31 ago on Tue 24 Jan 2023 02:28:30 PM UTC. 2023-01-24T14:42:03.050 INFO:teuthology.orchestra.run.smithi002.stdout:Dependencies resolved. 2023-01-24T14:42:03.051 INFO:teuthology.orchestra.run.smithi002.stdout:================================================================================ 2023-01-24T14:42:03.051 INFO:teuthology.orchestra.run.smithi002.stdout: Package Architecture Version Repository Size 2023-01-24T14:42:03.052 INFO:teuthology.orchestra.run.smithi002.stdout:================================================================================ 2023-01-24T14:42:03.052 INFO:teuthology.orchestra.run.smithi002.stdout:Installing: 2023-01-24T14:42:03.052 INFO:teuthology.orchestra.run.smithi002.stdout: rbd-nbd x86_64 2:18.0.0-1954.g1a671c19.el8 ceph 177 k 2023-01-24T14:42:03.052 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-01-24T14:42:03.053 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction Summary 2023-01-24T14:42:03.053 INFO:teuthology.orchestra.run.smithi002.stdout:================================================================================ 2023-01-24T14:42:03.053 INFO:teuthology.orchestra.run.smithi002.stdout:Install 1 Package 2023-01-24T14:42:03.053 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-01-24T14:42:03.053 INFO:teuthology.orchestra.run.smithi002.stdout:Total download size: 177 k 2023-01-24T14:42:03.054 INFO:teuthology.orchestra.run.smithi002.stdout:Installed size: 626 k 2023-01-24T14:42:03.054 INFO:teuthology.orchestra.run.smithi002.stdout:Downloading Packages: 2023-01-24T14:42:03.582 INFO:teuthology.orchestra.run.smithi002.stdout:rbd-nbd-18.0.0-1954.g1a671c19.el8.x86_64.rpm 333 kB/s | 177 kB 00:00 2023-01-24T14:42:03.582 INFO:teuthology.orchestra.run.smithi002.stdout:-------------------------------------------------------------------------------- 2023-01-24T14:42:03.583 INFO:teuthology.orchestra.run.smithi002.stdout:Total 333 kB/s | 177 kB 00:00 2023-01-24T14:42:03.583 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction check 2023-01-24T14:42:03.595 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction check succeeded. 2023-01-24T14:42:03.595 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction test 2023-01-24T14:42:03.652 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction test succeeded. 2023-01-24T14:42:03.656 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction 2023-01-24T14:42:03.781 INFO:teuthology.orchestra.run.smithi002.stdout: Preparing : 1/1 2023-01-24T14:42:03.990 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : rbd-nbd-2:18.0.0-1954.g1a671c19.el8.x86_64 1/1 2023-01-24T14:42:04.294 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: rbd-nbd-2:18.0.0-1954.g1a671c19.el8.x86_64 1/1 2023-01-24T14:42:04.773 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : rbd-nbd-2:18.0.0-1954.g1a671c19.el8.x86_64 1/1 2023-01-24T14:42:04.773 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-01-24T14:42:04.773 INFO:teuthology.orchestra.run.smithi002.stdout:Installed: 2023-01-24T14:42:04.774 INFO:teuthology.orchestra.run.smithi002.stdout: rbd-nbd-2:18.0.0-1954.g1a671c19.el8.x86_64 2023-01-24T14:42:04.774 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-01-24T14:42:04.774 INFO:teuthology.orchestra.run.smithi002.stdout:Complete! 2023-01-24T14:42:04.996 DEBUG:teuthology.parallel:result is None 2023-01-24T14:42:04.997 DEBUG:teuthology.orchestra.run.smithi002:> sudo lsb_release -is 2023-01-24T14:42:05.036 INFO:teuthology.orchestra.run.smithi002.stdout:CentOSStream 2023-01-24T14:42:05.038 DEBUG:teuthology.misc:System to be installed: CentOSStream 2023-01-24T14:42:05.038 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F8%2Fx86_64&sha1=1a671c198559f54389cd5afa22085a0c7b29a1ef 2023-01-24T14:42:05.191 DEBUG:teuthology.orchestra.run.smithi002:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-01-24T14:42:05.215 INFO:teuthology.orchestra.run.smithi002.stdout:18.0.0-1954.g1a671c19.el8 2023-01-24T14:42:05.216 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-1954.g1a671c19.el8 2023-01-24T14:42:05.217 INFO:teuthology.task.install:The correct ceph version 18.0.0-1954.g1a671c19 is installed. 2023-01-24T14:42:05.218 DEBUG:teuthology.orchestra.run.smithi039:> sudo lsb_release -is 2023-01-24T14:42:05.260 INFO:teuthology.orchestra.run.smithi039.stdout:CentOSStream 2023-01-24T14:42:05.261 DEBUG:teuthology.misc:System to be installed: CentOSStream 2023-01-24T14:42:05.262 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F8%2Fx86_64&sha1=1a671c198559f54389cd5afa22085a0c7b29a1ef 2023-01-24T14:42:05.412 DEBUG:teuthology.orchestra.run.smithi039:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-01-24T14:42:05.437 INFO:teuthology.orchestra.run.smithi039.stdout:18.0.0-1954.g1a671c19.el8 2023-01-24T14:42:05.437 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-1954.g1a671c19.el8 2023-01-24T14:42:05.437 INFO:teuthology.task.install:The correct ceph version 18.0.0-1954.g1a671c19 is installed. 2023-01-24T14:42:05.438 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-01-24T14:42:05.438 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2023-01-24T14:42:05.439 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-01-24T14:42:05.467 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2023-01-24T14:42:05.467 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-01-24T14:42:05.507 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-01-24T14:42:05.507 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2023-01-24T14:42:05.508 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd of=/usr/bin/daemon-helper 2023-01-24T14:42:05.536 DEBUG:teuthology.orchestra.run.smithi002:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-01-24T14:42:05.604 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2023-01-24T14:42:05.605 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd of=/usr/bin/daemon-helper 2023-01-24T14:42:05.634 DEBUG:teuthology.orchestra.run.smithi039:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-01-24T14:42:05.703 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-01-24T14:42:05.704 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2023-01-24T14:42:05.704 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd of=/usr/bin/adjust-ulimits 2023-01-24T14:42:05.732 DEBUG:teuthology.orchestra.run.smithi002:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-01-24T14:42:05.801 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2023-01-24T14:42:05.801 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd of=/usr/bin/adjust-ulimits 2023-01-24T14:42:05.830 DEBUG:teuthology.orchestra.run.smithi039:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-01-24T14:42:05.898 INFO:teuthology.run_tasks:Running task ceph... 2023-01-24T14:42:05.964 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2023-01-24T14:42:05.965 DEBUG:teuthology.orchestra.run.smithi002:> sudo chmod 777 /var/log/ceph 2023-01-24T14:42:05.968 DEBUG:teuthology.orchestra.run.smithi039:> sudo chmod 777 /var/log/ceph 2023-01-24T14:42:05.996 INFO:tasks.ceph:Disabling ceph logrotate... 2023-01-24T14:42:05.996 DEBUG:teuthology.orchestra.run.smithi002:> sudo rm -f -- /etc/logrotate.d/ceph 2023-01-24T14:42:06.038 DEBUG:teuthology.orchestra.run.smithi039:> sudo rm -f -- /etc/logrotate.d/ceph 2023-01-24T14:42:06.066 INFO:tasks.ceph:Creating extra log directories... 2023-01-24T14:42:06.066 DEBUG:teuthology.orchestra.run.smithi002:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-01-24T14:42:06.106 DEBUG:teuthology.orchestra.run.smithi039:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-01-24T14:42:06.144 INFO:tasks.ceph:Creating ceph cluster ceph... 2023-01-24T14:42:06.144 INFO:tasks.ceph:config {'conf': {'mon': {'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10, 'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'osd shutdown pgref assert': True}, 'global': {'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(PG_', '\\(OSD_', '\\(OBJECT_', '\\(POOL_APP_NOT_ENABLED\\)'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-01-24T14:42:06.145 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/rzarzynski-2023-01-24_14:08:39-crimson-rados-main-distro-default-smithi/7135315', 'branch': 'main', 'description': 'crimson-rados/basic/{centos_latest clusters/fixed-2 crimson_qa_overrides deploy/ceph tasks/rados_python}', 'email': None, 'first_in_suite': False, 'job_id': '7135315', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'rzarzynski-2023-01-24_14:08:39-crimson-rados-main-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'os_type': 'centos', 'os_version': '8.stream', 'overrides': {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'global': {'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(PG_', '\\(OSD_', '\\(OBJECT_', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': '1a671c198559f54389cd5afa22085a0c7b29a1ef'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}}}, 'install': {'ceph': {'flavor': 'crimson', 'sha1': '1a671c198559f54389cd5afa22085a0c7b29a1ef'}}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'workunit': {'branch': 'main', 'env': {'CRIMSON_COMPAT': '1'}, 'sha1': '510284b66513490445619d1430aa869868c71a09'}}, 'owner': 'scheduled_rzarzynski@teuthology', 'priority': 60, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'osd.0', 'osd.1', 'client.0', 'node-exporter.a'], ['mgr.x', 'osd.2', 'osd.3', 'client.1', 'prometheus.a', 'node-exporter.b']], 'sha1': '1a671c198559f54389cd5afa22085a0c7b29a1ef', 'sleep_before_teardown': 0, 'suite': 'crimson-rados', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_510284b66513490445619d1430aa869868c71a09/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '510284b66513490445619d1430aa869868c71a09', 'targets': {'smithi002.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJx0+gMikQqwdjPEHuD51w6FELOppJvYH1UJ4tUlNN9c5SvEHEwW7Upjb8SVQrE3vpg0ftDnWrtyoUpxLS6Mnq8=', 'smithi039.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC9DJRQAR3yJe+Cm2yKBWSi166E6Ri+4Ig4WOqTjM+nJWiRQ+TQp4q28NtgDtMBJFykmeBqA98y5mpifcEN10Dt852OyQQWDWCT7t8oWA6w32C/jc8llTrodbuFvZwuCcFBOn9TWziNmysTJNLMd0BfvEmTc4L1wrYy/ds3R/29K41E5Q1RWhkb+E5Z5byG/kWkrgb2jomXexJZZHt5lFgx7mGDxr+pyIGM8PmSfSoiRhyQ2Tf8Sd6KbiNwmFjkglOOkweLwvjbykAmLoKjItl/o+AWDvgc+0/n/gkzBCtDXFpijbWN4ZHLrv6vtiGkoU6rtg7l5eRs5kxK6QZ8HDAv9P1pUpSfIMGCbtI1Z1LXWFHnsRsuFJfPhLyK/CF1sG9svHBBbOzlC3K4OfS6Xv8MF1V0iDhd7R3Thcih/XkYQ2nF0dWD/ktc5+tvVmhTIsuoexAKdjhqv82HYVQgPpwl/wynbhB1UebXQWl1oHrwF4LhYC90Whkpg4CLTNnvHVc='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'conf': {'mon': {'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10, 'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'osd shutdown pgref assert': True}, 'global': {'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(PG_', '\\(OSD_', '\\(OBJECT_', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': '1a671c198559f54389cd5afa22085a0c7b29a1ef', 'cluster': 'ceph'}}, {'workunit': {'clients': {'client.0': ["rados/test_python.sh --eval-attr 'not (wait or tier or ec or bench or stats)'"]}, 'timeout': '1h'}}], 'teuthology_branch': 'main', 'teuthology_sha1': '8e1bba12e584b7ae912e92e94afe9c075ab51fa3', 'timestamp': '2023-01-24_14:08:39', 'tube': 'smithi', 'user': 'rzarzynski', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.17735'} 2023-01-24T14:42:06.145 DEBUG:teuthology.orchestra.run.smithi002:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-01-24T14:42:06.176 DEBUG:teuthology.orchestra.run.smithi039:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-01-24T14:42:06.194 DEBUG:teuthology.orchestra.run.smithi002:> sudo install -d -m0777 -- /var/run/ceph 2023-01-24T14:42:06.234 DEBUG:teuthology.orchestra.run.smithi039:> sudo install -d -m0777 -- /var/run/ceph 2023-01-24T14:42:06.263 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2023-01-24T14:42:06.263 DEBUG:teuthology.orchestra.run.smithi002:> dd if=/scratch_devs of=/dev/stdout 2023-01-24T14:42:06.319 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-01-24T14:42:06.319 DEBUG:teuthology.orchestra.run.smithi002:> stat /dev/vg_nvme/lv_1 2023-01-24T14:42:06.378 INFO:teuthology.orchestra.run.smithi002.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-01-24T14:42:06.378 INFO:teuthology.orchestra.run.smithi002.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-01-24T14:42:06.379 INFO:teuthology.orchestra.run.smithi002.stdout:Device: 6h/6d Inode: 374061 Links: 1 2023-01-24T14:42:06.379 INFO:teuthology.orchestra.run.smithi002.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-01-24T14:42:06.379 INFO:teuthology.orchestra.run.smithi002.stdout:Context: system_u:object_r:device_t:s0 2023-01-24T14:42:06.379 INFO:teuthology.orchestra.run.smithi002.stdout:Access: 2023-01-24 14:41:59.993177094 +0000 2023-01-24T14:42:06.380 INFO:teuthology.orchestra.run.smithi002.stdout:Modify: 2023-01-24 14:27:40.362471184 +0000 2023-01-24T14:42:06.380 INFO:teuthology.orchestra.run.smithi002.stdout:Change: 2023-01-24 14:27:40.362471184 +0000 2023-01-24T14:42:06.380 INFO:teuthology.orchestra.run.smithi002.stdout: Birth: - 2023-01-24T14:42:06.380 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-01-24T14:42:06.445 INFO:teuthology.orchestra.run.smithi002.stderr:1+0 records in 2023-01-24T14:42:06.445 INFO:teuthology.orchestra.run.smithi002.stderr:1+0 records out 2023-01-24T14:42:06.446 INFO:teuthology.orchestra.run.smithi002.stderr:512 bytes copied, 8.9831e-05 s, 5.7 MB/s 2023-01-24T14:42:06.447 DEBUG:teuthology.orchestra.run.smithi002:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-01-24T14:42:06.506 DEBUG:teuthology.orchestra.run.smithi002:> stat /dev/vg_nvme/lv_2 2023-01-24T14:42:06.564 INFO:teuthology.orchestra.run.smithi002.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-01-24T14:42:06.565 INFO:teuthology.orchestra.run.smithi002.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-01-24T14:42:06.565 INFO:teuthology.orchestra.run.smithi002.stdout:Device: 6h/6d Inode: 375603 Links: 1 2023-01-24T14:42:06.565 INFO:teuthology.orchestra.run.smithi002.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-01-24T14:42:06.565 INFO:teuthology.orchestra.run.smithi002.stdout:Context: system_u:object_r:device_t:s0 2023-01-24T14:42:06.565 INFO:teuthology.orchestra.run.smithi002.stdout:Access: 2023-01-24 14:41:59.993177094 +0000 2023-01-24T14:42:06.566 INFO:teuthology.orchestra.run.smithi002.stdout:Modify: 2023-01-24 14:27:40.897459204 +0000 2023-01-24T14:42:06.566 INFO:teuthology.orchestra.run.smithi002.stdout:Change: 2023-01-24 14:27:40.897459204 +0000 2023-01-24T14:42:06.566 INFO:teuthology.orchestra.run.smithi002.stdout: Birth: - 2023-01-24T14:42:06.567 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-01-24T14:42:06.630 INFO:teuthology.orchestra.run.smithi002.stderr:1+0 records in 2023-01-24T14:42:06.631 INFO:teuthology.orchestra.run.smithi002.stderr:1+0 records out 2023-01-24T14:42:06.631 INFO:teuthology.orchestra.run.smithi002.stderr:512 bytes copied, 9.3216e-05 s, 5.5 MB/s 2023-01-24T14:42:06.632 DEBUG:teuthology.orchestra.run.smithi002:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-01-24T14:42:06.692 DEBUG:teuthology.orchestra.run.smithi002:> stat /dev/vg_nvme/lv_3 2023-01-24T14:42:06.750 INFO:teuthology.orchestra.run.smithi002.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-01-24T14:42:06.750 INFO:teuthology.orchestra.run.smithi002.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-01-24T14:42:06.750 INFO:teuthology.orchestra.run.smithi002.stdout:Device: 6h/6d Inode: 373420 Links: 1 2023-01-24T14:42:06.751 INFO:teuthology.orchestra.run.smithi002.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-01-24T14:42:06.751 INFO:teuthology.orchestra.run.smithi002.stdout:Context: system_u:object_r:device_t:s0 2023-01-24T14:42:06.751 INFO:teuthology.orchestra.run.smithi002.stdout:Access: 2023-01-24 14:41:59.993177094 +0000 2023-01-24T14:42:06.751 INFO:teuthology.orchestra.run.smithi002.stdout:Modify: 2023-01-24 14:27:41.413447649 +0000 2023-01-24T14:42:06.752 INFO:teuthology.orchestra.run.smithi002.stdout:Change: 2023-01-24 14:27:41.413447649 +0000 2023-01-24T14:42:06.752 INFO:teuthology.orchestra.run.smithi002.stdout: Birth: - 2023-01-24T14:42:06.753 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-01-24T14:42:06.817 INFO:teuthology.orchestra.run.smithi002.stderr:1+0 records in 2023-01-24T14:42:06.817 INFO:teuthology.orchestra.run.smithi002.stderr:1+0 records out 2023-01-24T14:42:06.818 INFO:teuthology.orchestra.run.smithi002.stderr:512 bytes copied, 9.9589e-05 s, 5.1 MB/s 2023-01-24T14:42:06.819 DEBUG:teuthology.orchestra.run.smithi002:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-01-24T14:42:06.879 DEBUG:teuthology.orchestra.run.smithi002:> stat /dev/vg_nvme/lv_4 2023-01-24T14:42:06.938 INFO:teuthology.orchestra.run.smithi002.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-01-24T14:42:06.938 INFO:teuthology.orchestra.run.smithi002.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-01-24T14:42:06.938 INFO:teuthology.orchestra.run.smithi002.stdout:Device: 6h/6d Inode: 370265 Links: 1 2023-01-24T14:42:06.939 INFO:teuthology.orchestra.run.smithi002.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-01-24T14:42:06.939 INFO:teuthology.orchestra.run.smithi002.stdout:Context: system_u:object_r:device_t:s0 2023-01-24T14:42:06.939 INFO:teuthology.orchestra.run.smithi002.stdout:Access: 2023-01-24 14:41:59.993177094 +0000 2023-01-24T14:42:06.940 INFO:teuthology.orchestra.run.smithi002.stdout:Modify: 2023-01-24 14:27:41.941435826 +0000 2023-01-24T14:42:06.940 INFO:teuthology.orchestra.run.smithi002.stdout:Change: 2023-01-24 14:27:41.941435826 +0000 2023-01-24T14:42:06.940 INFO:teuthology.orchestra.run.smithi002.stdout: Birth: - 2023-01-24T14:42:06.941 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-01-24T14:42:07.005 INFO:teuthology.orchestra.run.smithi002.stderr:1+0 records in 2023-01-24T14:42:07.005 INFO:teuthology.orchestra.run.smithi002.stderr:1+0 records out 2023-01-24T14:42:07.005 INFO:teuthology.orchestra.run.smithi002.stderr:512 bytes copied, 9.8495e-05 s, 5.2 MB/s 2023-01-24T14:42:07.007 DEBUG:teuthology.orchestra.run.smithi002:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-01-24T14:42:07.065 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2'} 2023-01-24T14:42:07.066 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2023-01-24T14:42:07.066 DEBUG:teuthology.orchestra.run.smithi039:> dd if=/scratch_devs of=/dev/stdout 2023-01-24T14:42:07.086 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-01-24T14:42:07.086 DEBUG:teuthology.orchestra.run.smithi039:> stat /dev/vg_nvme/lv_1 2023-01-24T14:42:07.145 INFO:teuthology.orchestra.run.smithi039.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-01-24T14:42:07.146 INFO:teuthology.orchestra.run.smithi039.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-01-24T14:42:07.146 INFO:teuthology.orchestra.run.smithi039.stdout:Device: 6h/6d Inode: 371885 Links: 1 2023-01-24T14:42:07.146 INFO:teuthology.orchestra.run.smithi039.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-01-24T14:42:07.146 INFO:teuthology.orchestra.run.smithi039.stdout:Context: system_u:object_r:device_t:s0 2023-01-24T14:42:07.147 INFO:teuthology.orchestra.run.smithi039.stdout:Access: 2023-01-24 14:40:46.878425753 +0000 2023-01-24T14:42:07.147 INFO:teuthology.orchestra.run.smithi039.stdout:Modify: 2023-01-24 14:27:40.390302808 +0000 2023-01-24T14:42:07.147 INFO:teuthology.orchestra.run.smithi039.stdout:Change: 2023-01-24 14:27:40.390302808 +0000 2023-01-24T14:42:07.147 INFO:teuthology.orchestra.run.smithi039.stdout: Birth: - 2023-01-24T14:42:07.148 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-01-24T14:42:07.215 INFO:teuthology.orchestra.run.smithi039.stderr:1+0 records in 2023-01-24T14:42:07.215 INFO:teuthology.orchestra.run.smithi039.stderr:1+0 records out 2023-01-24T14:42:07.215 INFO:teuthology.orchestra.run.smithi039.stderr:512 bytes copied, 0.000392689 s, 1.3 MB/s 2023-01-24T14:42:07.216 DEBUG:teuthology.orchestra.run.smithi039:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-01-24T14:42:07.277 DEBUG:teuthology.orchestra.run.smithi039:> stat /dev/vg_nvme/lv_2 2023-01-24T14:42:07.337 INFO:teuthology.orchestra.run.smithi039.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-01-24T14:42:07.337 INFO:teuthology.orchestra.run.smithi039.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-01-24T14:42:07.338 INFO:teuthology.orchestra.run.smithi039.stdout:Device: 6h/6d Inode: 374865 Links: 1 2023-01-24T14:42:07.338 INFO:teuthology.orchestra.run.smithi039.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-01-24T14:42:07.338 INFO:teuthology.orchestra.run.smithi039.stdout:Context: system_u:object_r:device_t:s0 2023-01-24T14:42:07.338 INFO:teuthology.orchestra.run.smithi039.stdout:Access: 2023-01-24 14:40:46.878425753 +0000 2023-01-24T14:42:07.338 INFO:teuthology.orchestra.run.smithi039.stdout:Modify: 2023-01-24 14:27:40.929291975 +0000 2023-01-24T14:42:07.339 INFO:teuthology.orchestra.run.smithi039.stdout:Change: 2023-01-24 14:27:40.929291975 +0000 2023-01-24T14:42:07.339 INFO:teuthology.orchestra.run.smithi039.stdout: Birth: - 2023-01-24T14:42:07.339 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-01-24T14:42:07.406 INFO:teuthology.orchestra.run.smithi039.stderr:1+0 records in 2023-01-24T14:42:07.407 INFO:teuthology.orchestra.run.smithi039.stderr:1+0 records out 2023-01-24T14:42:07.407 INFO:teuthology.orchestra.run.smithi039.stderr:512 bytes copied, 0.000296537 s, 1.7 MB/s 2023-01-24T14:42:07.408 DEBUG:teuthology.orchestra.run.smithi039:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-01-24T14:42:07.469 DEBUG:teuthology.orchestra.run.smithi039:> stat /dev/vg_nvme/lv_3 2023-01-24T14:42:07.528 INFO:teuthology.orchestra.run.smithi039.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-01-24T14:42:07.528 INFO:teuthology.orchestra.run.smithi039.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-01-24T14:42:07.528 INFO:teuthology.orchestra.run.smithi039.stdout:Device: 6h/6d Inode: 373889 Links: 1 2023-01-24T14:42:07.528 INFO:teuthology.orchestra.run.smithi039.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-01-24T14:42:07.529 INFO:teuthology.orchestra.run.smithi039.stdout:Context: system_u:object_r:device_t:s0 2023-01-24T14:42:07.529 INFO:teuthology.orchestra.run.smithi039.stdout:Access: 2023-01-24 14:40:46.878425753 +0000 2023-01-24T14:42:07.529 INFO:teuthology.orchestra.run.smithi039.stdout:Modify: 2023-01-24 14:27:41.462281263 +0000 2023-01-24T14:42:07.529 INFO:teuthology.orchestra.run.smithi039.stdout:Change: 2023-01-24 14:27:41.462281263 +0000 2023-01-24T14:42:07.529 INFO:teuthology.orchestra.run.smithi039.stdout: Birth: - 2023-01-24T14:42:07.530 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-01-24T14:42:07.595 INFO:teuthology.orchestra.run.smithi039.stderr:1+0 records in 2023-01-24T14:42:07.596 INFO:teuthology.orchestra.run.smithi039.stderr:1+0 records out 2023-01-24T14:42:07.596 INFO:teuthology.orchestra.run.smithi039.stderr:512 bytes copied, 0.000295982 s, 1.7 MB/s 2023-01-24T14:42:07.597 DEBUG:teuthology.orchestra.run.smithi039:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-01-24T14:42:07.656 DEBUG:teuthology.orchestra.run.smithi039:> stat /dev/vg_nvme/lv_4 2023-01-24T14:42:07.715 INFO:teuthology.orchestra.run.smithi039.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-01-24T14:42:07.715 INFO:teuthology.orchestra.run.smithi039.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-01-24T14:42:07.716 INFO:teuthology.orchestra.run.smithi039.stdout:Device: 6h/6d Inode: 375116 Links: 1 2023-01-24T14:42:07.716 INFO:teuthology.orchestra.run.smithi039.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-01-24T14:42:07.716 INFO:teuthology.orchestra.run.smithi039.stdout:Context: system_u:object_r:device_t:s0 2023-01-24T14:42:07.716 INFO:teuthology.orchestra.run.smithi039.stdout:Access: 2023-01-24 14:40:46.879425732 +0000 2023-01-24T14:42:07.717 INFO:teuthology.orchestra.run.smithi039.stdout:Modify: 2023-01-24 14:27:42.047269507 +0000 2023-01-24T14:42:07.717 INFO:teuthology.orchestra.run.smithi039.stdout:Change: 2023-01-24 14:27:42.047269507 +0000 2023-01-24T14:42:07.717 INFO:teuthology.orchestra.run.smithi039.stdout: Birth: - 2023-01-24T14:42:07.717 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-01-24T14:42:07.782 INFO:teuthology.orchestra.run.smithi039.stderr:1+0 records in 2023-01-24T14:42:07.783 INFO:teuthology.orchestra.run.smithi039.stderr:1+0 records out 2023-01-24T14:42:07.783 INFO:teuthology.orchestra.run.smithi039.stderr:512 bytes copied, 0.00035083 s, 1.5 MB/s 2023-01-24T14:42:07.784 DEBUG:teuthology.orchestra.run.smithi039:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-01-24T14:42:07.844 INFO:tasks.ceph:osd dev map: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.3': '/dev/vg_nvme/lv_2'} 2023-01-24T14:42:07.844 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi002.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2'}, Remote(name='ubuntu@smithi039.front.sepia.ceph.com'): {'osd.2': '/dev/vg_nvme/lv_1', 'osd.3': '/dev/vg_nvme/lv_2'}} 2023-01-24T14:42:07.844 INFO:tasks.ceph:Generating config... 2023-01-24T14:42:07.846 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2023-01-24T14:42:07.846 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2023-01-24T14:42:07.847 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2023-01-24T14:42:07.847 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2023-01-24T14:42:07.847 INFO:tasks.ceph:[mon] paxos service trim min = 10 2023-01-24T14:42:07.848 INFO:tasks.ceph:[mon] debug mon = 20 2023-01-24T14:42:07.848 INFO:tasks.ceph:[mon] debug ms = 1 2023-01-24T14:42:07.848 INFO:tasks.ceph:[mon] debug paxos = 20 2023-01-24T14:42:07.849 INFO:tasks.ceph:[osd] debug monc = 20 2023-01-24T14:42:07.849 INFO:tasks.ceph:[osd] debug ms = 1 2023-01-24T14:42:07.849 INFO:tasks.ceph:[osd] debug osd = 20 2023-01-24T14:42:07.849 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2023-01-24T14:42:07.850 INFO:tasks.ceph:[global] ms client mode = crc 2023-01-24T14:42:07.850 INFO:tasks.ceph:[global] ms cluster mode = crc 2023-01-24T14:42:07.850 INFO:tasks.ceph:[global] ms mon client mode = crc 2023-01-24T14:42:07.850 INFO:tasks.ceph:[global] ms mon cluster mode = crc 2023-01-24T14:42:07.851 INFO:tasks.ceph:[global] ms mon service mode = crc 2023-01-24T14:42:07.851 INFO:tasks.ceph:[global] ms service mode = crc 2023-01-24T14:42:07.851 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-01-24T14:42:07.851 INFO:tasks.ceph:[mgr] debug ms = 1 2023-01-24T14:42:07.852 INFO:tasks.ceph:Setting up mon.a... 2023-01-24T14:42:07.852 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2023-01-24T14:42:07.910 INFO:teuthology.orchestra.run.smithi002.stdout:creating /etc/ceph/ceph.keyring 2023-01-24T14:42:07.914 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2023-01-24T14:42:07.968 DEBUG:teuthology.orchestra.run.smithi002:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-01-24T14:42:07.997 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.2')] 2023-01-24T14:42:07.997 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.2', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}} 2023-01-24T14:42:07.998 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2023-01-24T14:42:07.999 DEBUG:teuthology.orchestra.run.smithi002:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-01-24T14:42:08.055 DEBUG:teuthology.orchestra.run.smithi002:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.2 --print /home/ubuntu/cephtest/ceph.monmap 2023-01-24T14:42:08.146 INFO:teuthology.orchestra.run.smithi002.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2023-01-24T14:42:08.147 INFO:teuthology.orchestra.run.smithi002.stdout:monmaptool: generated fsid 3808514d-c43d-4c1f-b296-499f8064f677 2023-01-24T14:42:08.147 INFO:teuthology.orchestra.run.smithi002.stdout:setting min_mon_release = pacific 2023-01-24T14:42:08.147 INFO:teuthology.orchestra.run.smithi002.stdout:epoch 0 2023-01-24T14:42:08.147 INFO:teuthology.orchestra.run.smithi002.stdout:fsid 3808514d-c43d-4c1f-b296-499f8064f677 2023-01-24T14:42:08.147 INFO:teuthology.orchestra.run.smithi002.stdout:last_changed 2023-01-24T14:42:08.146260+0000 2023-01-24T14:42:08.148 INFO:teuthology.orchestra.run.smithi002.stdout:created 2023-01-24T14:42:08.146260+0000 2023-01-24T14:42:08.148 INFO:teuthology.orchestra.run.smithi002.stdout:min_mon_release 16 (pacific) 2023-01-24T14:42:08.148 INFO:teuthology.orchestra.run.smithi002.stdout:election_strategy: 1 2023-01-24T14:42:08.148 INFO:teuthology.orchestra.run.smithi002.stdout:0: [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] mon.a 2023-01-24T14:42:08.148 INFO:teuthology.orchestra.run.smithi002.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (1 monitors) 2023-01-24T14:42:08.150 DEBUG:teuthology.orchestra.run.smithi002:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-01-24T14:42:08.169 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 3808514d-c43d-4c1f-b296-499f8064f677... 2023-01-24T14:42:08.170 DEBUG:teuthology.orchestra.run.smithi002:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-01-24T14:42:08.212 DEBUG:teuthology.orchestra.run.smithi039:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-01-24T14:42:08.256 INFO:teuthology.orchestra.run.smithi002.stdout:[global] 2023-01-24T14:42:08.257 INFO:teuthology.orchestra.run.smithi002.stdout: chdir = "" 2023-01-24T14:42:08.257 INFO:teuthology.orchestra.run.smithi002.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-01-24T14:42:08.257 INFO:teuthology.orchestra.run.smithi002.stdout: auth supported = cephx 2023-01-24T14:42:08.257 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-01-24T14:42:08.258 INFO:teuthology.orchestra.run.smithi002.stdout: filestore xattr use omap = true 2023-01-24T14:42:08.258 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-01-24T14:42:08.258 INFO:teuthology.orchestra.run.smithi002.stdout: mon clock drift allowed = 1.000 2023-01-24T14:42:08.258 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-01-24T14:42:08.259 INFO:teuthology.orchestra.run.smithi002.stdout: osd crush chooseleaf type = 0 2023-01-24T14:42:08.259 INFO:teuthology.orchestra.run.smithi002.stdout: auth debug = true 2023-01-24T14:42:08.259 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-01-24T14:42:08.259 INFO:teuthology.orchestra.run.smithi002.stdout: ms die on old message = true 2023-01-24T14:42:08.260 INFO:teuthology.orchestra.run.smithi002.stdout: ms die on bug = true 2023-01-24T14:42:08.260 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-01-24T14:42:08.260 INFO:teuthology.orchestra.run.smithi002.stdout: mon max pg per osd = 10000 # >= luminous 2023-01-24T14:42:08.260 INFO:teuthology.orchestra.run.smithi002.stdout: mon pg warn max object skew = 0 2023-01-24T14:42:08.260 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-01-24T14:42:08.261 INFO:teuthology.orchestra.run.smithi002.stdout: # disable pg_autoscaler by default for new pools 2023-01-24T14:42:08.261 INFO:teuthology.orchestra.run.smithi002.stdout: osd_pool_default_pg_autoscale_mode = off 2023-01-24T14:42:08.261 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-01-24T14:42:08.261 INFO:teuthology.orchestra.run.smithi002.stdout: osd pool default size = 2 2023-01-24T14:42:08.261 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-01-24T14:42:08.262 INFO:teuthology.orchestra.run.smithi002.stdout: mon osd allow primary affinity = true 2023-01-24T14:42:08.262 INFO:teuthology.orchestra.run.smithi002.stdout: mon osd allow pg remap = true 2023-01-24T14:42:08.262 INFO:teuthology.orchestra.run.smithi002.stdout: mon warn on legacy crush tunables = false 2023-01-24T14:42:08.262 INFO:teuthology.orchestra.run.smithi002.stdout: mon warn on crush straw calc version zero = false 2023-01-24T14:42:08.262 INFO:teuthology.orchestra.run.smithi002.stdout: mon warn on no sortbitwise = false 2023-01-24T14:42:08.263 INFO:teuthology.orchestra.run.smithi002.stdout: mon warn on osd down out interval zero = false 2023-01-24T14:42:08.263 INFO:teuthology.orchestra.run.smithi002.stdout: mon warn on too few osds = false 2023-01-24T14:42:08.263 INFO:teuthology.orchestra.run.smithi002.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-01-24T14:42:08.263 INFO:teuthology.orchestra.run.smithi002.stdout: mon_warn_on_pool_no_redundancy = false 2023-01-24T14:42:08.263 INFO:teuthology.orchestra.run.smithi002.stdout: mon_allow_pool_size_one = true 2023-01-24T14:42:08.264 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-01-24T14:42:08.264 INFO:teuthology.orchestra.run.smithi002.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-01-24T14:42:08.264 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-01-24T14:42:08.264 INFO:teuthology.orchestra.run.smithi002.stdout: osd default data pool replay window = 5 2023-01-24T14:42:08.264 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-01-24T14:42:08.265 INFO:teuthology.orchestra.run.smithi002.stdout: mon allow pool delete = true 2023-01-24T14:42:08.265 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-01-24T14:42:08.265 INFO:teuthology.orchestra.run.smithi002.stdout: mon cluster log file level = debug 2023-01-24T14:42:08.265 INFO:teuthology.orchestra.run.smithi002.stdout: debug asserts on shutdown = true 2023-01-24T14:42:08.265 INFO:teuthology.orchestra.run.smithi002.stdout: mon health detail to clog = false 2023-01-24T14:42:08.265 INFO:teuthology.orchestra.run.smithi002.stdout: mon host = 172.21.15.2 2023-01-24T14:42:08.266 INFO:teuthology.orchestra.run.smithi002.stdout: ms client mode = crc 2023-01-24T14:42:08.266 INFO:teuthology.orchestra.run.smithi002.stdout: ms cluster mode = crc 2023-01-24T14:42:08.266 INFO:teuthology.orchestra.run.smithi002.stdout: ms mon client mode = crc 2023-01-24T14:42:08.266 INFO:teuthology.orchestra.run.smithi002.stdout: ms mon cluster mode = crc 2023-01-24T14:42:08.266 INFO:teuthology.orchestra.run.smithi002.stdout: ms mon service mode = crc 2023-01-24T14:42:08.267 INFO:teuthology.orchestra.run.smithi002.stdout: ms service mode = crc 2023-01-24T14:42:08.267 INFO:teuthology.orchestra.run.smithi002.stdout: fsid = 3808514d-c43d-4c1f-b296-499f8064f677 2023-01-24T14:42:08.267 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-01-24T14:42:08.267 INFO:teuthology.orchestra.run.smithi002.stdout:[osd] 2023-01-24T14:42:08.268 INFO:teuthology.orchestra.run.smithi002.stdout: osd journal size = 100 2023-01-24T14:42:08.268 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-01-24T14:42:08.268 INFO:teuthology.orchestra.run.smithi002.stdout: osd scrub load threshold = 5.0 2023-01-24T14:42:08.268 INFO:teuthology.orchestra.run.smithi002.stdout: osd scrub max interval = 600 2023-01-24T14:42:08.268 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-01-24T14:42:08.269 INFO:teuthology.orchestra.run.smithi002.stdout: osd recover clone overlap = true 2023-01-24T14:42:08.269 INFO:teuthology.orchestra.run.smithi002.stdout: osd recovery max chunk = 1048576 2023-01-24T14:42:08.269 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-01-24T14:42:08.269 INFO:teuthology.orchestra.run.smithi002.stdout: osd debug shutdown = true 2023-01-24T14:42:08.269 INFO:teuthology.orchestra.run.smithi002.stdout: osd debug op order = true 2023-01-24T14:42:08.269 INFO:teuthology.orchestra.run.smithi002.stdout: osd debug verify stray on activate = true 2023-01-24T14:42:08.270 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-01-24T14:42:08.270 INFO:teuthology.orchestra.run.smithi002.stdout: osd open classes on start = true 2023-01-24T14:42:08.270 INFO:teuthology.orchestra.run.smithi002.stdout: osd debug pg log writeout = true 2023-01-24T14:42:08.270 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-01-24T14:42:08.270 INFO:teuthology.orchestra.run.smithi002.stdout: osd deep scrub update digest min age = 30 2023-01-24T14:42:08.271 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-01-24T14:42:08.271 INFO:teuthology.orchestra.run.smithi002.stdout: osd map max advance = 10 2023-01-24T14:42:08.271 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-01-24T14:42:08.271 INFO:teuthology.orchestra.run.smithi002.stdout: journal zero on create = true 2023-01-24T14:42:08.271 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-01-24T14:42:08.272 INFO:teuthology.orchestra.run.smithi002.stdout: filestore ondisk finisher threads = 3 2023-01-24T14:42:08.272 INFO:teuthology.orchestra.run.smithi002.stdout: filestore apply finisher threads = 3 2023-01-24T14:42:08.272 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-01-24T14:42:08.272 INFO:teuthology.orchestra.run.smithi002.stdout: bdev debug aio = true 2023-01-24T14:42:08.272 INFO:teuthology.orchestra.run.smithi002.stdout: osd debug misdirected ops = true 2023-01-24T14:42:08.273 INFO:teuthology.orchestra.run.smithi002.stdout: debug monc = 20 2023-01-24T14:42:08.273 INFO:teuthology.orchestra.run.smithi002.stdout: debug ms = 1 2023-01-24T14:42:08.273 INFO:teuthology.orchestra.run.smithi002.stdout: debug osd = 20 2023-01-24T14:42:08.273 INFO:teuthology.orchestra.run.smithi002.stdout: osd shutdown pgref assert = True 2023-01-24T14:42:08.273 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-01-24T14:42:08.273 INFO:teuthology.orchestra.run.smithi002.stdout:[mgr] 2023-01-24T14:42:08.274 INFO:teuthology.orchestra.run.smithi002.stdout: debug ms = 1 2023-01-24T14:42:08.274 INFO:teuthology.orchestra.run.smithi002.stdout: debug mgr = 20 2023-01-24T14:42:08.274 INFO:teuthology.orchestra.run.smithi002.stdout: debug mon = 20 2023-01-24T14:42:08.274 INFO:teuthology.orchestra.run.smithi002.stdout: debug auth = 20 2023-01-24T14:42:08.274 INFO:teuthology.orchestra.run.smithi002.stdout: mon reweight min pgs per osd = 4 2023-01-24T14:42:08.275 INFO:teuthology.orchestra.run.smithi002.stdout: mon reweight min bytes per osd = 10 2023-01-24T14:42:08.275 INFO:teuthology.orchestra.run.smithi002.stdout: mgr/telemetry/nag = false 2023-01-24T14:42:08.275 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-01-24T14:42:08.275 INFO:teuthology.orchestra.run.smithi002.stdout:[mon] 2023-01-24T14:42:08.275 INFO:teuthology.orchestra.run.smithi002.stdout: debug ms = 1 2023-01-24T14:42:08.276 INFO:teuthology.orchestra.run.smithi002.stdout: debug mon = 20 2023-01-24T14:42:08.276 INFO:teuthology.orchestra.run.smithi002.stdout: debug paxos = 20 2023-01-24T14:42:08.276 INFO:teuthology.orchestra.run.smithi002.stdout: debug auth = 20 2023-01-24T14:42:08.276 INFO:teuthology.orchestra.run.smithi002.stdout: mon data avail warn = 5 2023-01-24T14:42:08.276 INFO:teuthology.orchestra.run.smithi002.stdout: mon mgr mkfs grace = 240 2023-01-24T14:42:08.277 INFO:teuthology.orchestra.run.smithi002.stdout: mon reweight min pgs per osd = 4 2023-01-24T14:42:08.277 INFO:teuthology.orchestra.run.smithi002.stdout: mon osd reporter subtree level = osd 2023-01-24T14:42:08.277 INFO:teuthology.orchestra.run.smithi002.stdout: mon osd prime pg temp = true 2023-01-24T14:42:08.277 INFO:teuthology.orchestra.run.smithi002.stdout: mon reweight min bytes per osd = 10 2023-01-24T14:42:08.277 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-01-24T14:42:08.278 INFO:teuthology.orchestra.run.smithi002.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-01-24T14:42:08.278 INFO:teuthology.orchestra.run.smithi002.stdout: auth mon ticket ttl = 660 # 11m 2023-01-24T14:42:08.278 INFO:teuthology.orchestra.run.smithi002.stdout: auth service ticket ttl = 240 # 4m 2023-01-24T14:42:08.278 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-01-24T14:42:08.278 INFO:teuthology.orchestra.run.smithi002.stdout: # don't complain about insecure global_id in the test suite 2023-01-24T14:42:08.279 INFO:teuthology.orchestra.run.smithi002.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-01-24T14:42:08.279 INFO:teuthology.orchestra.run.smithi002.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-01-24T14:42:08.279 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-01-24T14:42:08.279 INFO:teuthology.orchestra.run.smithi002.stdout: # 1m isn't quite enough 2023-01-24T14:42:08.279 INFO:teuthology.orchestra.run.smithi002.stdout: mon_down_mkfs_grace = 2m 2023-01-24T14:42:08.280 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-01-24T14:42:08.280 INFO:teuthology.orchestra.run.smithi002.stdout: mon_warn_on_filestore_osds = false 2023-01-24T14:42:08.280 INFO:teuthology.orchestra.run.smithi002.stdout: mon min osdmap epochs = 50 2023-01-24T14:42:08.280 INFO:teuthology.orchestra.run.smithi002.stdout: mon osdmap full prune interval = 2 2023-01-24T14:42:08.280 INFO:teuthology.orchestra.run.smithi002.stdout: mon osdmap full prune min = 15 2023-01-24T14:42:08.280 INFO:teuthology.orchestra.run.smithi002.stdout: mon osdmap full prune txsize = 2 2023-01-24T14:42:08.281 INFO:teuthology.orchestra.run.smithi002.stdout: paxos service trim min = 10 2023-01-24T14:42:08.281 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-01-24T14:42:08.281 INFO:teuthology.orchestra.run.smithi002.stdout:[client] 2023-01-24T14:42:08.281 INFO:teuthology.orchestra.run.smithi002.stdout: rgw cache enabled = true 2023-01-24T14:42:08.281 INFO:teuthology.orchestra.run.smithi002.stdout: rgw enable ops log = true 2023-01-24T14:42:08.282 INFO:teuthology.orchestra.run.smithi002.stdout: rgw enable usage log = true 2023-01-24T14:42:08.282 INFO:teuthology.orchestra.run.smithi002.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-01-24T14:42:08.282 INFO:teuthology.orchestra.run.smithi002.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-01-24T14:42:08.282 INFO:teuthology.orchestra.run.smithi002.stdout:[mon.a] 2023-01-24T14:42:08.284 INFO:teuthology.orchestra.run.smithi039.stdout:[global] 2023-01-24T14:42:08.284 INFO:teuthology.orchestra.run.smithi039.stdout: chdir = "" 2023-01-24T14:42:08.284 INFO:teuthology.orchestra.run.smithi039.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-01-24T14:42:08.285 INFO:teuthology.orchestra.run.smithi039.stdout: auth supported = cephx 2023-01-24T14:42:08.285 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-01-24T14:42:08.285 INFO:teuthology.orchestra.run.smithi039.stdout: filestore xattr use omap = true 2023-01-24T14:42:08.285 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-01-24T14:42:08.285 INFO:teuthology.orchestra.run.smithi039.stdout: mon clock drift allowed = 1.000 2023-01-24T14:42:08.286 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-01-24T14:42:08.286 INFO:teuthology.orchestra.run.smithi039.stdout: osd crush chooseleaf type = 0 2023-01-24T14:42:08.286 INFO:teuthology.orchestra.run.smithi039.stdout: auth debug = true 2023-01-24T14:42:08.286 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-01-24T14:42:08.286 INFO:teuthology.orchestra.run.smithi039.stdout: ms die on old message = true 2023-01-24T14:42:08.287 INFO:teuthology.orchestra.run.smithi039.stdout: ms die on bug = true 2023-01-24T14:42:08.287 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-01-24T14:42:08.287 INFO:teuthology.orchestra.run.smithi039.stdout: mon max pg per osd = 10000 # >= luminous 2023-01-24T14:42:08.287 INFO:teuthology.orchestra.run.smithi039.stdout: mon pg warn max object skew = 0 2023-01-24T14:42:08.287 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-01-24T14:42:08.288 INFO:teuthology.orchestra.run.smithi039.stdout: # disable pg_autoscaler by default for new pools 2023-01-24T14:42:08.288 INFO:teuthology.orchestra.run.smithi039.stdout: osd_pool_default_pg_autoscale_mode = off 2023-01-24T14:42:08.288 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-01-24T14:42:08.288 INFO:teuthology.orchestra.run.smithi039.stdout: osd pool default size = 2 2023-01-24T14:42:08.288 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-01-24T14:42:08.289 INFO:teuthology.orchestra.run.smithi039.stdout: mon osd allow primary affinity = true 2023-01-24T14:42:08.289 INFO:teuthology.orchestra.run.smithi039.stdout: mon osd allow pg remap = true 2023-01-24T14:42:08.289 INFO:teuthology.orchestra.run.smithi039.stdout: mon warn on legacy crush tunables = false 2023-01-24T14:42:08.289 INFO:teuthology.orchestra.run.smithi039.stdout: mon warn on crush straw calc version zero = false 2023-01-24T14:42:08.289 INFO:teuthology.orchestra.run.smithi039.stdout: mon warn on no sortbitwise = false 2023-01-24T14:42:08.290 INFO:teuthology.orchestra.run.smithi039.stdout: mon warn on osd down out interval zero = false 2023-01-24T14:42:08.290 INFO:teuthology.orchestra.run.smithi039.stdout: mon warn on too few osds = false 2023-01-24T14:42:08.290 INFO:teuthology.orchestra.run.smithi039.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-01-24T14:42:08.290 INFO:teuthology.orchestra.run.smithi039.stdout: mon_warn_on_pool_no_redundancy = false 2023-01-24T14:42:08.290 INFO:teuthology.orchestra.run.smithi039.stdout: mon_allow_pool_size_one = true 2023-01-24T14:42:08.290 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-01-24T14:42:08.291 INFO:teuthology.orchestra.run.smithi039.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-01-24T14:42:08.291 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-01-24T14:42:08.291 INFO:teuthology.orchestra.run.smithi039.stdout: osd default data pool replay window = 5 2023-01-24T14:42:08.291 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-01-24T14:42:08.291 INFO:teuthology.orchestra.run.smithi039.stdout: mon allow pool delete = true 2023-01-24T14:42:08.292 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-01-24T14:42:08.292 INFO:teuthology.orchestra.run.smithi039.stdout: mon cluster log file level = debug 2023-01-24T14:42:08.292 INFO:teuthology.orchestra.run.smithi039.stdout: debug asserts on shutdown = true 2023-01-24T14:42:08.292 INFO:teuthology.orchestra.run.smithi039.stdout: mon health detail to clog = false 2023-01-24T14:42:08.292 INFO:teuthology.orchestra.run.smithi039.stdout: mon host = 172.21.15.2 2023-01-24T14:42:08.293 INFO:teuthology.orchestra.run.smithi039.stdout: ms client mode = crc 2023-01-24T14:42:08.293 INFO:teuthology.orchestra.run.smithi039.stdout: ms cluster mode = crc 2023-01-24T14:42:08.293 INFO:teuthology.orchestra.run.smithi039.stdout: ms mon client mode = crc 2023-01-24T14:42:08.293 INFO:teuthology.orchestra.run.smithi039.stdout: ms mon cluster mode = crc 2023-01-24T14:42:08.293 INFO:teuthology.orchestra.run.smithi039.stdout: ms mon service mode = crc 2023-01-24T14:42:08.294 INFO:teuthology.orchestra.run.smithi039.stdout: ms service mode = crc 2023-01-24T14:42:08.294 INFO:teuthology.orchestra.run.smithi039.stdout: fsid = 3808514d-c43d-4c1f-b296-499f8064f677 2023-01-24T14:42:08.294 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-01-24T14:42:08.294 INFO:teuthology.orchestra.run.smithi039.stdout:[osd] 2023-01-24T14:42:08.294 INFO:teuthology.orchestra.run.smithi039.stdout: osd journal size = 100 2023-01-24T14:42:08.295 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-01-24T14:42:08.295 INFO:teuthology.orchestra.run.smithi039.stdout: osd scrub load threshold = 5.0 2023-01-24T14:42:08.295 INFO:teuthology.orchestra.run.smithi039.stdout: osd scrub max interval = 600 2023-01-24T14:42:08.295 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-01-24T14:42:08.295 INFO:teuthology.orchestra.run.smithi039.stdout: osd recover clone overlap = true 2023-01-24T14:42:08.296 INFO:teuthology.orchestra.run.smithi039.stdout: osd recovery max chunk = 1048576 2023-01-24T14:42:08.296 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-01-24T14:42:08.296 INFO:teuthology.orchestra.run.smithi039.stdout: osd debug shutdown = true 2023-01-24T14:42:08.296 INFO:teuthology.orchestra.run.smithi039.stdout: osd debug op order = true 2023-01-24T14:42:08.296 INFO:teuthology.orchestra.run.smithi039.stdout: osd debug verify stray on activate = true 2023-01-24T14:42:08.297 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-01-24T14:42:08.297 INFO:teuthology.orchestra.run.smithi039.stdout: osd open classes on start = true 2023-01-24T14:42:08.297 INFO:teuthology.orchestra.run.smithi039.stdout: osd debug pg log writeout = true 2023-01-24T14:42:08.297 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-01-24T14:42:08.297 INFO:teuthology.orchestra.run.smithi039.stdout: osd deep scrub update digest min age = 30 2023-01-24T14:42:08.298 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-01-24T14:42:08.298 INFO:teuthology.orchestra.run.smithi039.stdout: osd map max advance = 10 2023-01-24T14:42:08.298 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-01-24T14:42:08.298 INFO:teuthology.orchestra.run.smithi039.stdout: journal zero on create = true 2023-01-24T14:42:08.298 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-01-24T14:42:08.299 INFO:teuthology.orchestra.run.smithi039.stdout: filestore ondisk finisher threads = 3 2023-01-24T14:42:08.299 INFO:teuthology.orchestra.run.smithi039.stdout: filestore apply finisher threads = 3 2023-01-24T14:42:08.299 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-01-24T14:42:08.299 INFO:teuthology.orchestra.run.smithi039.stdout: bdev debug aio = true 2023-01-24T14:42:08.299 INFO:teuthology.orchestra.run.smithi039.stdout: osd debug misdirected ops = true 2023-01-24T14:42:08.300 INFO:teuthology.orchestra.run.smithi039.stdout: debug monc = 20 2023-01-24T14:42:08.300 INFO:teuthology.orchestra.run.smithi039.stdout: debug ms = 1 2023-01-24T14:42:08.300 INFO:teuthology.orchestra.run.smithi039.stdout: debug osd = 20 2023-01-24T14:42:08.300 INFO:teuthology.orchestra.run.smithi039.stdout: osd shutdown pgref assert = True 2023-01-24T14:42:08.300 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-01-24T14:42:08.301 INFO:teuthology.orchestra.run.smithi039.stdout:[mgr] 2023-01-24T14:42:08.301 INFO:teuthology.orchestra.run.smithi039.stdout: debug ms = 1 2023-01-24T14:42:08.301 INFO:teuthology.orchestra.run.smithi039.stdout: debug mgr = 20 2023-01-24T14:42:08.301 INFO:teuthology.orchestra.run.smithi039.stdout: debug mon = 20 2023-01-24T14:42:08.301 INFO:teuthology.orchestra.run.smithi039.stdout: debug auth = 20 2023-01-24T14:42:08.302 INFO:teuthology.orchestra.run.smithi039.stdout: mon reweight min pgs per osd = 4 2023-01-24T14:42:08.302 INFO:teuthology.orchestra.run.smithi039.stdout: mon reweight min bytes per osd = 10 2023-01-24T14:42:08.302 INFO:teuthology.orchestra.run.smithi039.stdout: mgr/telemetry/nag = false 2023-01-24T14:42:08.302 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-01-24T14:42:08.302 INFO:teuthology.orchestra.run.smithi039.stdout:[mon] 2023-01-24T14:42:08.302 INFO:teuthology.orchestra.run.smithi039.stdout: debug ms = 1 2023-01-24T14:42:08.303 INFO:teuthology.orchestra.run.smithi039.stdout: debug mon = 20 2023-01-24T14:42:08.303 INFO:teuthology.orchestra.run.smithi039.stdout: debug paxos = 20 2023-01-24T14:42:08.303 INFO:teuthology.orchestra.run.smithi039.stdout: debug auth = 20 2023-01-24T14:42:08.303 INFO:teuthology.orchestra.run.smithi039.stdout: mon data avail warn = 5 2023-01-24T14:42:08.303 INFO:teuthology.orchestra.run.smithi039.stdout: mon mgr mkfs grace = 240 2023-01-24T14:42:08.304 INFO:teuthology.orchestra.run.smithi039.stdout: mon reweight min pgs per osd = 4 2023-01-24T14:42:08.304 INFO:teuthology.orchestra.run.smithi039.stdout: mon osd reporter subtree level = osd 2023-01-24T14:42:08.304 INFO:teuthology.orchestra.run.smithi039.stdout: mon osd prime pg temp = true 2023-01-24T14:42:08.304 INFO:teuthology.orchestra.run.smithi039.stdout: mon reweight min bytes per osd = 10 2023-01-24T14:42:08.304 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-01-24T14:42:08.305 INFO:teuthology.orchestra.run.smithi039.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-01-24T14:42:08.305 INFO:teuthology.orchestra.run.smithi039.stdout: auth mon ticket ttl = 660 # 11m 2023-01-24T14:42:08.305 INFO:teuthology.orchestra.run.smithi039.stdout: auth service ticket ttl = 240 # 4m 2023-01-24T14:42:08.305 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-01-24T14:42:08.305 INFO:teuthology.orchestra.run.smithi039.stdout: # don't complain about insecure global_id in the test suite 2023-01-24T14:42:08.306 INFO:teuthology.orchestra.run.smithi039.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-01-24T14:42:08.306 INFO:teuthology.orchestra.run.smithi039.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-01-24T14:42:08.306 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-01-24T14:42:08.306 INFO:teuthology.orchestra.run.smithi039.stdout: # 1m isn't quite enough 2023-01-24T14:42:08.306 INFO:teuthology.orchestra.run.smithi039.stdout: mon_down_mkfs_grace = 2m 2023-01-24T14:42:08.306 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-01-24T14:42:08.307 INFO:teuthology.orchestra.run.smithi039.stdout: mon_warn_on_filestore_osds = false 2023-01-24T14:42:08.307 INFO:teuthology.orchestra.run.smithi039.stdout: mon min osdmap epochs = 50 2023-01-24T14:42:08.307 INFO:teuthology.orchestra.run.smithi039.stdout: mon osdmap full prune interval = 2 2023-01-24T14:42:08.307 INFO:teuthology.orchestra.run.smithi039.stdout: mon osdmap full prune min = 15 2023-01-24T14:42:08.307 INFO:teuthology.orchestra.run.smithi039.stdout: mon osdmap full prune txsize = 2 2023-01-24T14:42:08.308 INFO:teuthology.orchestra.run.smithi039.stdout: paxos service trim min = 10 2023-01-24T14:42:08.308 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-01-24T14:42:08.308 INFO:teuthology.orchestra.run.smithi039.stdout:[client] 2023-01-24T14:42:08.308 INFO:teuthology.orchestra.run.smithi039.stdout: rgw cache enabled = true 2023-01-24T14:42:08.308 INFO:teuthology.orchestra.run.smithi039.stdout: rgw enable ops log = true 2023-01-24T14:42:08.309 INFO:teuthology.orchestra.run.smithi039.stdout: rgw enable usage log = true 2023-01-24T14:42:08.309 INFO:teuthology.orchestra.run.smithi039.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-01-24T14:42:08.309 INFO:teuthology.orchestra.run.smithi039.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-01-24T14:42:08.309 INFO:teuthology.orchestra.run.smithi039.stdout:[mon.a] 2023-01-24T14:42:08.310 INFO:tasks.ceph:Creating admin key on mon.a... 2023-01-24T14:42:08.310 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2023-01-24T14:42:08.390 INFO:tasks.ceph:Copying monmap to all nodes... 2023-01-24T14:42:08.390 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2023-01-24T14:42:08.391 DEBUG:teuthology.orchestra.run.smithi002:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2023-01-24T14:42:08.409 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2023-01-24T14:42:08.409 DEBUG:teuthology.orchestra.run.smithi002:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2023-01-24T14:42:08.467 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi002.front.sepia.ceph.com 2023-01-24T14:42:08.468 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2023-01-24T14:42:08.468 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd of=/etc/ceph/ceph.keyring 2023-01-24T14:42:08.469 DEBUG:teuthology.orchestra.run.smithi002:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-01-24T14:42:08.545 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2023-01-24T14:42:08.546 DEBUG:teuthology.orchestra.run.smithi002:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-01-24T14:42:08.604 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi039.front.sepia.ceph.com 2023-01-24T14:42:08.604 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2023-01-24T14:42:08.605 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd of=/etc/ceph/ceph.keyring 2023-01-24T14:42:08.605 DEBUG:teuthology.orchestra.run.smithi039:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-01-24T14:42:08.647 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2023-01-24T14:42:08.647 DEBUG:teuthology.orchestra.run.smithi039:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-01-24T14:42:08.707 INFO:tasks.ceph:Setting up mon nodes... 2023-01-24T14:42:08.707 INFO:tasks.ceph:Setting up mgr nodes... 2023-01-24T14:42:08.708 DEBUG:teuthology.orchestra.run.smithi039:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2023-01-24T14:42:08.825 INFO:teuthology.orchestra.run.smithi039.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2023-01-24T14:42:08.828 INFO:tasks.ceph:Setting up mds nodes... 2023-01-24T14:42:08.829 INFO:tasks.ceph_client:Setting up client nodes... 2023-01-24T14:42:08.829 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-01-24T14:42:08.881 INFO:teuthology.orchestra.run.smithi002.stdout:creating /etc/ceph/ceph.client.0.keyring 2023-01-24T14:42:08.895 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2023-01-24T14:42:08.948 INFO:teuthology.orchestra.run.smithi039.stdout:creating /etc/ceph/ceph.client.1.keyring 2023-01-24T14:42:08.961 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-01-24T14:42:08.961 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi002.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2'}, Remote(name='ubuntu@smithi039.front.sepia.ceph.com'): {'osd.2': '/dev/vg_nvme/lv_1', 'osd.3': '/dev/vg_nvme/lv_2'}} 2023-01-24T14:42:08.962 DEBUG:teuthology.orchestra.run.smithi002:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2023-01-24T14:42:08.990 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2'} 2023-01-24T14:42:08.990 INFO:tasks.ceph:role: osd.0 2023-01-24T14:42:08.991 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi002.front.sepia.ceph.com 2023-01-24T14:42:08.991 DEBUG:teuthology.orchestra.run.smithi002:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-01-24T14:42:09.058 INFO:teuthology.orchestra.run.smithi002.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-01-24T14:42:09.058 INFO:teuthology.orchestra.run.smithi002.stdout: = sectsz=512 attr=2, projid32bit=1 2023-01-24T14:42:09.059 INFO:teuthology.orchestra.run.smithi002.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-01-24T14:42:09.059 INFO:teuthology.orchestra.run.smithi002.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-01-24T14:42:09.059 INFO:teuthology.orchestra.run.smithi002.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-01-24T14:42:09.059 INFO:teuthology.orchestra.run.smithi002.stdout: = sunit=0 swidth=0 blks 2023-01-24T14:42:09.060 INFO:teuthology.orchestra.run.smithi002.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-01-24T14:42:09.060 INFO:teuthology.orchestra.run.smithi002.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-01-24T14:42:09.060 INFO:teuthology.orchestra.run.smithi002.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-01-24T14:42:09.060 INFO:teuthology.orchestra.run.smithi002.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-01-24T14:42:09.082 INFO:teuthology.orchestra.run.smithi002.stdout:Discarding blocks...Done. 2023-01-24T14:42:09.083 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi002.front.sepia.ceph.com -o noatime 2023-01-24T14:42:09.083 DEBUG:teuthology.orchestra.run.smithi002:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2023-01-24T14:42:09.166 DEBUG:teuthology.orchestra.run.smithi002:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2023-01-24T14:42:09.239 DEBUG:teuthology.orchestra.run.smithi002:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2023-01-24T14:42:09.308 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2'} 2023-01-24T14:42:09.308 INFO:tasks.ceph:role: osd.1 2023-01-24T14:42:09.309 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi002.front.sepia.ceph.com 2023-01-24T14:42:09.309 DEBUG:teuthology.orchestra.run.smithi002:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-01-24T14:42:09.377 INFO:teuthology.orchestra.run.smithi002.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-01-24T14:42:09.377 INFO:teuthology.orchestra.run.smithi002.stdout: = sectsz=512 attr=2, projid32bit=1 2023-01-24T14:42:09.377 INFO:teuthology.orchestra.run.smithi002.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-01-24T14:42:09.378 INFO:teuthology.orchestra.run.smithi002.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-01-24T14:42:09.379 INFO:teuthology.orchestra.run.smithi002.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-01-24T14:42:09.379 INFO:teuthology.orchestra.run.smithi002.stdout: = sunit=0 swidth=0 blks 2023-01-24T14:42:09.379 INFO:teuthology.orchestra.run.smithi002.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-01-24T14:42:09.380 INFO:teuthology.orchestra.run.smithi002.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-01-24T14:42:09.380 INFO:teuthology.orchestra.run.smithi002.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-01-24T14:42:09.380 INFO:teuthology.orchestra.run.smithi002.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-01-24T14:42:09.400 INFO:teuthology.orchestra.run.smithi002.stdout:Discarding blocks...Done. 2023-01-24T14:42:09.402 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi002.front.sepia.ceph.com -o noatime 2023-01-24T14:42:09.403 DEBUG:teuthology.orchestra.run.smithi002:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2023-01-24T14:42:09.484 DEBUG:teuthology.orchestra.run.smithi002:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2023-01-24T14:42:09.556 DEBUG:teuthology.orchestra.run.smithi002:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-01-24T14:42:10.409 INFO:teuthology.orchestra.run.smithi002.stderr:WARNING: debug mode. Not for benchmarking or production 2023-01-24T14:42:10.410 INFO:teuthology.orchestra.run.smithi002.stderr:INFO 2023-01-24 14:42:10,410 seastar - Reactor backend: epoll 2023-01-24T14:42:10.418 INFO:teuthology.orchestra.run.smithi002.stderr:INFO 2023-01-24 14:42:10,417 [shard 0] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-01-24T14:42:10.418 INFO:teuthology.orchestra.run.smithi002.stderr:INFO 2023-01-24 14:42:10,417 [shard 0] seastar - IO queue uses 0.75ms latency goal for device 0 2023-01-24T14:42:10.419 INFO:teuthology.orchestra.run.smithi002.stderr:INFO 2023-01-24 14:42:10,418 [shard 0] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-01-24T14:42:10.419 INFO:teuthology.orchestra.run.smithi002.stderr:INFO 2023-01-24 14:42:10,418 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-01-24T14:42:10.420 INFO:teuthology.orchestra.run.smithi002.stderr:==100344==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-01-24T14:42:10.427 INFO:teuthology.orchestra.run.smithi002.stderr:==100344==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7ffe60f0c000; bottom 0x7f4b98486000; size: 0x00b2c8a86000 (767870656512) 2023-01-24T14:42:10.427 INFO:teuthology.orchestra.run.smithi002.stderr:False positive error reports may follow 2023-01-24T14:42:10.427 INFO:teuthology.orchestra.run.smithi002.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-01-24T14:42:10.464 INFO:teuthology.orchestra.run.smithi002.stderr:INFO 2023-01-24 14:42:10,463 [shard 0] osd - bypassing the config fetch due to --no-mon-config 2023-01-24T14:42:10.464 INFO:teuthology.orchestra.run.smithi002.stderr:INFO 2023-01-24 14:42:10,463 [shard 0] osd - get_early_config: set --smp 1 2023-01-24T14:42:10.970 INFO:teuthology.orchestra.run.smithi002.stderr:WARNING: debug mode. Not for benchmarking or production 2023-01-24T14:42:10.970 INFO:teuthology.orchestra.run.smithi002.stderr:INFO 2023-01-24 14:42:10,970 seastar - Reactor backend: epoll 2023-01-24T14:42:10.973 INFO:teuthology.orchestra.run.smithi002.stderr:INFO 2023-01-24 14:42:10,972 [shard 0] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-01-24T14:42:10.973 INFO:teuthology.orchestra.run.smithi002.stderr:INFO 2023-01-24 14:42:10,972 [shard 0] seastar - IO queue uses 0.75ms latency goal for device 0 2023-01-24T14:42:10.974 INFO:teuthology.orchestra.run.smithi002.stderr:INFO 2023-01-24 14:42:10,972 [shard 0] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-01-24T14:42:10.974 INFO:teuthology.orchestra.run.smithi002.stderr:INFO 2023-01-24 14:42:10,972 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-01-24T14:42:10.975 INFO:teuthology.orchestra.run.smithi002.stderr:==100342==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-01-24T14:42:10.980 INFO:teuthology.orchestra.run.smithi002.stderr:==100342==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7ffe60f0c000; bottom 0x7f4b9847f000; size: 0x00b2c8a8d000 (767870685184) 2023-01-24T14:42:10.980 INFO:teuthology.orchestra.run.smithi002.stderr:False positive error reports may follow 2023-01-24T14:42:10.981 INFO:teuthology.orchestra.run.smithi002.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-01-24T14:42:13.929 INFO:teuthology.orchestra.run.smithi002.stdout:created object store /var/lib/ceph/osd/ceph-0 for osd.0 fsid 3808514d-c43d-4c1f-b296-499f8064f677 2023-01-24T14:42:13.993 DEBUG:teuthology.orchestra.run.smithi002:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2023-01-24T14:42:14.022 DEBUG:teuthology.orchestra.run.smithi002:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-01-24T14:42:14.865 INFO:teuthology.orchestra.run.smithi002.stderr:WARNING: debug mode. Not for benchmarking or production 2023-01-24T14:42:14.866 INFO:teuthology.orchestra.run.smithi002.stderr:INFO 2023-01-24 14:42:14,866 seastar - Reactor backend: epoll 2023-01-24T14:42:14.874 INFO:teuthology.orchestra.run.smithi002.stderr:INFO 2023-01-24 14:42:14,873 [shard 0] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-01-24T14:42:14.874 INFO:teuthology.orchestra.run.smithi002.stderr:INFO 2023-01-24 14:42:14,873 [shard 0] seastar - IO queue uses 0.75ms latency goal for device 0 2023-01-24T14:42:14.875 INFO:teuthology.orchestra.run.smithi002.stderr:INFO 2023-01-24 14:42:14,873 [shard 0] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-01-24T14:42:14.875 INFO:teuthology.orchestra.run.smithi002.stderr:INFO 2023-01-24 14:42:14,873 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-01-24T14:42:14.876 INFO:teuthology.orchestra.run.smithi002.stderr:==101162==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-01-24T14:42:14.883 INFO:teuthology.orchestra.run.smithi002.stderr:==101162==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7ffcfb11d000; bottom 0x7f914b398000; size: 0x006bafd85000 (462511689728) 2023-01-24T14:42:14.883 INFO:teuthology.orchestra.run.smithi002.stderr:False positive error reports may follow 2023-01-24T14:42:14.883 INFO:teuthology.orchestra.run.smithi002.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-01-24T14:42:14.919 INFO:teuthology.orchestra.run.smithi002.stderr:INFO 2023-01-24 14:42:14,919 [shard 0] osd - bypassing the config fetch due to --no-mon-config 2023-01-24T14:42:14.920 INFO:teuthology.orchestra.run.smithi002.stderr:INFO 2023-01-24 14:42:14,919 [shard 0] osd - get_early_config: set --smp 1 2023-01-24T14:42:15.427 INFO:teuthology.orchestra.run.smithi002.stderr:WARNING: debug mode. Not for benchmarking or production 2023-01-24T14:42:15.428 INFO:teuthology.orchestra.run.smithi002.stderr:INFO 2023-01-24 14:42:15,427 seastar - Reactor backend: epoll 2023-01-24T14:42:15.429 INFO:teuthology.orchestra.run.smithi002.stderr:INFO 2023-01-24 14:42:15,429 [shard 0] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-01-24T14:42:15.429 INFO:teuthology.orchestra.run.smithi002.stderr:INFO 2023-01-24 14:42:15,429 [shard 0] seastar - IO queue uses 0.75ms latency goal for device 0 2023-01-24T14:42:15.430 INFO:teuthology.orchestra.run.smithi002.stderr:INFO 2023-01-24 14:42:15,429 [shard 0] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-01-24T14:42:15.430 INFO:teuthology.orchestra.run.smithi002.stderr:INFO 2023-01-24 14:42:15,429 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-01-24T14:42:15.431 INFO:teuthology.orchestra.run.smithi002.stderr:==101160==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-01-24T14:42:15.437 INFO:teuthology.orchestra.run.smithi002.stderr:==101160==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7ffcfb11d000; bottom 0x7f914b391000; size: 0x006bafd8c000 (462511718400) 2023-01-24T14:42:15.437 INFO:teuthology.orchestra.run.smithi002.stderr:False positive error reports may follow 2023-01-24T14:42:15.438 INFO:teuthology.orchestra.run.smithi002.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-01-24T14:42:18.386 INFO:teuthology.orchestra.run.smithi002.stdout:created object store /var/lib/ceph/osd/ceph-1 for osd.1 fsid 3808514d-c43d-4c1f-b296-499f8064f677 2023-01-24T14:42:18.476 DEBUG:teuthology.orchestra.run.smithi002:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2023-01-24T14:42:18.504 DEBUG:teuthology.orchestra.run.smithi039:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2023-01-24T14:42:18.539 INFO:tasks.ceph:roles_to_devs: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.3': '/dev/vg_nvme/lv_2'} 2023-01-24T14:42:18.539 INFO:tasks.ceph:role: osd.2 2023-01-24T14:42:18.539 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi039.front.sepia.ceph.com 2023-01-24T14:42:18.540 DEBUG:teuthology.orchestra.run.smithi039:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-01-24T14:42:18.639 INFO:teuthology.orchestra.run.smithi039.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-01-24T14:42:18.639 INFO:teuthology.orchestra.run.smithi039.stdout: = sectsz=512 attr=2, projid32bit=1 2023-01-24T14:42:18.639 INFO:teuthology.orchestra.run.smithi039.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-01-24T14:42:18.640 INFO:teuthology.orchestra.run.smithi039.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-01-24T14:42:18.640 INFO:teuthology.orchestra.run.smithi039.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-01-24T14:42:18.640 INFO:teuthology.orchestra.run.smithi039.stdout: = sunit=0 swidth=0 blks 2023-01-24T14:42:18.640 INFO:teuthology.orchestra.run.smithi039.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-01-24T14:42:18.640 INFO:teuthology.orchestra.run.smithi039.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-01-24T14:42:18.641 INFO:teuthology.orchestra.run.smithi039.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-01-24T14:42:18.641 INFO:teuthology.orchestra.run.smithi039.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-01-24T14:42:19.915 INFO:teuthology.orchestra.run.smithi039.stdout:Discarding blocks...Done. 2023-01-24T14:42:19.917 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi039.front.sepia.ceph.com -o noatime 2023-01-24T14:42:19.918 DEBUG:teuthology.orchestra.run.smithi039:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-2 2023-01-24T14:42:19.971 DEBUG:teuthology.orchestra.run.smithi039:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2023-01-24T14:42:20.007 DEBUG:teuthology.orchestra.run.smithi039:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2023-01-24T14:42:20.077 INFO:tasks.ceph:roles_to_devs: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.3': '/dev/vg_nvme/lv_2'} 2023-01-24T14:42:20.078 INFO:tasks.ceph:role: osd.3 2023-01-24T14:42:20.078 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi039.front.sepia.ceph.com 2023-01-24T14:42:20.078 DEBUG:teuthology.orchestra.run.smithi039:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-01-24T14:42:20.175 INFO:teuthology.orchestra.run.smithi039.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-01-24T14:42:20.176 INFO:teuthology.orchestra.run.smithi039.stdout: = sectsz=512 attr=2, projid32bit=1 2023-01-24T14:42:20.176 INFO:teuthology.orchestra.run.smithi039.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-01-24T14:42:20.177 INFO:teuthology.orchestra.run.smithi039.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-01-24T14:42:20.177 INFO:teuthology.orchestra.run.smithi039.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-01-24T14:42:20.177 INFO:teuthology.orchestra.run.smithi039.stdout: = sunit=0 swidth=0 blks 2023-01-24T14:42:20.178 INFO:teuthology.orchestra.run.smithi039.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-01-24T14:42:20.178 INFO:teuthology.orchestra.run.smithi039.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-01-24T14:42:20.178 INFO:teuthology.orchestra.run.smithi039.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-01-24T14:42:20.179 INFO:teuthology.orchestra.run.smithi039.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-01-24T14:42:21.460 INFO:teuthology.orchestra.run.smithi039.stdout:Discarding blocks...Done. 2023-01-24T14:42:21.462 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi039.front.sepia.ceph.com -o noatime 2023-01-24T14:42:21.463 DEBUG:teuthology.orchestra.run.smithi039:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-3 2023-01-24T14:42:21.518 DEBUG:teuthology.orchestra.run.smithi039:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2023-01-24T14:42:21.553 DEBUG:teuthology.orchestra.run.smithi039:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-01-24T14:42:22.413 INFO:teuthology.orchestra.run.smithi039.stderr:WARNING: debug mode. Not for benchmarking or production 2023-01-24T14:42:22.413 INFO:teuthology.orchestra.run.smithi039.stderr:INFO 2023-01-24 14:42:22,413 seastar - Reactor backend: epoll 2023-01-24T14:42:22.427 INFO:teuthology.orchestra.run.smithi039.stderr:INFO 2023-01-24 14:42:22,426 [shard 0] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-01-24T14:42:22.427 INFO:teuthology.orchestra.run.smithi039.stderr:INFO 2023-01-24 14:42:22,426 [shard 0] seastar - IO queue uses 0.75ms latency goal for device 0 2023-01-24T14:42:22.427 INFO:teuthology.orchestra.run.smithi039.stderr:INFO 2023-01-24 14:42:22,426 [shard 0] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-01-24T14:42:22.427 INFO:teuthology.orchestra.run.smithi039.stderr:INFO 2023-01-24 14:42:22,426 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-01-24T14:42:22.429 INFO:teuthology.orchestra.run.smithi039.stderr:==100148==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-01-24T14:42:22.435 INFO:teuthology.orchestra.run.smithi039.stderr:==100148==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7ffe3c270000; bottom 0x7f8c52a0e000; size: 0x0071e9862000 (489249185792) 2023-01-24T14:42:22.435 INFO:teuthology.orchestra.run.smithi039.stderr:False positive error reports may follow 2023-01-24T14:42:22.436 INFO:teuthology.orchestra.run.smithi039.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-01-24T14:42:22.484 INFO:teuthology.orchestra.run.smithi039.stderr:INFO 2023-01-24 14:42:22,483 [shard 0] osd - bypassing the config fetch due to --no-mon-config 2023-01-24T14:42:22.484 INFO:teuthology.orchestra.run.smithi039.stderr:INFO 2023-01-24 14:42:22,483 [shard 0] osd - get_early_config: set --smp 1 2023-01-24T14:42:23.017 INFO:teuthology.orchestra.run.smithi039.stderr:WARNING: debug mode. Not for benchmarking or production 2023-01-24T14:42:23.018 INFO:teuthology.orchestra.run.smithi039.stderr:INFO 2023-01-24 14:42:23,018 seastar - Reactor backend: epoll 2023-01-24T14:42:23.020 INFO:teuthology.orchestra.run.smithi039.stderr:INFO 2023-01-24 14:42:23,019 [shard 0] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-01-24T14:42:23.020 INFO:teuthology.orchestra.run.smithi039.stderr:INFO 2023-01-24 14:42:23,019 [shard 0] seastar - IO queue uses 0.75ms latency goal for device 0 2023-01-24T14:42:23.020 INFO:teuthology.orchestra.run.smithi039.stderr:INFO 2023-01-24 14:42:23,019 [shard 0] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-01-24T14:42:23.020 INFO:teuthology.orchestra.run.smithi039.stderr:INFO 2023-01-24 14:42:23,019 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-01-24T14:42:23.021 INFO:teuthology.orchestra.run.smithi039.stderr:==100146==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-01-24T14:42:23.028 INFO:teuthology.orchestra.run.smithi039.stderr:==100146==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7ffe3c270000; bottom 0x7f8c52a07000; size: 0x0071e9869000 (489249214464) 2023-01-24T14:42:23.028 INFO:teuthology.orchestra.run.smithi039.stderr:False positive error reports may follow 2023-01-24T14:42:23.028 INFO:teuthology.orchestra.run.smithi039.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-01-24T14:42:26.001 INFO:teuthology.orchestra.run.smithi039.stdout:created object store /var/lib/ceph/osd/ceph-2 for osd.2 fsid 3808514d-c43d-4c1f-b296-499f8064f677 2023-01-24T14:42:26.065 DEBUG:teuthology.orchestra.run.smithi039:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2023-01-24T14:42:26.094 DEBUG:teuthology.orchestra.run.smithi039:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-01-24T14:42:26.936 INFO:teuthology.orchestra.run.smithi039.stderr:WARNING: debug mode. Not for benchmarking or production 2023-01-24T14:42:26.936 INFO:teuthology.orchestra.run.smithi039.stderr:INFO 2023-01-24 14:42:26,936 seastar - Reactor backend: epoll 2023-01-24T14:42:26.945 INFO:teuthology.orchestra.run.smithi039.stderr:INFO 2023-01-24 14:42:26,944 [shard 0] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-01-24T14:42:26.945 INFO:teuthology.orchestra.run.smithi039.stderr:INFO 2023-01-24 14:42:26,944 [shard 0] seastar - IO queue uses 0.75ms latency goal for device 0 2023-01-24T14:42:26.946 INFO:teuthology.orchestra.run.smithi039.stderr:INFO 2023-01-24 14:42:26,944 [shard 0] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-01-24T14:42:26.946 INFO:teuthology.orchestra.run.smithi039.stderr:INFO 2023-01-24 14:42:26,944 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-01-24T14:42:26.947 INFO:teuthology.orchestra.run.smithi039.stderr:==100974==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-01-24T14:42:26.953 INFO:teuthology.orchestra.run.smithi039.stderr:==100974==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7ffc1fcb0000; bottom 0x7f15b556f000; size: 0x00e66a741000 (989628469248) 2023-01-24T14:42:26.954 INFO:teuthology.orchestra.run.smithi039.stderr:False positive error reports may follow 2023-01-24T14:42:26.954 INFO:teuthology.orchestra.run.smithi039.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-01-24T14:42:26.990 INFO:teuthology.orchestra.run.smithi039.stderr:INFO 2023-01-24 14:42:26,989 [shard 0] osd - bypassing the config fetch due to --no-mon-config 2023-01-24T14:42:26.990 INFO:teuthology.orchestra.run.smithi039.stderr:INFO 2023-01-24 14:42:26,990 [shard 0] osd - get_early_config: set --smp 1 2023-01-24T14:42:27.506 INFO:teuthology.orchestra.run.smithi039.stderr:WARNING: debug mode. Not for benchmarking or production 2023-01-24T14:42:27.507 INFO:teuthology.orchestra.run.smithi039.stderr:INFO 2023-01-24 14:42:27,507 seastar - Reactor backend: epoll 2023-01-24T14:42:27.508 INFO:teuthology.orchestra.run.smithi039.stderr:INFO 2023-01-24 14:42:27,508 [shard 0] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-01-24T14:42:27.509 INFO:teuthology.orchestra.run.smithi039.stderr:INFO 2023-01-24 14:42:27,508 [shard 0] seastar - IO queue uses 0.75ms latency goal for device 0 2023-01-24T14:42:27.509 INFO:teuthology.orchestra.run.smithi039.stderr:INFO 2023-01-24 14:42:27,508 [shard 0] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-01-24T14:42:27.509 INFO:teuthology.orchestra.run.smithi039.stderr:INFO 2023-01-24 14:42:27,508 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-01-24T14:42:27.510 INFO:teuthology.orchestra.run.smithi039.stderr:==100972==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-01-24T14:42:27.516 INFO:teuthology.orchestra.run.smithi039.stderr:==100972==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7ffc1fcb0000; bottom 0x7f15b5568000; size: 0x00e66a748000 (989628497920) 2023-01-24T14:42:27.517 INFO:teuthology.orchestra.run.smithi039.stderr:False positive error reports may follow 2023-01-24T14:42:27.517 INFO:teuthology.orchestra.run.smithi039.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-01-24T14:42:30.481 INFO:teuthology.orchestra.run.smithi039.stdout:created object store /var/lib/ceph/osd/ceph-3 for osd.3 fsid 3808514d-c43d-4c1f-b296-499f8064f677 2023-01-24T14:42:30.570 DEBUG:teuthology.orchestra.run.smithi039:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2023-01-24T14:42:30.598 INFO:tasks.ceph:Reading keys from all nodes... 2023-01-24T14:42:30.599 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2023-01-24T14:42:30.599 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2023-01-24T14:42:30.634 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2023-01-24T14:42:30.634 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2023-01-24T14:42:30.707 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2023-01-24T14:42:30.708 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2023-01-24T14:42:30.736 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2023-01-24T14:42:30.736 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2023-01-24T14:42:30.803 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2023-01-24T14:42:30.804 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2023-01-24T14:42:30.872 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2023-01-24T14:42:30.872 DEBUG:teuthology.orchestra.run.smithi002:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2023-01-24T14:42:30.894 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2023-01-24T14:42:30.895 DEBUG:teuthology.orchestra.run.smithi039:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2023-01-24T14:42:30.930 INFO:tasks.ceph:Adding keys to all mons... 2023-01-24T14:42:30.931 DEBUG:teuthology.orchestra.run.smithi002:> sudo tee -a /etc/ceph/ceph.keyring 2023-01-24T14:42:30.965 INFO:teuthology.orchestra.run.smithi002.stdout:[osd.0] 2023-01-24T14:42:30.966 INFO:teuthology.orchestra.run.smithi002.stdout: key = AQBD7s9jBtH2ARAARnrA7zx3v5C4vp79TJ0yig== 2023-01-24T14:42:30.966 INFO:teuthology.orchestra.run.smithi002.stdout:[osd.1] 2023-01-24T14:42:30.966 INFO:teuthology.orchestra.run.smithi002.stdout: key = AQBH7s9jO8dDHRAAbmFunFDlR5ZJ1uhj+N+KDg== 2023-01-24T14:42:30.966 INFO:teuthology.orchestra.run.smithi002.stdout:[mgr.x] 2023-01-24T14:42:30.967 INFO:teuthology.orchestra.run.smithi002.stdout: key = AQBA7s9j20M0MRAAdonYgEQawSsyjkWX7LVwyQ== 2023-01-24T14:42:30.967 INFO:teuthology.orchestra.run.smithi002.stdout:[osd.2] 2023-01-24T14:42:30.967 INFO:teuthology.orchestra.run.smithi002.stdout: key = AQBP7s9jJPDkBBAAo3rZPenNlIFiWe85fHq02A== 2023-01-24T14:42:30.968 INFO:teuthology.orchestra.run.smithi002.stdout:[osd.3] 2023-01-24T14:42:30.968 INFO:teuthology.orchestra.run.smithi002.stdout: key = AQBT7s9jnSrtIRAAJVOm+Dfo8HqEo3qRuIb8jQ== 2023-01-24T14:42:30.968 INFO:teuthology.orchestra.run.smithi002.stdout:[client.0] 2023-01-24T14:42:30.968 INFO:teuthology.orchestra.run.smithi002.stdout: key = AQBA7s9jroKJNBAANEcZX4EyrvPq0IG2D81EFw== 2023-01-24T14:42:30.968 INFO:teuthology.orchestra.run.smithi002.stdout:[client.1] 2023-01-24T14:42:30.969 INFO:teuthology.orchestra.run.smithi002.stdout: key = AQBA7s9jedF9OBAAdg+hHFVscAK3v/TMRbqlTw== 2023-01-24T14:42:30.970 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-01-24T14:42:31.079 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-01-24T14:42:31.143 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-01-24T14:42:31.206 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-01-24T14:42:31.268 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-01-24T14:42:31.330 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-01-24T14:42:31.397 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-01-24T14:42:31.462 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-01-24T14:42:31.463 DEBUG:teuthology.orchestra.run.smithi002:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2023-01-24T14:42:31.491 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-01-24T14:42:31.627 DEBUG:teuthology.orchestra.run.smithi002:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2023-01-24T14:42:31.655 DEBUG:teuthology.orchestra.run.smithi002:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-01-24T14:42:31.713 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2023-01-24T14:42:31.713 INFO:tasks.ceph.mon.a:Restarting daemon 2023-01-24T14:42:31.714 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2023-01-24T14:42:31.756 INFO:tasks.ceph.mon.a:Started 2023-01-24T14:42:31.756 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2023-01-24T14:42:31.756 INFO:tasks.ceph.mgr.x:Restarting daemon 2023-01-24T14:42:31.757 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2023-01-24T14:42:31.759 INFO:tasks.ceph.mgr.x:Started 2023-01-24T14:42:31.759 INFO:tasks.ceph:Setting crush tunables to default 2023-01-24T14:42:31.760 DEBUG:teuthology.orchestra.run.smithi002:> sudo ceph --cluster ceph osd crush tunables default 2023-01-24T14:42:31.991 INFO:teuthology.orchestra.run.smithi002.stderr:adjusted tunables profile to default 2023-01-24T14:42:32.002 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2023-01-24T14:42:32.003 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2023-01-24T14:42:32.003 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2023-01-24T14:42:32.031 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2023-01-24T14:42:32.032 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2023-01-24T14:42:32.099 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2023-01-24T14:42:32.100 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2023-01-24T14:42:32.129 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2023-01-24T14:42:32.129 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2023-01-24T14:42:32.197 DEBUG:teuthology.orchestra.run.smithi039:> sudo ceph --cluster ceph osd new 5e1a8f93-714b-43f5-abb0-337ed2dff81e 0 2023-01-24T14:42:32.463 INFO:teuthology.orchestra.run.smithi039.stdout:0 2023-01-24T14:42:32.476 DEBUG:teuthology.orchestra.run.smithi039:> sudo ceph --cluster ceph osd new ef33d692-a846-4999-8754-c88dc16a9e47 1 2023-01-24T14:42:32.762 INFO:teuthology.orchestra.run.smithi039.stdout:1 2023-01-24T14:42:32.775 DEBUG:teuthology.orchestra.run.smithi039:> sudo ceph --cluster ceph osd new b03e5360-6d49-4073-804f-90be8c39d8eb 2 2023-01-24T14:42:33.002 INFO:teuthology.orchestra.run.smithi039.stdout:2 2023-01-24T14:42:33.015 DEBUG:teuthology.orchestra.run.smithi039:> sudo ceph --cluster ceph osd new 4a337027-3e3d-4d6a-914e-2d8b4ff89ddc 3 2023-01-24T14:42:33.080 INFO:tasks.ceph.mgr.x.smithi039.stderr:2023-01-24T14:42:33.079+0000 7f78f7a7b1c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-01-24T14:42:33.157 INFO:tasks.ceph.mgr.x.smithi039.stderr:2023-01-24T14:42:33.156+0000 7f78f7a7b1c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-01-24T14:42:33.245 INFO:teuthology.orchestra.run.smithi039.stdout:3 2023-01-24T14:42:33.246 INFO:tasks.ceph.mgr.x.smithi039.stderr:2023-01-24T14:42:33.244+0000 7f78f7a7b1c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-01-24T14:42:33.258 INFO:tasks.ceph.osd.0:Restarting daemon 2023-01-24T14:42:33.259 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2023-01-24T14:42:33.262 INFO:tasks.ceph.osd.0:Started 2023-01-24T14:42:33.262 INFO:tasks.ceph.osd.1:Restarting daemon 2023-01-24T14:42:33.262 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2023-01-24T14:42:33.265 INFO:tasks.ceph.osd.1:Started 2023-01-24T14:42:33.265 INFO:tasks.ceph.osd.2:Restarting daemon 2023-01-24T14:42:33.265 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2023-01-24T14:42:33.268 INFO:tasks.ceph.osd.2:Started 2023-01-24T14:42:33.268 INFO:tasks.ceph.osd.3:Restarting daemon 2023-01-24T14:42:33.268 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2023-01-24T14:42:33.270 INFO:tasks.ceph.osd.3:Started 2023-01-24T14:42:33.271 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-01-24T14:42:33.463 INFO:tasks.ceph.mgr.x.smithi039.stderr:2023-01-24T14:42:33.462+0000 7f78f7a7b1c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-01-24T14:42:33.511 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-01-24T14:42:33.512 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":6,"fsid":"3808514d-c43d-4c1f-b296-499f8064f677","created":"2023-01-24T14:42:31.895770+0000","modified":"2023-01-24T14:42:33.243250+0000","last_up_change":"0.000000","last_in_change":"2023-01-24T14:42:33.243250+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","pools":[],"osds":[{"osd":0,"uuid":"5e1a8f93-714b-43f5-abb0-337ed2dff81e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ef33d692-a846-4999-8754-c88dc16a9e47","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b03e5360-6d49-4073-804f-90be8c39d8eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4a337027-3e3d-4d6a-914e-2d8b4ff89ddc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-01-24T14:42:33.529 INFO:tasks.ceph.ceph_manager.ceph:[] 2023-01-24T14:42:33.529 INFO:tasks.ceph:Waiting for OSDs to come up 2023-01-24T14:42:33.932 DEBUG:teuthology.orchestra.run.smithi002:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-01-24T14:42:34.149 INFO:teuthology.misc.health.smithi002.stdout: 2023-01-24T14:42:34.150 INFO:teuthology.misc.health.smithi002.stdout:{"epoch":6,"fsid":"3808514d-c43d-4c1f-b296-499f8064f677","created":"2023-01-24T14:42:31.895770+0000","modified":"2023-01-24T14:42:33.243250+0000","last_up_change":"0.000000","last_in_change":"2023-01-24T14:42:33.243250+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","pools":[],"osds":[{"osd":0,"uuid":"5e1a8f93-714b-43f5-abb0-337ed2dff81e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ef33d692-a846-4999-8754-c88dc16a9e47","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b03e5360-6d49-4073-804f-90be8c39d8eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4a337027-3e3d-4d6a-914e-2d8b4ff89ddc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-01-24T14:42:34.161 DEBUG:teuthology.misc:0 of 4 OSDs are up 2023-01-24T14:42:34.186 INFO:tasks.ceph.osd.3.smithi039.stderr:WARNING: debug mode. Not for benchmarking or production 2023-01-24T14:42:34.187 INFO:tasks.ceph.osd.2.smithi039.stderr:WARNING: debug mode. Not for benchmarking or production 2023-01-24T14:42:34.188 INFO:tasks.ceph.osd.2.smithi039.stderr:INFO 2023-01-24 14:42:34,187 seastar - Reactor backend: epoll 2023-01-24T14:42:34.188 INFO:tasks.ceph.osd.3.smithi039.stderr:INFO 2023-01-24 14:42:34,188 seastar - Reactor backend: epoll 2023-01-24T14:42:34.191 INFO:tasks.ceph.osd.1.smithi002.stderr:WARNING: debug mode. Not for benchmarking or production 2023-01-24T14:42:34.192 INFO:tasks.ceph.osd.1.smithi002.stderr:INFO 2023-01-24 14:42:34,191 seastar - Reactor backend: epoll 2023-01-24T14:42:34.199 INFO:tasks.ceph.osd.0.smithi002.stderr:WARNING: debug mode. Not for benchmarking or production 2023-01-24T14:42:34.200 INFO:tasks.ceph.osd.0.smithi002.stderr:INFO 2023-01-24 14:42:34,200 seastar - Reactor backend: epoll 2023-01-24T14:42:34.248 INFO:tasks.ceph.osd.1.smithi002.stderr:INFO 2023-01-24 14:42:34,247 [shard 0] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-01-24T14:42:34.248 INFO:tasks.ceph.osd.1.smithi002.stderr:INFO 2023-01-24 14:42:34,247 [shard 0] seastar - IO queue uses 0.75ms latency goal for device 0 2023-01-24T14:42:34.249 INFO:tasks.ceph.osd.1.smithi002.stderr:INFO 2023-01-24 14:42:34,247 [shard 0] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-01-24T14:42:34.249 INFO:tasks.ceph.osd.1.smithi002.stderr:INFO 2023-01-24 14:42:34,247 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-01-24T14:42:34.250 INFO:tasks.ceph.osd.1.smithi002.stderr:==102658==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-01-24T14:42:34.256 INFO:tasks.ceph.osd.1.smithi002.stderr:==102658==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7ffceea99000; bottom 0x7fc61230c000; size: 0x0036dc78d000 (235627139072) 2023-01-24T14:42:34.257 INFO:tasks.ceph.osd.1.smithi002.stderr:False positive error reports may follow 2023-01-24T14:42:34.257 INFO:tasks.ceph.osd.1.smithi002.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-01-24T14:42:34.263 INFO:tasks.ceph.osd.0.smithi002.stderr:INFO 2023-01-24 14:42:34,262 [shard 0] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-01-24T14:42:34.263 INFO:tasks.ceph.osd.0.smithi002.stderr:INFO 2023-01-24 14:42:34,262 [shard 0] seastar - IO queue uses 0.75ms latency goal for device 0 2023-01-24T14:42:34.264 INFO:tasks.ceph.osd.0.smithi002.stderr:INFO 2023-01-24 14:42:34,262 [shard 0] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-01-24T14:42:34.264 INFO:tasks.ceph.osd.0.smithi002.stderr:INFO 2023-01-24 14:42:34,262 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-01-24T14:42:34.265 INFO:tasks.ceph.osd.2.smithi039.stderr:INFO 2023-01-24 14:42:34,263 [shard 0] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-01-24T14:42:34.266 INFO:tasks.ceph.osd.2.smithi039.stderr:INFO 2023-01-24 14:42:34,263 [shard 0] seastar - IO queue uses 0.75ms latency goal for device 0 2023-01-24T14:42:34.266 INFO:tasks.ceph.osd.2.smithi039.stderr:INFO 2023-01-24 14:42:34,263 [shard 0] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-01-24T14:42:34.266 INFO:tasks.ceph.osd.2.smithi039.stderr:INFO 2023-01-24 14:42:34,263 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-01-24T14:42:34.267 INFO:tasks.ceph.osd.0.smithi002.stderr:==102659==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-01-24T14:42:34.268 INFO:tasks.ceph.osd.2.smithi039.stderr:==102213==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-01-24T14:42:34.272 INFO:tasks.ceph.osd.0.smithi002.stderr:==102659==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7fff3fb81000; bottom 0x7f9523014000; size: 0x006a1cb6d000 (455748276224) 2023-01-24T14:42:34.272 INFO:tasks.ceph.osd.0.smithi002.stderr:False positive error reports may follow 2023-01-24T14:42:34.272 INFO:tasks.ceph.osd.0.smithi002.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-01-24T14:42:34.273 INFO:tasks.ceph.osd.2.smithi039.stderr:==102213==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7ffce5d15000; bottom 0x7f2714990000; size: 0x00d5d1385000 (918338162688) 2023-01-24T14:42:34.274 INFO:tasks.ceph.osd.2.smithi039.stderr:False positive error reports may follow 2023-01-24T14:42:34.274 INFO:tasks.ceph.osd.2.smithi039.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-01-24T14:42:34.279 INFO:tasks.ceph.osd.3.smithi039.stderr:INFO 2023-01-24 14:42:34,278 [shard 0] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-01-24T14:42:34.279 INFO:tasks.ceph.osd.3.smithi039.stderr:INFO 2023-01-24 14:42:34,278 [shard 0] seastar - IO queue uses 0.75ms latency goal for device 0 2023-01-24T14:42:34.279 INFO:tasks.ceph.osd.3.smithi039.stderr:INFO 2023-01-24 14:42:34,278 [shard 0] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-01-24T14:42:34.279 INFO:tasks.ceph.osd.3.smithi039.stderr:INFO 2023-01-24 14:42:34,278 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-01-24T14:42:34.281 INFO:tasks.ceph.osd.3.smithi039.stderr:==102214==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-01-24T14:42:34.287 INFO:tasks.ceph.osd.3.smithi039.stderr:==102214==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7ffc8c012000; bottom 0x7f5c639d2000; size: 0x00a028640000 (687872409600) 2023-01-24T14:42:34.288 INFO:tasks.ceph.osd.3.smithi039.stderr:False positive error reports may follow 2023-01-24T14:42:34.288 INFO:tasks.ceph.osd.3.smithi039.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-01-24T14:42:34.326 INFO:tasks.ceph.osd.0.smithi002.stderr:INFO 2023-01-24 14:42:34,325 [shard 0] osd - populating config from monitor 2023-01-24T14:42:34.327 INFO:tasks.ceph.osd.0.smithi002.stderr:INFO 2023-01-24 14:42:34,326 [shard 0] auth - AuthRegistry(0x7f9522fd86b0) adding auth protocol: cephx 2023-01-24T14:42:34.327 INFO:tasks.ceph.osd.0.smithi002.stderr:INFO 2023-01-24 14:42:34,326 [shard 0] auth - AuthRegistry(0x7f9522fd86b0) adding auth protocol: cephx 2023-01-24T14:42:34.328 INFO:tasks.ceph.osd.0.smithi002.stderr:INFO 2023-01-24 14:42:34,326 [shard 0] auth - AuthRegistry(0x7f9522fd86b0) adding auth protocol: cephx 2023-01-24T14:42:34.328 INFO:tasks.ceph.osd.0.smithi002.stderr:INFO 2023-01-24 14:42:34,326 [shard 0] auth - AuthRegistry(0x7f9522fd86b0) adding con mode: crc 2023-01-24T14:42:34.328 INFO:tasks.ceph.osd.0.smithi002.stderr:INFO 2023-01-24 14:42:34,326 [shard 0] auth - AuthRegistry(0x7f9522fd86b0) adding con mode: crc 2023-01-24T14:42:34.328 INFO:tasks.ceph.osd.1.smithi002.stderr:INFO 2023-01-24 14:42:34,327 [shard 0] osd - populating config from monitor 2023-01-24T14:42:34.329 INFO:tasks.ceph.osd.0.smithi002.stderr:INFO 2023-01-24 14:42:34,327 [shard 0] auth - AuthRegistry(0x7f9522fd86b0) adding con mode: crc 2023-01-24T14:42:34.330 INFO:tasks.ceph.osd.0.smithi002.stderr:INFO 2023-01-24 14:42:34,327 [shard 0] auth - AuthRegistry(0x7f9522fd86b0) adding con mode: crc 2023-01-24T14:42:34.330 INFO:tasks.ceph.osd.0.smithi002.stderr:INFO 2023-01-24 14:42:34,327 [shard 0] auth - AuthRegistry(0x7f9522fd86b0) adding con mode: crc 2023-01-24T14:42:34.330 INFO:tasks.ceph.osd.0.smithi002.stderr:INFO 2023-01-24 14:42:34,327 [shard 0] auth - AuthRegistry(0x7f9522fd86b0) adding con mode: crc 2023-01-24T14:42:34.330 INFO:tasks.ceph.osd.0.smithi002.stderr:INFO 2023-01-24 14:42:34,327 [shard 0] auth - auth: KeyRing::load: loaded key file /var/lib/ceph/osd/ceph-0/keyring 2023-01-24T14:42:34.331 INFO:tasks.ceph.osd.1.smithi002.stderr:INFO 2023-01-24 14:42:34,328 [shard 0] auth - AuthRegistry(0x7fc6122d06b0) adding auth protocol: cephx 2023-01-24T14:42:34.332 INFO:tasks.ceph.osd.1.smithi002.stderr:INFO 2023-01-24 14:42:34,328 [shard 0] auth - AuthRegistry(0x7fc6122d06b0) adding auth protocol: cephx 2023-01-24T14:42:34.332 INFO:tasks.ceph.osd.1.smithi002.stderr:INFO 2023-01-24 14:42:34,328 [shard 0] auth - AuthRegistry(0x7fc6122d06b0) adding auth protocol: cephx 2023-01-24T14:42:34.332 INFO:tasks.ceph.osd.1.smithi002.stderr:INFO 2023-01-24 14:42:34,328 [shard 0] auth - AuthRegistry(0x7fc6122d06b0) adding con mode: crc 2023-01-24T14:42:34.332 INFO:tasks.ceph.osd.1.smithi002.stderr:INFO 2023-01-24 14:42:34,328 [shard 0] auth - AuthRegistry(0x7fc6122d06b0) adding con mode: crc 2023-01-24T14:42:34.333 INFO:tasks.ceph.osd.1.smithi002.stderr:INFO 2023-01-24 14:42:34,328 [shard 0] auth - AuthRegistry(0x7fc6122d06b0) adding con mode: crc 2023-01-24T14:42:34.333 INFO:tasks.ceph.osd.1.smithi002.stderr:INFO 2023-01-24 14:42:34,328 [shard 0] auth - AuthRegistry(0x7fc6122d06b0) adding con mode: crc 2023-01-24T14:42:34.333 INFO:tasks.ceph.osd.1.smithi002.stderr:INFO 2023-01-24 14:42:34,328 [shard 0] auth - AuthRegistry(0x7fc6122d06b0) adding con mode: crc 2023-01-24T14:42:34.334 INFO:tasks.ceph.osd.1.smithi002.stderr:INFO 2023-01-24 14:42:34,328 [shard 0] auth - AuthRegistry(0x7fc6122d06b0) adding con mode: crc 2023-01-24T14:42:34.334 INFO:tasks.ceph.osd.1.smithi002.stderr:INFO 2023-01-24 14:42:34,329 [shard 0] auth - auth: KeyRing::load: loaded key file /var/lib/ceph/osd/ceph-1/keyring 2023-01-24T14:42:34.335 INFO:tasks.ceph.osd.0.smithi002.stderr:DEBUG 2023-01-24 14:42:34,331 [shard 0] monc - build_monmap: for_mkfs=false 2023-01-24T14:42:34.335 INFO:tasks.ceph.osd.0.smithi002.stderr:INFO 2023-01-24 14:42:34,331 [shard 0] monc - reopen_session to mon.-1 2023-01-24T14:42:34.335 INFO:tasks.ceph.osd.0.smithi002.stderr:INFO 2023-01-24 14:42:34,332 [shard 0] monc - connecting to mon.1 2023-01-24T14:42:34.335 INFO:tasks.ceph.osd.0.smithi002.stderr:INFO 2023-01-24 14:42:34,332 [shard 0] ms - [0x614000008040 client.?(temp_mon_client) - >> mon.? v2:172.21.15.2:3300/0] ProtocolV2::start_connect(): peer_addr=v2:172.21.15.2:3300/0, peer_name=mon.?, cc=3290262112156510106 policy(lossy=false, server=false, standby=false, resetcheck=true) 2023-01-24T14:42:34.336 INFO:tasks.ceph.osd.0.smithi002.stderr:INFO 2023-01-24 14:42:34,332 [shard 0] ms - [0x614000008040 client.?(temp_mon_client) - >> mon.? v2:172.21.15.2:3300/0] write_event: delay ... 2023-01-24T14:42:34.336 INFO:tasks.ceph.osd.0.smithi002.stderr:WARN 2023-01-24 14:42:34,332 [shard 0] monc - mon.0 does not have an addr compatible with me 2023-01-24T14:42:34.337 INFO:tasks.ceph.osd.1.smithi002.stderr:DEBUG 2023-01-24 14:42:34,332 [shard 0] monc - build_monmap: for_mkfs=false 2023-01-24T14:42:34.337 INFO:tasks.ceph.osd.1.smithi002.stderr:INFO 2023-01-24 14:42:34,333 [shard 0] monc - reopen_session to mon.-1 2023-01-24T14:42:34.337 INFO:tasks.ceph.osd.1.smithi002.stderr:INFO 2023-01-24 14:42:34,333 [shard 0] monc - connecting to mon.1 2023-01-24T14:42:34.337 INFO:tasks.ceph.osd.1.smithi002.stderr:INFO 2023-01-24 14:42:34,333 [shard 0] ms - [0x614000008040 client.?(temp_mon_client) - >> mon.? v2:172.21.15.2:3300/0] ProtocolV2::start_connect(): peer_addr=v2:172.21.15.2:3300/0, peer_name=mon.?, cc=157973429745555612 policy(lossy=false, server=false, standby=false, resetcheck=true) 2023-01-24T14:42:34.338 INFO:tasks.ceph.osd.1.smithi002.stderr:INFO 2023-01-24 14:42:34,333 [shard 0] ms - [0x614000008040 client.?(temp_mon_client) - >> mon.? v2:172.21.15.2:3300/0] write_event: delay ... 2023-01-24T14:42:34.338 INFO:tasks.ceph.osd.1.smithi002.stderr:WARN 2023-01-24 14:42:34,333 [shard 0] monc - mon.0 does not have an addr compatible with me 2023-01-24T14:42:34.339 INFO:tasks.ceph.osd.0.smithi002.stderr:INFO 2023-01-24 14:42:34,334 [shard 0] ms - [0x614000008040 client.?(temp_mon_client) -@65351 >> mon.? v2:172.21.15.2:3300/0] peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:65351/0 (socket says 172.21.15.2:65351) 2023-01-24T14:42:34.339 INFO:tasks.ceph.osd.0.smithi002.stderr:INFO 2023-01-24 14:42:34,334 [shard 0] ms - [0x614000008040 client.?(temp_mon_client) 172.21.15.2:0/102659@65351 >> mon.? v2:172.21.15.2:3300/0] learned myaddr=172.21.15.2:0/102659 (unbound) 2023-01-24T14:42:34.339 INFO:tasks.ceph.osd.0.smithi002.stderr:INFO 2023-01-24 14:42:34,334 [shard 0] monc - get_auth_request(conn=[0x614000008040 client.?(temp_mon_client) 172.21.15.2:0/102659@65351 >> mon.? v2:172.21.15.2:3300/0], auth_method=0) 2023-01-24T14:42:34.339 INFO:tasks.ceph.osd.0.smithi002.stderr:INFO 2023-01-24 14:42:34,334 [shard 0] monc - method 2 preferred_modes [1] 2023-01-24T14:42:34.340 INFO:tasks.ceph.osd.1.smithi002.stderr:INFO 2023-01-24 14:42:34,335 [shard 0] ms - [0x614000008040 client.?(temp_mon_client) -@54856 >> mon.? v2:172.21.15.2:3300/0] peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:54856/0 (socket says 172.21.15.2:54856) 2023-01-24T14:42:34.340 INFO:tasks.ceph.osd.1.smithi002.stderr:INFO 2023-01-24 14:42:34,335 [shard 0] ms - [0x614000008040 client.?(temp_mon_client) 172.21.15.2:0/102658@54856 >> mon.? v2:172.21.15.2:3300/0] learned myaddr=172.21.15.2:0/102658 (unbound) 2023-01-24T14:42:34.340 INFO:tasks.ceph.osd.1.smithi002.stderr:INFO 2023-01-24 14:42:34,335 [shard 0] monc - get_auth_request(conn=[0x614000008040 client.?(temp_mon_client) 172.21.15.2:0/102658@54856 >> mon.? v2:172.21.15.2:3300/0], auth_method=0) 2023-01-24T14:42:34.341 INFO:tasks.ceph.osd.1.smithi002.stderr:INFO 2023-01-24 14:42:34,335 [shard 0] monc - method 2 preferred_modes [1] 2023-01-24T14:42:34.353 INFO:tasks.ceph.osd.2.smithi039.stderr:INFO 2023-01-24 14:42:34,352 [shard 0] osd - populating config from monitor 2023-01-24T14:42:34.354 INFO:tasks.ceph.osd.2.smithi039.stderr:INFO 2023-01-24 14:42:34,353 [shard 0] auth - AuthRegistry(0x7f27149546b0) adding auth protocol: cephx 2023-01-24T14:42:34.354 INFO:tasks.ceph.osd.2.smithi039.stderr:INFO 2023-01-24 14:42:34,353 [shard 0] auth - AuthRegistry(0x7f27149546b0) adding auth protocol: cephx 2023-01-24T14:42:34.355 INFO:tasks.ceph.osd.2.smithi039.stderr:INFO 2023-01-24 14:42:34,353 [shard 0] auth - AuthRegistry(0x7f27149546b0) adding auth protocol: cephx 2023-01-24T14:42:34.355 INFO:tasks.ceph.osd.2.smithi039.stderr:INFO 2023-01-24 14:42:34,353 [shard 0] auth - AuthRegistry(0x7f27149546b0) adding con mode: crc 2023-01-24T14:42:34.355 INFO:tasks.ceph.osd.2.smithi039.stderr:INFO 2023-01-24 14:42:34,353 [shard 0] auth - AuthRegistry(0x7f27149546b0) adding con mode: crc 2023-01-24T14:42:34.355 INFO:tasks.ceph.osd.2.smithi039.stderr:INFO 2023-01-24 14:42:34,353 [shard 0] auth - AuthRegistry(0x7f27149546b0) adding con mode: crc 2023-01-24T14:42:34.356 INFO:tasks.ceph.osd.2.smithi039.stderr:INFO 2023-01-24 14:42:34,353 [shard 0] auth - AuthRegistry(0x7f27149546b0) adding con mode: crc 2023-01-24T14:42:34.356 INFO:tasks.ceph.osd.2.smithi039.stderr:INFO 2023-01-24 14:42:34,353 [shard 0] auth - AuthRegistry(0x7f27149546b0) adding con mode: crc 2023-01-24T14:42:34.356 INFO:tasks.ceph.osd.2.smithi039.stderr:INFO 2023-01-24 14:42:34,353 [shard 0] auth - AuthRegistry(0x7f27149546b0) adding con mode: crc 2023-01-24T14:42:34.356 INFO:tasks.ceph.osd.2.smithi039.stderr:INFO 2023-01-24 14:42:34,354 [shard 0] auth - auth: KeyRing::load: loaded key file /var/lib/ceph/osd/ceph-2/keyring 2023-01-24T14:42:34.358 INFO:tasks.ceph.osd.2.smithi039.stderr:DEBUG 2023-01-24 14:42:34,358 [shard 0] monc - build_monmap: for_mkfs=false 2023-01-24T14:42:34.359 INFO:tasks.ceph.osd.2.smithi039.stderr:INFO 2023-01-24 14:42:34,359 [shard 0] monc - reopen_session to mon.-1 2023-01-24T14:42:34.360 INFO:tasks.ceph.osd.2.smithi039.stderr:INFO 2023-01-24 14:42:34,359 [shard 0] monc - connecting to mon.1 2023-01-24T14:42:34.360 INFO:tasks.ceph.osd.2.smithi039.stderr:INFO 2023-01-24 14:42:34,359 [shard 0] ms - [0x614000008040 client.?(temp_mon_client) - >> mon.? v2:172.21.15.2:3300/0] ProtocolV2::start_connect(): peer_addr=v2:172.21.15.2:3300/0, peer_name=mon.?, cc=17739831029903215164 policy(lossy=false, server=false, standby=false, resetcheck=true) 2023-01-24T14:42:34.360 INFO:tasks.ceph.osd.2.smithi039.stderr:INFO 2023-01-24 14:42:34,359 [shard 0] ms - [0x614000008040 client.?(temp_mon_client) - >> mon.? v2:172.21.15.2:3300/0] write_event: delay ... 2023-01-24T14:42:34.360 INFO:tasks.ceph.osd.2.smithi039.stderr:WARN 2023-01-24 14:42:34,359 [shard 0] monc - mon.0 does not have an addr compatible with me 2023-01-24T14:42:34.361 INFO:tasks.ceph.osd.2.smithi039.stderr:INFO 2023-01-24 14:42:34,361 [shard 0] ms - [0x614000008040 client.?(temp_mon_client) -@63620 >> mon.? v2:172.21.15.2:3300/0] peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.39:63620/0 (socket says 172.21.15.39:63620) 2023-01-24T14:42:34.362 INFO:tasks.ceph.osd.2.smithi039.stderr:INFO 2023-01-24 14:42:34,361 [shard 0] ms - [0x614000008040 client.?(temp_mon_client) 172.21.15.39:0/102213@63620 >> mon.? v2:172.21.15.2:3300/0] learned myaddr=172.21.15.39:0/102213 (unbound) 2023-01-24T14:42:34.362 INFO:tasks.ceph.osd.2.smithi039.stderr:INFO 2023-01-24 14:42:34,362 [shard 0] monc - get_auth_request(conn=[0x614000008040 client.?(temp_mon_client) 172.21.15.39:0/102213@63620 >> mon.? v2:172.21.15.2:3300/0], auth_method=0) 2023-01-24T14:42:34.363 INFO:tasks.ceph.osd.2.smithi039.stderr:INFO 2023-01-24 14:42:34,362 [shard 0] monc - method 2 preferred_modes [1] 2023-01-24T14:42:34.367 INFO:tasks.ceph.osd.3.smithi039.stderr:INFO 2023-01-24 14:42:34,367 [shard 0] osd - populating config from monitor 2023-01-24T14:42:34.368 INFO:tasks.ceph.osd.3.smithi039.stderr:INFO 2023-01-24 14:42:34,367 [shard 0] auth - AuthRegistry(0x7f5c639966b0) adding auth protocol: cephx 2023-01-24T14:42:34.368 INFO:tasks.ceph.osd.3.smithi039.stderr:INFO 2023-01-24 14:42:34,367 [shard 0] auth - AuthRegistry(0x7f5c639966b0) adding auth protocol: cephx 2023-01-24T14:42:34.369 INFO:tasks.ceph.osd.3.smithi039.stderr:INFO 2023-01-24 14:42:34,367 [shard 0] auth - AuthRegistry(0x7f5c639966b0) adding auth protocol: cephx 2023-01-24T14:42:34.369 INFO:tasks.ceph.osd.3.smithi039.stderr:INFO 2023-01-24 14:42:34,367 [shard 0] auth - AuthRegistry(0x7f5c639966b0) adding con mode: crc 2023-01-24T14:42:34.369 INFO:tasks.ceph.osd.3.smithi039.stderr:INFO 2023-01-24 14:42:34,367 [shard 0] auth - AuthRegistry(0x7f5c639966b0) adding con mode: crc 2023-01-24T14:42:34.369 INFO:tasks.ceph.osd.3.smithi039.stderr:INFO 2023-01-24 14:42:34,367 [shard 0] auth - AuthRegistry(0x7f5c639966b0) adding con mode: crc 2023-01-24T14:42:34.370 INFO:tasks.ceph.osd.3.smithi039.stderr:INFO 2023-01-24 14:42:34,367 [shard 0] auth - AuthRegistry(0x7f5c639966b0) adding con mode: crc 2023-01-24T14:42:34.370 INFO:tasks.ceph.osd.3.smithi039.stderr:INFO 2023-01-24 14:42:34,367 [shard 0] auth - AuthRegistry(0x7f5c639966b0) adding con mode: crc 2023-01-24T14:42:34.370 INFO:tasks.ceph.osd.3.smithi039.stderr:INFO 2023-01-24 14:42:34,367 [shard 0] auth - AuthRegistry(0x7f5c639966b0) adding con mode: crc 2023-01-24T14:42:34.370 INFO:tasks.ceph.osd.3.smithi039.stderr:INFO 2023-01-24 14:42:34,368 [shard 0] auth - auth: KeyRing::load: loaded key file /var/lib/ceph/osd/ceph-3/keyring 2023-01-24T14:42:34.375 INFO:tasks.ceph.osd.1.smithi002.stderr:INFO 2023-01-24 14:42:34,375 [shard 0] monc - responding with 36 bytes 2023-01-24T14:42:34.376 INFO:tasks.ceph.mgr.x.smithi039.stderr:2023-01-24T14:42:34.375+0000 7f78f7a7b1c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-01-24T14:42:34.376 INFO:tasks.ceph.osd.0.smithi002.stderr:INFO 2023-01-24 14:42:34,375 [shard 0] monc - responding with 36 bytes 2023-01-24T14:42:34.377 INFO:tasks.ceph.osd.0.smithi002.stderr:INFO 2023-01-24 14:42:34,376 [shard 0] monc - found mon.noname-a 2023-01-24T14:42:34.377 INFO:tasks.ceph.osd.0.smithi002.stderr:DEBUG 2023-01-24 14:42:34,376 [shard 0] monc - reopen_session mon connection attempts complete 2023-01-24T14:42:34.377 INFO:tasks.ceph.osd.0.smithi002.stderr:INFO 2023-01-24 14:42:34,376 [shard 0] monc - renew_rotating_keyring renewing rotating keys (they expired before 1674571324.3766382) 2023-01-24T14:42:34.378 INFO:tasks.ceph.osd.0.smithi002.stderr:INFO 2023-01-24 14:42:34,376 [shard 0] monc - sending auth(proto 2 2 bytes epoch 0) v1 2023-01-24T14:42:34.378 INFO:tasks.ceph.osd.0.smithi002.stderr:INFO 2023-01-24 14:42:34,376 [shard 0] monc - waiting 2023-01-24T14:42:34.379 INFO:tasks.ceph.osd.1.smithi002.stderr:INFO 2023-01-24 14:42:34,377 [shard 0] monc - found mon.noname-a 2023-01-24T14:42:34.379 INFO:tasks.ceph.osd.1.smithi002.stderr:DEBUG 2023-01-24 14:42:34,377 [shard 0] monc - reopen_session mon connection attempts complete 2023-01-24T14:42:34.379 INFO:tasks.ceph.osd.1.smithi002.stderr:INFO 2023-01-24 14:42:34,377 [shard 0] monc - renew_rotating_keyring renewing rotating keys (they expired before 1674571324.3771532) 2023-01-24T14:42:34.379 INFO:tasks.ceph.osd.1.smithi002.stderr:INFO 2023-01-24 14:42:34,377 [shard 0] monc - sending auth(proto 2 2 bytes epoch 0) v1 2023-01-24T14:42:34.379 INFO:tasks.ceph.osd.1.smithi002.stderr:INFO 2023-01-24 14:42:34,377 [shard 0] monc - waiting 2023-01-24T14:42:34.380 INFO:tasks.ceph.osd.3.smithi039.stderr:DEBUG 2023-01-24 14:42:34,380 [shard 0] monc - build_monmap: for_mkfs=false 2023-01-24T14:42:34.381 INFO:tasks.ceph.osd.3.smithi039.stderr:INFO 2023-01-24 14:42:34,380 [shard 0] monc - reopen_session to mon.-1 2023-01-24T14:42:34.381 INFO:tasks.ceph.osd.3.smithi039.stderr:WARN 2023-01-24 14:42:34,380 [shard 0] monc - mon.0 does not have an addr compatible with me 2023-01-24T14:42:34.381 INFO:tasks.ceph.osd.3.smithi039.stderr:INFO 2023-01-24 14:42:34,380 [shard 0] monc - connecting to mon.1 2023-01-24T14:42:34.381 INFO:tasks.ceph.osd.3.smithi039.stderr:INFO 2023-01-24 14:42:34,380 [shard 0] ms - [0x614000008040 client.?(temp_mon_client) - >> mon.? v2:172.21.15.2:3300/0] ProtocolV2::start_connect(): peer_addr=v2:172.21.15.2:3300/0, peer_name=mon.?, cc=864080357717066120 policy(lossy=false, server=false, standby=false, resetcheck=true) 2023-01-24T14:42:34.381 INFO:tasks.ceph.osd.3.smithi039.stderr:INFO 2023-01-24 14:42:34,380 [shard 0] ms - [0x614000008040 client.?(temp_mon_client) - >> mon.? v2:172.21.15.2:3300/0] write_event: delay ... 2023-01-24T14:42:34.382 INFO:tasks.ceph.osd.3.smithi039.stderr:INFO 2023-01-24 14:42:34,381 [shard 0] ms - [0x614000008040 client.?(temp_mon_client) -@59033 >> mon.? v2:172.21.15.2:3300/0] peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.39:59033/0 (socket says 172.21.15.39:59033) 2023-01-24T14:42:34.383 INFO:tasks.ceph.osd.3.smithi039.stderr:INFO 2023-01-24 14:42:34,381 [shard 0] ms - [0x614000008040 client.?(temp_mon_client) 172.21.15.39:0/102214@59033 >> mon.? v2:172.21.15.2:3300/0] learned myaddr=172.21.15.39:0/102214 (unbound) 2023-01-24T14:42:34.383 INFO:tasks.ceph.osd.3.smithi039.stderr:INFO 2023-01-24 14:42:34,381 [shard 0] monc - get_auth_request(conn=[0x614000008040 client.?(temp_mon_client) 172.21.15.39:0/102214@59033 >> mon.? v2:172.21.15.2:3300/0], auth_method=0) 2023-01-24T14:42:34.383 INFO:tasks.ceph.osd.3.smithi039.stderr:INFO 2023-01-24 14:42:34,381 [shard 0] monc - method 2 preferred_modes [1] 2023-01-24T14:42:34.402 INFO:tasks.ceph.osd.2.smithi039.stderr:INFO 2023-01-24 14:42:34,402 [shard 0] monc - responding with 36 bytes 2023-01-24T14:42:34.403 INFO:tasks.ceph.osd.2.smithi039.stderr:INFO 2023-01-24 14:42:34,403 [shard 0] monc - found mon.noname-a 2023-01-24T14:42:34.404 INFO:tasks.ceph.osd.2.smithi039.stderr:DEBUG 2023-01-24 14:42:34,403 [shard 0] monc - reopen_session mon connection attempts complete 2023-01-24T14:42:34.404 INFO:tasks.ceph.osd.2.smithi039.stderr:INFO 2023-01-24 14:42:34,403 [shard 0] monc - renew_rotating_keyring renewing rotating keys (they expired before 1674571324.4035854) 2023-01-24T14:42:34.404 INFO:tasks.ceph.osd.2.smithi039.stderr:INFO 2023-01-24 14:42:34,403 [shard 0] monc - sending auth(proto 2 2 bytes epoch 0) v1 2023-01-24T14:42:34.404 INFO:tasks.ceph.osd.2.smithi039.stderr:INFO 2023-01-24 14:42:34,403 [shard 0] monc - waiting 2023-01-24T14:42:34.417 INFO:tasks.ceph.osd.0.smithi002.stderr:WARN 2023-01-24 14:42:34,417 [shard 0] ms - [0x614000008040 client.?(temp_mon_client) 172.21.15.2:0/102659@65351 >> mon.0 v2:172.21.15.2:3300/0] UPDATE Policy(lossy=true) from server flags 2023-01-24T14:42:34.418 INFO:tasks.ceph.osd.0.smithi002.stderr:INFO 2023-01-24 14:42:34,417 [shard 0] ms - [0x614000008040 client.?(temp_mon_client) 172.21.15.2:0/102659@65351 >> mon.0 v2:172.21.15.2:3300/0] connected: gs=1, pgs=25, cs=0, client_cookie=3290262112156510106, server_cookie=0, in_seq=0, out_seq=0, out_q=2 2023-01-24T14:42:34.419 INFO:tasks.ceph.osd.0.smithi002.stderr:INFO 2023-01-24 14:42:34,418 [shard 0] monc - got monmap 1, mon.a, is now rank 0 2023-01-24T14:42:34.419 INFO:tasks.ceph.osd.0.smithi002.stderr:INFO 2023-01-24 14:42:34,418 [shard 0] monc - handle_monmap: renewing tickets 2023-01-24T14:42:34.419 INFO:tasks.ceph.osd.0.smithi002.stderr:INFO 2023-01-24 14:42:34,418 [shard 0] monc - renew_rotating_keyring renewing rotating keys (they expired before 1674571324.4180388) 2023-01-24T14:42:34.419 INFO:tasks.ceph.osd.0.smithi002.stderr:INFO 2023-01-24 14:42:34,418 [shard 0] monc - renew_rotating_keyring called too often (last: 1674571354.3766382) 2023-01-24T14:42:34.420 INFO:tasks.ceph.osd.0.smithi002.stderr:DEBUG 2023-01-24 14:42:34,418 [shard 0] monc - renew_tickets: don't need new tickets 2023-01-24T14:42:34.421 INFO:tasks.ceph.osd.1.smithi002.stderr:WARN 2023-01-24 14:42:34,418 [shard 0] ms - [0x614000008040 client.?(temp_mon_client) 172.21.15.2:0/102658@54856 >> mon.0 v2:172.21.15.2:3300/0] UPDATE Policy(lossy=true) from server flags 2023-01-24T14:42:34.421 INFO:tasks.ceph.osd.1.smithi002.stderr:INFO 2023-01-24 14:42:34,418 [shard 0] ms - [0x614000008040 client.?(temp_mon_client) 172.21.15.2:0/102658@54856 >> mon.0 v2:172.21.15.2:3300/0] connected: gs=1, pgs=26, cs=0, client_cookie=157973429745555612, server_cookie=0, in_seq=0, out_seq=0, out_q=2 2023-01-24T14:42:34.421 INFO:tasks.ceph.osd.1.smithi002.stderr:INFO 2023-01-24 14:42:34,418 [shard 0] monc - got monmap 1, mon.a, is now rank 0 2023-01-24T14:42:34.421 INFO:tasks.ceph.osd.1.smithi002.stderr:INFO 2023-01-24 14:42:34,418 [shard 0] monc - handle_monmap: renewing tickets 2023-01-24T14:42:34.421 INFO:tasks.ceph.osd.1.smithi002.stderr:INFO 2023-01-24 14:42:34,418 [shard 0] monc - renew_rotating_keyring renewing rotating keys (they expired before 1674571324.4189024) 2023-01-24T14:42:34.422 INFO:tasks.ceph.osd.1.smithi002.stderr:INFO 2023-01-24 14:42:34,418 [shard 0] monc - renew_rotating_keyring called too often (last: 1674571354.3771532) 2023-01-24T14:42:34.422 INFO:tasks.ceph.osd.1.smithi002.stderr:DEBUG 2023-01-24 14:42:34,419 [shard 0] monc - renew_tickets: don't need new tickets 2023-01-24T14:42:34.422 INFO:tasks.ceph.osd.1.smithi002.stderr:INFO 2023-01-24 14:42:34,419 [shard 0] monc - handle_mon_map: renewed tickets 2023-01-24T14:42:34.423 INFO:tasks.ceph.osd.0.smithi002.stderr:INFO 2023-01-24 14:42:34,419 [shard 0] monc - handle_mon_map: renewed tickets 2023-01-24T14:42:34.423 INFO:tasks.ceph.osd.0.smithi002.stderr:INFO 2023-01-24 14:42:34,419 [shard 0] monc - handle_auth_reply [0x614000008040 client.?(temp_mon_client) 172.21.15.2:0/102659@65351 >> mon.0 v2:172.21.15.2:3300/0] returns auth_reply(proto 2 0 (0) Success) v1: 0 2023-01-24T14:42:34.424 INFO:tasks.ceph.osd.0.smithi002.stderr:INFO 2023-01-24 14:42:34,419 [shard 0] monc - handle_auth_reply 2023-01-24T14:42:34.424 INFO:tasks.ceph.osd.0.smithi002.stderr:DEBUG 2023-01-24 14:42:34,419 [shard 0] monc - renew_tickets: don't need new tickets 2023-01-24T14:42:34.424 INFO:tasks.ceph.osd.0.smithi002.stderr:INFO 2023-01-24 14:42:34,419 [shard 0] monc - renew_rotating_keyring renewing rotating keys (they expired before 1674571324.4195523) 2023-01-24T14:42:34.424 INFO:tasks.ceph.osd.0.smithi002.stderr:INFO 2023-01-24 14:42:34,419 [shard 0] monc - renew_rotating_keyring called too often (last: 1674571354.3766382) 2023-01-24T14:42:34.424 INFO:tasks.ceph.osd.0.smithi002.stderr:INFO 2023-01-24 14:42:34,419 [shard 0] monc - do_auth_single: [0x614000008040 client.?(temp_mon_client) 172.21.15.2:0/102659@65351 >> mon.0 v2:172.21.15.2:3300/0] returns auth_reply(proto 2 0 (0) Success) v1: 0 2023-01-24T14:42:34.426 INFO:tasks.ceph.osd.1.smithi002.stderr:INFO 2023-01-24 14:42:34,419 [shard 0] monc - handle_auth_reply [0x614000008040 client.?(temp_mon_client) 172.21.15.2:0/102658@54856 >> mon.0 v2:172.21.15.2:3300/0] returns auth_reply(proto 2 0 (0) Success) v1: 0 2023-01-24T14:42:34.426 INFO:tasks.ceph.osd.1.smithi002.stderr:INFO 2023-01-24 14:42:34,419 [shard 0] monc - handle_auth_reply 2023-01-24T14:42:34.426 INFO:tasks.ceph.osd.1.smithi002.stderr:DEBUG 2023-01-24 14:42:34,419 [shard 0] monc - renew_tickets: don't need new tickets 2023-01-24T14:42:34.426 INFO:tasks.ceph.osd.1.smithi002.stderr:INFO 2023-01-24 14:42:34,419 [shard 0] monc - renew_rotating_keyring renewing rotating keys (they expired before 1674571324.4198477) 2023-01-24T14:42:34.427 INFO:tasks.ceph.osd.1.smithi002.stderr:INFO 2023-01-24 14:42:34,419 [shard 0] monc - renew_rotating_keyring called too often (last: 1674571354.3771532) 2023-01-24T14:42:34.427 INFO:tasks.ceph.osd.1.smithi002.stderr:INFO 2023-01-24 14:42:34,419 [shard 0] monc - do_auth_single: [0x614000008040 client.?(temp_mon_client) 172.21.15.2:0/102658@54856 >> mon.0 v2:172.21.15.2:3300/0] returns auth_reply(proto 2 0 (0) Success) v1: 0 2023-01-24T14:42:34.427 INFO:tasks.ceph.osd.1.smithi002.stderr:INFO 2023-01-24 14:42:34,421 [shard 0] monc - set_mon_vals no callback set 2023-01-24T14:42:34.428 INFO:tasks.ceph.osd.3.smithi039.stderr:INFO 2023-01-24 14:42:34,422 [shard 0] monc - responding with 36 bytes 2023-01-24T14:42:34.428 INFO:tasks.ceph.osd.3.smithi039.stderr:INFO 2023-01-24 14:42:34,423 [shard 0] monc - found mon.noname-a 2023-01-24T14:42:34.428 INFO:tasks.ceph.osd.3.smithi039.stderr:DEBUG 2023-01-24 14:42:34,423 [shard 0] monc - reopen_session mon connection attempts complete 2023-01-24T14:42:34.429 INFO:tasks.ceph.osd.3.smithi039.stderr:INFO 2023-01-24 14:42:34,423 [shard 0] monc - renew_rotating_keyring renewing rotating keys (they expired before 1674571324.4231515) 2023-01-24T14:42:34.429 INFO:tasks.ceph.osd.3.smithi039.stderr:INFO 2023-01-24 14:42:34,423 [shard 0] monc - sending auth(proto 2 2 bytes epoch 0) v1 2023-01-24T14:42:34.429 INFO:tasks.ceph.osd.3.smithi039.stderr:INFO 2023-01-24 14:42:34,423 [shard 0] monc - waiting 2023-01-24T14:42:34.429 INFO:tasks.ceph.osd.0.smithi002.stderr:INFO 2023-01-24 14:42:34,420 [shard 0] monc - set_mon_vals no callback set 2023-01-24T14:42:34.429 INFO:tasks.ceph.osd.0.smithi002.stderr:DEBUG 2023-01-24 14:42:34,427 [shard 0] osd - operator(): got config from monitor, fsid 3808514d-c43d-4c1f-b296-499f8064f677 2023-01-24T14:42:34.430 INFO:tasks.ceph.osd.0.smithi002.stderr:INFO 2023-01-24 14:42:34,427 [shard 0] monc - stop 2023-01-24T14:42:34.430 INFO:tasks.ceph.osd.0.smithi002.stderr:INFO 2023-01-24 14:42:34,427 [shard 0] monc - close 2023-01-24T14:42:34.430 INFO:tasks.ceph.osd.0.smithi002.stderr:INFO 2023-01-24 14:42:34,427 [shard 0] ms - [0x614000008040 client.?(temp_mon_client) 172.21.15.2:0/102659@65351 >> mon.0 v2:172.21.15.2:3300/0] closing: reset no, replace no 2023-01-24T14:42:34.431 INFO:tasks.ceph.osd.1.smithi002.stderr:DEBUG 2023-01-24 14:42:34,427 [shard 0] osd - operator(): got config from monitor, fsid 3808514d-c43d-4c1f-b296-499f8064f677 2023-01-24T14:42:34.431 INFO:tasks.ceph.osd.1.smithi002.stderr:INFO 2023-01-24 14:42:34,427 [shard 0] monc - stop 2023-01-24T14:42:34.431 INFO:tasks.ceph.osd.1.smithi002.stderr:INFO 2023-01-24 14:42:34,427 [shard 0] monc - close 2023-01-24T14:42:34.431 INFO:tasks.ceph.osd.1.smithi002.stderr:INFO 2023-01-24 14:42:34,427 [shard 0] ms - [0x614000008040 client.?(temp_mon_client) 172.21.15.2:0/102658@54856 >> mon.0 v2:172.21.15.2:3300/0] closing: reset no, replace no 2023-01-24T14:42:34.431 INFO:tasks.ceph.osd.1.smithi002.stderr:INFO 2023-01-24 14:42:34,428 [shard 0] ms - [0x614000008040 client.?(temp_mon_client) 172.21.15.2:0/102658@54856 >> mon.0 v2:172.21.15.2:3300/0] execute_ready(): protocol aborted at CLOSING -- std::system_error (error crimson::net:4, read eof) 2023-01-24T14:42:34.432 INFO:tasks.ceph.osd.0.smithi002.stderr:INFO 2023-01-24 14:42:34,427 [shard 0] ms - [0x614000008040 client.?(temp_mon_client) 172.21.15.2:0/102659@65351 >> mon.0 v2:172.21.15.2:3300/0] execute_ready(): protocol aborted at CLOSING -- std::system_error (error crimson::net:4, read eof) 2023-01-24T14:42:34.432 INFO:tasks.ceph.osd.0.smithi002.stderr:INFO 2023-01-24 14:42:34,428 [shard 0] osd - get_early_config: set --smp 1 2023-01-24T14:42:34.433 INFO:tasks.ceph.osd.1.smithi002.stderr:INFO 2023-01-24 14:42:34,428 [shard 0] osd - get_early_config: set --smp 1 2023-01-24T14:42:34.445 INFO:tasks.ceph.osd.2.smithi039.stderr:WARN 2023-01-24 14:42:34,444 [shard 0] ms - [0x614000008040 client.?(temp_mon_client) 172.21.15.39:0/102213@63620 >> mon.0 v2:172.21.15.2:3300/0] UPDATE Policy(lossy=true) from server flags 2023-01-24T14:42:34.445 INFO:tasks.ceph.osd.2.smithi039.stderr:INFO 2023-01-24 14:42:34,444 [shard 0] ms - [0x614000008040 client.?(temp_mon_client) 172.21.15.39:0/102213@63620 >> mon.0 v2:172.21.15.2:3300/0] connected: gs=1, pgs=27, cs=0, client_cookie=17739831029903215164, server_cookie=0, in_seq=0, out_seq=0, out_q=2 2023-01-24T14:42:34.446 INFO:tasks.ceph.osd.2.smithi039.stderr:INFO 2023-01-24 14:42:34,445 [shard 0] monc - got monmap 1, mon.a, is now rank 0 2023-01-24T14:42:34.447 INFO:tasks.ceph.osd.2.smithi039.stderr:INFO 2023-01-24 14:42:34,445 [shard 0] monc - handle_monmap: renewing tickets 2023-01-24T14:42:34.447 INFO:tasks.ceph.osd.2.smithi039.stderr:INFO 2023-01-24 14:42:34,445 [shard 0] monc - renew_rotating_keyring renewing rotating keys (they expired before 1674571324.4454103) 2023-01-24T14:42:34.447 INFO:tasks.ceph.osd.2.smithi039.stderr:INFO 2023-01-24 14:42:34,445 [shard 0] monc - renew_rotating_keyring called too often (last: 1674571354.4035854) 2023-01-24T14:42:34.447 INFO:tasks.ceph.osd.2.smithi039.stderr:DEBUG 2023-01-24 14:42:34,445 [shard 0] monc - renew_tickets: don't need new tickets 2023-01-24T14:42:34.447 INFO:tasks.ceph.osd.2.smithi039.stderr:INFO 2023-01-24 14:42:34,445 [shard 0] monc - handle_mon_map: renewed tickets 2023-01-24T14:42:34.448 INFO:tasks.ceph.osd.2.smithi039.stderr:INFO 2023-01-24 14:42:34,445 [shard 0] monc - handle_auth_reply [0x614000008040 client.?(temp_mon_client) 172.21.15.39:0/102213@63620 >> mon.0 v2:172.21.15.2:3300/0] returns auth_reply(proto 2 0 (0) Success) v1: 0 2023-01-24T14:42:34.448 INFO:tasks.ceph.osd.2.smithi039.stderr:INFO 2023-01-24 14:42:34,445 [shard 0] monc - handle_auth_reply 2023-01-24T14:42:34.448 INFO:tasks.ceph.osd.2.smithi039.stderr:DEBUG 2023-01-24 14:42:34,445 [shard 0] monc - renew_tickets: don't need new tickets 2023-01-24T14:42:34.448 INFO:tasks.ceph.osd.2.smithi039.stderr:INFO 2023-01-24 14:42:34,445 [shard 0] monc - renew_rotating_keyring renewing rotating keys (they expired before 1674571324.4458952) 2023-01-24T14:42:34.448 INFO:tasks.ceph.osd.2.smithi039.stderr:INFO 2023-01-24 14:42:34,445 [shard 0] monc - renew_rotating_keyring called too often (last: 1674571354.4035854) 2023-01-24T14:42:34.449 INFO:tasks.ceph.osd.2.smithi039.stderr:INFO 2023-01-24 14:42:34,445 [shard 0] monc - do_auth_single: [0x614000008040 client.?(temp_mon_client) 172.21.15.39:0/102213@63620 >> mon.0 v2:172.21.15.2:3300/0] returns auth_reply(proto 2 0 (0) Success) v1: 0 2023-01-24T14:42:34.449 INFO:tasks.ceph.osd.2.smithi039.stderr:INFO 2023-01-24 14:42:34,446 [shard 0] monc - set_mon_vals no callback set 2023-01-24T14:42:34.450 INFO:tasks.ceph.osd.2.smithi039.stderr:DEBUG 2023-01-24 14:42:34,449 [shard 0] osd - operator(): got config from monitor, fsid 3808514d-c43d-4c1f-b296-499f8064f677 2023-01-24T14:42:34.450 INFO:tasks.ceph.osd.2.smithi039.stderr:INFO 2023-01-24 14:42:34,449 [shard 0] monc - stop 2023-01-24T14:42:34.450 INFO:tasks.ceph.osd.2.smithi039.stderr:INFO 2023-01-24 14:42:34,449 [shard 0] monc - close 2023-01-24T14:42:34.450 INFO:tasks.ceph.osd.2.smithi039.stderr:INFO 2023-01-24 14:42:34,449 [shard 0] ms - [0x614000008040 client.?(temp_mon_client) 172.21.15.39:0/102213@63620 >> mon.0 v2:172.21.15.2:3300/0] closing: reset no, replace no 2023-01-24T14:42:34.451 INFO:tasks.ceph.osd.2.smithi039.stderr:INFO 2023-01-24 14:42:34,450 [shard 0] ms - [0x614000008040 client.?(temp_mon_client) 172.21.15.39:0/102213@63620 >> mon.0 v2:172.21.15.2:3300/0] execute_ready(): protocol aborted at CLOSING -- std::system_error (error crimson::net:4, read eof) 2023-01-24T14:42:34.451 INFO:tasks.ceph.osd.2.smithi039.stderr:INFO 2023-01-24 14:42:34,450 [shard 0] osd - get_early_config: set --smp 1 2023-01-24T14:42:34.464 INFO:tasks.ceph.osd.3.smithi039.stderr:WARN 2023-01-24 14:42:34,464 [shard 0] ms - [0x614000008040 client.?(temp_mon_client) 172.21.15.39:0/102214@59033 >> mon.0 v2:172.21.15.2:3300/0] UPDATE Policy(lossy=true) from server flags 2023-01-24T14:42:34.465 INFO:tasks.ceph.osd.3.smithi039.stderr:INFO 2023-01-24 14:42:34,464 [shard 0] ms - [0x614000008040 client.?(temp_mon_client) 172.21.15.39:0/102214@59033 >> mon.0 v2:172.21.15.2:3300/0] connected: gs=1, pgs=28, cs=0, client_cookie=864080357717066120, server_cookie=0, in_seq=0, out_seq=0, out_q=2 2023-01-24T14:42:34.467 INFO:tasks.ceph.osd.3.smithi039.stderr:INFO 2023-01-24 14:42:34,465 [shard 0] monc - got monmap 1, mon.a, is now rank 0 2023-01-24T14:42:34.467 INFO:tasks.ceph.osd.3.smithi039.stderr:INFO 2023-01-24 14:42:34,465 [shard 0] monc - handle_monmap: renewing tickets 2023-01-24T14:42:34.468 INFO:tasks.ceph.osd.3.smithi039.stderr:INFO 2023-01-24 14:42:34,465 [shard 0] monc - renew_rotating_keyring renewing rotating keys (they expired before 1674571324.46507) 2023-01-24T14:42:34.468 INFO:tasks.ceph.osd.3.smithi039.stderr:INFO 2023-01-24 14:42:34,465 [shard 0] monc - renew_rotating_keyring called too often (last: 1674571354.4231515) 2023-01-24T14:42:34.468 INFO:tasks.ceph.osd.3.smithi039.stderr:DEBUG 2023-01-24 14:42:34,465 [shard 0] monc - renew_tickets: don't need new tickets 2023-01-24T14:42:34.469 INFO:tasks.ceph.osd.3.smithi039.stderr:INFO 2023-01-24 14:42:34,465 [shard 0] monc - handle_mon_map: renewed tickets 2023-01-24T14:42:34.469 INFO:tasks.ceph.osd.3.smithi039.stderr:INFO 2023-01-24 14:42:34,465 [shard 0] monc - handle_auth_reply [0x614000008040 client.?(temp_mon_client) 172.21.15.39:0/102214@59033 >> mon.0 v2:172.21.15.2:3300/0] returns auth_reply(proto 2 0 (0) Success) v1: 0 2023-01-24T14:42:34.469 INFO:tasks.ceph.osd.3.smithi039.stderr:INFO 2023-01-24 14:42:34,465 [shard 0] monc - handle_auth_reply 2023-01-24T14:42:34.469 INFO:tasks.ceph.osd.3.smithi039.stderr:DEBUG 2023-01-24 14:42:34,465 [shard 0] monc - renew_tickets: don't need new tickets 2023-01-24T14:42:34.469 INFO:tasks.ceph.osd.3.smithi039.stderr:INFO 2023-01-24 14:42:34,465 [shard 0] monc - renew_rotating_keyring renewing rotating keys (they expired before 1674571324.4655683) 2023-01-24T14:42:34.470 INFO:tasks.ceph.osd.3.smithi039.stderr:INFO 2023-01-24 14:42:34,465 [shard 0] monc - renew_rotating_keyring called too often (last: 1674571354.4231515) 2023-01-24T14:42:34.470 INFO:tasks.ceph.osd.3.smithi039.stderr:INFO 2023-01-24 14:42:34,465 [shard 0] monc - do_auth_single: [0x614000008040 client.?(temp_mon_client) 172.21.15.39:0/102214@59033 >> mon.0 v2:172.21.15.2:3300/0] returns auth_reply(proto 2 0 (0) Success) v1: 0 2023-01-24T14:42:34.470 INFO:tasks.ceph.osd.3.smithi039.stderr:INFO 2023-01-24 14:42:34,466 [shard 0] monc - set_mon_vals no callback set 2023-01-24T14:42:34.471 INFO:tasks.ceph.osd.3.smithi039.stderr:DEBUG 2023-01-24 14:42:34,469 [shard 0] osd - operator(): got config from monitor, fsid 3808514d-c43d-4c1f-b296-499f8064f677 2023-01-24T14:42:34.471 INFO:tasks.ceph.osd.3.smithi039.stderr:INFO 2023-01-24 14:42:34,469 [shard 0] monc - stop 2023-01-24T14:42:34.471 INFO:tasks.ceph.osd.3.smithi039.stderr:INFO 2023-01-24 14:42:34,469 [shard 0] monc - close 2023-01-24T14:42:34.472 INFO:tasks.ceph.osd.3.smithi039.stderr:INFO 2023-01-24 14:42:34,469 [shard 0] ms - [0x614000008040 client.?(temp_mon_client) 172.21.15.39:0/102214@59033 >> mon.0 v2:172.21.15.2:3300/0] closing: reset no, replace no 2023-01-24T14:42:34.472 INFO:tasks.ceph.osd.3.smithi039.stderr:INFO 2023-01-24 14:42:34,469 [shard 0] ms - [0x614000008040 client.?(temp_mon_client) 172.21.15.39:0/102214@59033 >> mon.0 v2:172.21.15.2:3300/0] execute_ready(): protocol aborted at CLOSING -- std::system_error (error crimson::net:4, read eof) 2023-01-24T14:42:34.472 INFO:tasks.ceph.osd.3.smithi039.stderr:INFO 2023-01-24 14:42:34,470 [shard 0] osd - get_early_config: set --smp 1 2023-01-24T14:42:34.528 INFO:tasks.ceph.mgr.x.smithi039.stderr:2023-01-24T14:42:34.527+0000 7f78f7a7b1c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-01-24T14:42:34.779 INFO:tasks.ceph.mgr.x.smithi039.stderr:2023-01-24T14:42:34.778+0000 7f78f7a7b1c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-01-24T14:42:34.859 INFO:tasks.ceph.mgr.x.smithi039.stderr:2023-01-24T14:42:34.858+0000 7f78f7a7b1c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-01-24T14:42:35.011 INFO:tasks.ceph.mgr.x.smithi039.stderr:2023-01-24T14:42:35.010+0000 7f78f7a7b1c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-01-24T14:42:35.305 INFO:tasks.ceph.mgr.x.smithi039.stderr:2023-01-24T14:42:35.304+0000 7f78f7a7b1c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-01-24T14:42:35.413 INFO:tasks.ceph.osd.0.smithi002.stderr:WARNING: debug mode. Not for benchmarking or production 2023-01-24T14:42:35.414 INFO:tasks.ceph.osd.0.smithi002.stderr:INFO 2023-01-24 14:42:35,413 seastar - Reactor backend: epoll 2023-01-24T14:42:35.415 INFO:tasks.ceph.osd.0.smithi002.stderr:INFO 2023-01-24 14:42:35,415 [shard 0] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-01-24T14:42:35.416 INFO:tasks.ceph.osd.0.smithi002.stderr:INFO 2023-01-24 14:42:35,415 [shard 0] seastar - IO queue uses 0.75ms latency goal for device 0 2023-01-24T14:42:35.416 INFO:tasks.ceph.osd.0.smithi002.stderr:INFO 2023-01-24 14:42:35,415 [shard 0] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-01-24T14:42:35.416 INFO:tasks.ceph.osd.0.smithi002.stderr:INFO 2023-01-24 14:42:35,415 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-01-24T14:42:35.420 INFO:tasks.ceph.osd.1.smithi002.stderr:WARNING: debug mode. Not for benchmarking or production 2023-01-24T14:42:35.420 INFO:tasks.ceph.osd.1.smithi002.stderr:INFO 2023-01-24 14:42:35,417 seastar - Reactor backend: epoll 2023-01-24T14:42:35.421 INFO:tasks.ceph.osd.0.smithi002.stderr:==102589==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-01-24T14:42:35.424 INFO:tasks.ceph.osd.0.smithi002.stderr:==102589==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7fff3fb81000; bottom 0x7f952300d000; size: 0x006a1cb74000 (455748304896) 2023-01-24T14:42:35.425 INFO:tasks.ceph.osd.0.smithi002.stderr:False positive error reports may follow 2023-01-24T14:42:35.425 INFO:tasks.ceph.osd.0.smithi002.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-01-24T14:42:35.435 INFO:tasks.ceph.osd.1.smithi002.stderr:INFO 2023-01-24 14:42:35,434 [shard 0] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-01-24T14:42:35.435 INFO:tasks.ceph.osd.1.smithi002.stderr:INFO 2023-01-24 14:42:35,434 [shard 0] seastar - IO queue uses 0.75ms latency goal for device 0 2023-01-24T14:42:35.435 INFO:tasks.ceph.osd.1.smithi002.stderr:INFO 2023-01-24 14:42:35,434 [shard 0] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-01-24T14:42:35.435 INFO:tasks.ceph.osd.1.smithi002.stderr:INFO 2023-01-24 14:42:35,434 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-01-24T14:42:35.436 INFO:tasks.ceph.osd.1.smithi002.stderr:==102590==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-01-24T14:42:35.438 INFO:tasks.ceph.osd.2.smithi039.stderr:WARNING: debug mode. Not for benchmarking or production 2023-01-24T14:42:35.439 INFO:tasks.ceph.osd.2.smithi039.stderr:INFO 2023-01-24 14:42:35,439 seastar - Reactor backend: epoll 2023-01-24T14:42:35.442 INFO:tasks.ceph.osd.3.smithi039.stderr:WARNING: debug mode. Not for benchmarking or production 2023-01-24T14:42:35.443 INFO:tasks.ceph.osd.1.smithi002.stderr:==102590==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7ffceea99000; bottom 0x7fc612305000; size: 0x0036dc794000 (235627167744) 2023-01-24T14:42:35.443 INFO:tasks.ceph.osd.1.smithi002.stderr:False positive error reports may follow 2023-01-24T14:42:35.443 INFO:tasks.ceph.osd.1.smithi002.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-01-24T14:42:35.444 INFO:tasks.ceph.osd.3.smithi039.stderr:INFO 2023-01-24 14:42:35,442 seastar - Reactor backend: epoll 2023-01-24T14:42:35.491 INFO:tasks.ceph.osd.3.smithi039.stderr:INFO 2023-01-24 14:42:35,490 [shard 0] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-01-24T14:42:35.491 INFO:tasks.ceph.osd.3.smithi039.stderr:INFO 2023-01-24 14:42:35,490 [shard 0] seastar - IO queue uses 0.75ms latency goal for device 0 2023-01-24T14:42:35.491 INFO:tasks.ceph.osd.3.smithi039.stderr:INFO 2023-01-24 14:42:35,490 [shard 0] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-01-24T14:42:35.491 INFO:tasks.ceph.osd.3.smithi039.stderr:INFO 2023-01-24 14:42:35,490 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-01-24T14:42:35.492 INFO:tasks.ceph.osd.3.smithi039.stderr:==102208==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-01-24T14:42:35.499 INFO:tasks.ceph.osd.3.smithi039.stderr:==102208==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7ffc8c012000; bottom 0x7f5c639cb000; size: 0x00a028647000 (687872438272) 2023-01-24T14:42:35.499 INFO:tasks.ceph.osd.3.smithi039.stderr:False positive error reports may follow 2023-01-24T14:42:35.499 INFO:tasks.ceph.osd.3.smithi039.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-01-24T14:42:35.502 INFO:tasks.ceph.mgr.x.smithi039.stderr:2023-01-24T14:42:35.501+0000 7f78f7a7b1c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-01-24T14:42:35.510 INFO:tasks.ceph.osd.2.smithi039.stderr:INFO 2023-01-24 14:42:35,509 [shard 0] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-01-24T14:42:35.510 INFO:tasks.ceph.osd.2.smithi039.stderr:INFO 2023-01-24 14:42:35,509 [shard 0] seastar - IO queue uses 0.75ms latency goal for device 0 2023-01-24T14:42:35.510 INFO:tasks.ceph.osd.2.smithi039.stderr:INFO 2023-01-24 14:42:35,509 [shard 0] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-01-24T14:42:35.510 INFO:tasks.ceph.osd.2.smithi039.stderr:INFO 2023-01-24 14:42:35,509 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-01-24T14:42:35.511 INFO:tasks.ceph.osd.2.smithi039.stderr:==102207==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-01-24T14:42:35.518 INFO:tasks.ceph.osd.2.smithi039.stderr:==102207==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7ffce5d15000; bottom 0x7f2714989000; size: 0x00d5d138c000 (918338191360) 2023-01-24T14:42:35.518 INFO:tasks.ceph.osd.2.smithi039.stderr:False positive error reports may follow 2023-01-24T14:42:35.518 INFO:tasks.ceph.osd.2.smithi039.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-01-24T14:42:35.651 INFO:tasks.ceph.mgr.x.smithi039.stderr:2023-01-24T14:42:35.650+0000 7f78f7a7b1c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-01-24T14:42:35.773 INFO:tasks.ceph.mgr.x.smithi039.stderr:2023-01-24T14:42:35.772+0000 7f78f7a7b1c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-01-24T14:42:35.925 INFO:tasks.ceph.mgr.x.smithi039.stderr:2023-01-24T14:42:35.924+0000 7f78f7a7b1c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-01-24T14:42:36.008 INFO:tasks.ceph.mgr.x.smithi039.stderr:2023-01-24T14:42:36.007+0000 7f78f7a7b1c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-01-24T14:42:36.268 INFO:tasks.ceph.mgr.x.smithi039.stderr:2023-01-24T14:42:36.267+0000 7f78f7a7b1c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-01-24T14:42:36.349 INFO:tasks.ceph.mgr.x.smithi039.stderr:2023-01-24T14:42:36.348+0000 7f78f7a7b1c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-01-24T14:42:36.996 INFO:tasks.ceph.mgr.x.smithi039.stderr:2023-01-24T14:42:36.995+0000 7f78f7a7b1c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-01-24T14:42:37.260 INFO:tasks.ceph.mgr.x.smithi039.stderr:2023-01-24T14:42:37.259+0000 7f78f7a7b1c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-01-24T14:42:37.342 INFO:tasks.ceph.mgr.x.smithi039.stderr:2023-01-24T14:42:37.341+0000 7f78f7a7b1c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-01-24T14:42:37.428 INFO:tasks.ceph.mgr.x.smithi039.stderr:2023-01-24T14:42:37.427+0000 7f78f7a7b1c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-01-24T14:42:37.619 INFO:tasks.ceph.mgr.x.smithi039.stderr:2023-01-24T14:42:37.618+0000 7f78f7a7b1c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-01-24T14:42:37.695 INFO:tasks.ceph.mgr.x.smithi039.stderr:2023-01-24T14:42:37.694+0000 7f78f7a7b1c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-01-24T14:42:37.856 INFO:tasks.ceph.mgr.x.smithi039.stderr:2023-01-24T14:42:37.855+0000 7f78f7a7b1c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-01-24T14:42:38.867 INFO:tasks.ceph.mgr.x.smithi039.stderr:2023-01-24T14:42:38.866+0000 7f78b40c1700 -1 mgr.server handle_report got status from non-daemon mon.a 2023-01-24T14:42:40.567 DEBUG:teuthology.orchestra.run.smithi002:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-01-24T14:42:40.910 INFO:teuthology.misc.health.smithi002.stdout: 2023-01-24T14:42:40.910 INFO:teuthology.misc.health.smithi002.stdout:{"epoch":11,"fsid":"3808514d-c43d-4c1f-b296-499f8064f677","created":"2023-01-24T14:42:31.895770+0000","modified":"2023-01-24T14:42:40.325648+0000","last_up_change":"2023-01-24T14:42:38.317142+0000","last_in_change":"2023-01-24T14:42:33.243250+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-01-24T14:42:37.919219+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"5e1a8f93-714b-43f5-abb0-337ed2dff81e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6803","nonce":102589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":102589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":102589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":102589}]},"public_addr":"172.21.15.2:6803/102589","cluster_addr":"172.21.15.2:6802/102589","heartbeat_back_addr":"172.21.15.2:6804/102589","heartbeat_front_addr":"172.21.15.2:6806/102589","state":["exists","up"]},{"osd":1,"uuid":"ef33d692-a846-4999-8754-c88dc16a9e47","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6800","nonce":102590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6801","nonce":102590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6807","nonce":102590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6805","nonce":102590}]},"public_addr":"172.21.15.2:6800/102590","cluster_addr":"172.21.15.2:6801/102590","heartbeat_back_addr":"172.21.15.2:6807/102590","heartbeat_front_addr":"172.21.15.2:6805/102590","state":["exists","up"]},{"osd":2,"uuid":"b03e5360-6d49-4073-804f-90be8c39d8eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6803","nonce":102207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":102207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6807","nonce":102207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":102207}]},"public_addr":"172.21.15.39:6803/102207","cluster_addr":"172.21.15.39:6802/102207","heartbeat_back_addr":"172.21.15.39:6807/102207","heartbeat_front_addr":"172.21.15.39:6806/102207","state":["exists","up"]},{"osd":3,"uuid":"4a337027-3e3d-4d6a-914e-2d8b4ff89ddc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6801","nonce":102208}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":102208}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6805","nonce":102208}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":102208}]},"public_addr":"172.21.15.39:6801/102208","cluster_addr":"172.21.15.39:6800/102208","heartbeat_back_addr":"172.21.15.39:6805/102208","heartbeat_front_addr":"172.21.15.39:6804/102208","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-01-24T14:42:40.923 DEBUG:teuthology.misc:4 of 4 OSDs are up 2023-01-24T14:42:40.923 INFO:tasks.ceph:Creating RBD pool 2023-01-24T14:42:40.924 DEBUG:teuthology.orchestra.run.smithi002:> sudo ceph --cluster ceph osd pool create rbd 8 2023-01-24T14:42:42.337 INFO:teuthology.orchestra.run.smithi002.stderr:pool 'rbd' created 2023-01-24T14:42:42.351 DEBUG:teuthology.orchestra.run.smithi002:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2023-01-24T14:42:43.342 INFO:teuthology.orchestra.run.smithi002.stderr:enabled application 'rbd' on pool 'rbd' 2023-01-24T14:42:43.361 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2023-01-24T14:42:43.362 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2023-01-24T14:42:43.362 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2023-01-24T14:42:43.362 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2023-01-24T14:42:43.363 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-01-24T14:42:43.761 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-01-24T14:42:43.774 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":5,"active_gid":4101,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":101886},{"type":"v1","addr":"172.21.15.39:6809","nonce":101886}]},"active_addr":"172.21.15.39:6809/101886","active_change":"2023-01-24T14:42:37.857778+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":0,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.39:0","nonce":4081113551}]},{"addrvec":[{"type":"v2","addr":"172.21.15.39:0","nonce":937911793}]},{"addrvec":[{"type":"v2","addr":"172.21.15.39:0","nonce":1832251186}]},{"addrvec":[{"type":"v2","addr":"172.21.15.39:0","nonce":3533999300}]}]}} 2023-01-24T14:42:43.777 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2023-01-24T14:42:43.777 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2023-01-24T14:42:43.777 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-01-24T14:42:44.114 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-01-24T14:42:44.114 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":13,"fsid":"3808514d-c43d-4c1f-b296-499f8064f677","created":"2023-01-24T14:42:31.895770+0000","modified":"2023-01-24T14:42:43.334191+0000","last_up_change":"2023-01-24T14:42:38.317142+0000","last_in_change":"2023-01-24T14:42:33.243250+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-01-24T14:42:37.919219+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-01-24T14:42:41.363171+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"5e1a8f93-714b-43f5-abb0-337ed2dff81e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6803","nonce":102589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":102589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":102589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":102589}]},"public_addr":"172.21.15.2:6803/102589","cluster_addr":"172.21.15.2:6802/102589","heartbeat_back_addr":"172.21.15.2:6804/102589","heartbeat_front_addr":"172.21.15.2:6806/102589","state":["exists","up"]},{"osd":1,"uuid":"ef33d692-a846-4999-8754-c88dc16a9e47","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6800","nonce":102590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6801","nonce":102590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6807","nonce":102590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6805","nonce":102590}]},"public_addr":"172.21.15.2:6800/102590","cluster_addr":"172.21.15.2:6801/102590","heartbeat_back_addr":"172.21.15.2:6807/102590","heartbeat_front_addr":"172.21.15.2:6805/102590","state":["exists","up"]},{"osd":2,"uuid":"b03e5360-6d49-4073-804f-90be8c39d8eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6803","nonce":102207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":102207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6807","nonce":102207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":102207}]},"public_addr":"172.21.15.39:6803/102207","cluster_addr":"172.21.15.39:6802/102207","heartbeat_back_addr":"172.21.15.39:6807/102207","heartbeat_front_addr":"172.21.15.39:6806/102207","state":["exists","up"]},{"osd":3,"uuid":"4a337027-3e3d-4d6a-914e-2d8b4ff89ddc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6801","nonce":102208}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":102208}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6805","nonce":102208}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":102208}]},"public_addr":"172.21.15.39:6801/102208","cluster_addr":"172.21.15.39:6800/102208","heartbeat_back_addr":"172.21.15.39:6805/102208","heartbeat_front_addr":"172.21.15.39:6804/102208","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-01-24T14:42:44.128 INFO:tasks.ceph.ceph_manager.ceph:all up! 2023-01-24T14:42:44.128 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-01-24T14:42:44.461 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-01-24T14:42:44.462 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":14,"fsid":"3808514d-c43d-4c1f-b296-499f8064f677","created":"2023-01-24T14:42:31.895770+0000","modified":"2023-01-24T14:42:44.338304+0000","last_up_change":"2023-01-24T14:42:38.317142+0000","last_in_change":"2023-01-24T14:42:33.243250+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-01-24T14:42:37.919219+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-01-24T14:42:41.363171+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"5e1a8f93-714b-43f5-abb0-337ed2dff81e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6803","nonce":102589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":102589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":102589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":102589}]},"public_addr":"172.21.15.2:6803/102589","cluster_addr":"172.21.15.2:6802/102589","heartbeat_back_addr":"172.21.15.2:6804/102589","heartbeat_front_addr":"172.21.15.2:6806/102589","state":["exists","up"]},{"osd":1,"uuid":"ef33d692-a846-4999-8754-c88dc16a9e47","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6800","nonce":102590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6801","nonce":102590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6807","nonce":102590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6805","nonce":102590}]},"public_addr":"172.21.15.2:6800/102590","cluster_addr":"172.21.15.2:6801/102590","heartbeat_back_addr":"172.21.15.2:6807/102590","heartbeat_front_addr":"172.21.15.2:6805/102590","state":["exists","up"]},{"osd":2,"uuid":"b03e5360-6d49-4073-804f-90be8c39d8eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6803","nonce":102207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":102207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6807","nonce":102207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":102207}]},"public_addr":"172.21.15.39:6803/102207","cluster_addr":"172.21.15.39:6802/102207","heartbeat_back_addr":"172.21.15.39:6807/102207","heartbeat_front_addr":"172.21.15.39:6806/102207","state":["exists","up"]},{"osd":3,"uuid":"4a337027-3e3d-4d6a-914e-2d8b4ff89ddc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6801","nonce":102208}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":102208}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6805","nonce":102208}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":102208}]},"public_addr":"172.21.15.39:6801/102208","cluster_addr":"172.21.15.39:6800/102208","heartbeat_back_addr":"172.21.15.39:6805/102208","heartbeat_front_addr":"172.21.15.39:6804/102208","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-01-24T14:42:44.476 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2023-01-24T14:42:44.477 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2023-01-24T14:42:44.477 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2023-01-24T14:42:44.478 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2023-01-24T14:42:44.670 INFO:teuthology.orchestra.run.smithi002.stdout:34359738375 2023-01-24T14:42:44.671 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-01-24T14:42:44.672 INFO:teuthology.orchestra.run.smithi002.stdout:34359738375 2023-01-24T14:42:44.672 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-01-24T14:42:44.673 INFO:teuthology.orchestra.run.smithi002.stdout:38654705670 2023-01-24T14:42:44.674 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-01-24T14:42:44.677 INFO:teuthology.orchestra.run.smithi002.stdout:38654705670 2023-01-24T14:42:44.677 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-01-24T14:42:45.075 INFO:teuthology.orchestra.run.smithi002.stdout:38654705670 2023-01-24T14:42:45.083 INFO:teuthology.orchestra.run.smithi002.stdout:34359738375 2023-01-24T14:42:45.091 INFO:teuthology.orchestra.run.smithi002.stdout:34359738375 2023-01-24T14:42:45.094 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 38654705670 for osd.3 2023-01-24T14:42:45.094 DEBUG:teuthology.parallel:result is None 2023-01-24T14:42:45.102 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738375 got 34359738375 for osd.0 2023-01-24T14:42:45.103 DEBUG:teuthology.parallel:result is None 2023-01-24T14:42:45.107 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738375 got 34359738375 for osd.1 2023-01-24T14:42:45.107 DEBUG:teuthology.parallel:result is None 2023-01-24T14:42:45.109 INFO:teuthology.orchestra.run.smithi002.stdout:38654705670 2023-01-24T14:42:45.124 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 38654705670 for osd.2 2023-01-24T14:42:45.124 DEBUG:teuthology.parallel:result is None 2023-01-24T14:42:45.125 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-01-24T14:42:45.125 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-01-24T14:42:45.495 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-01-24T14:42:45.497 INFO:teuthology.orchestra.run.smithi002.stderr:dumped all 2023-01-24T14:42:45.509 INFO:teuthology.orchestra.run.smithi002.stdout:{"pg_ready":true,"pg_map":{"version":9,"stamp":"2023-01-24T14:42:44.660421+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":39,"num_read_kb":29,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":419430400,"kb_used":22020,"kb_used_data":1384,"kb_used_omap":0,"kb_used_meta":20288,"kb_avail":419408380,"statfs":{"total":429496729600,"available":429474181120,"internally_reserved":0,"allocated":1417216,"data_stored":1262812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":20774912},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":8,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-01-24T14:42:43.462691+0000","last_change":"2023-01-24T14:42:43.463202+0000","last_active":"2023-01-24T14:42:43.462691+0000","last_peered":"2023-01-24T14:42:43.462691+0000","last_clean":"2023-01-24T14:42:42.330702+0000","last_became_active":"2023-01-24T14:42:43.459539+0000","last_became_peered":"2023-01-24T14:42:43.459539+0000","last_unstale":"2023-01-24T14:42:43.462691+0000","last_undegraded":"2023-01-24T14:42:43.462691+0000","last_fullsized":"2023-01-24T14:42:43.462691+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T14:42:42.330702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T14:42:42.330702+0000","last_clean_scrub_stamp":"2023-01-24T14:42:42.330702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-01-24T14:42:43.425383+0000","last_change":"2023-01-24T14:42:43.425383+0000","last_active":"2023-01-24T14:42:43.425383+0000","last_peered":"2023-01-24T14:42:43.425383+0000","last_clean":"2023-01-24T14:42:43.425383+0000","last_became_active":"2023-01-24T14:42:43.422741+0000","last_became_peered":"2023-01-24T14:42:43.422741+0000","last_unstale":"2023-01-24T14:42:43.425383+0000","last_undegraded":"2023-01-24T14:42:43.425383+0000","last_fullsized":"2023-01-24T14:42:43.425383+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T14:42:42.330702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T14:42:42.330702+0000","last_clean_scrub_stamp":"2023-01-24T14:42:42.330702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":8,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-01-24T14:42:43.439558+0000","last_change":"2023-01-24T14:42:43.439907+0000","last_active":"2023-01-24T14:42:43.439558+0000","last_peered":"2023-01-24T14:42:43.439558+0000","last_clean":"2023-01-24T14:42:42.330702+0000","last_became_active":"2023-01-24T14:42:43.436779+0000","last_became_peered":"2023-01-24T14:42:43.436779+0000","last_unstale":"2023-01-24T14:42:43.439558+0000","last_undegraded":"2023-01-24T14:42:43.439558+0000","last_fullsized":"2023-01-24T14:42:43.439558+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T14:42:42.330702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T14:42:42.330702+0000","last_clean_scrub_stamp":"2023-01-24T14:42:42.330702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-01-24T14:42:43.425664+0000","last_change":"2023-01-24T14:42:43.426068+0000","last_active":"2023-01-24T14:42:43.425664+0000","last_peered":"2023-01-24T14:42:43.425664+0000","last_clean":"2023-01-24T14:42:42.330702+0000","last_became_active":"2023-01-24T14:42:43.422382+0000","last_became_peered":"2023-01-24T14:42:43.422382+0000","last_unstale":"2023-01-24T14:42:43.425664+0000","last_undegraded":"2023-01-24T14:42:43.425664+0000","last_fullsized":"2023-01-24T14:42:43.425664+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T14:42:42.330702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T14:42:42.330702+0000","last_clean_scrub_stamp":"2023-01-24T14:42:42.330702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":108,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-01-24T14:42:42.355562+0000","last_change":"2023-01-24T14:42:39.395501+0000","last_active":"2023-01-24T14:42:42.355562+0000","last_peered":"2023-01-24T14:42:42.355562+0000","last_clean":"2023-01-24T14:42:42.355562+0000","last_became_active":"2023-01-24T14:42:39.392286+0000","last_became_peered":"2023-01-24T14:42:39.392286+0000","last_unstale":"2023-01-24T14:42:42.355562+0000","last_undegraded":"2023-01-24T14:42:42.355562+0000","last_fullsized":"2023-01-24T14:42:42.355562+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T14:42:38.317142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T14:42:38.317142+0000","last_clean_scrub_stamp":"2023-01-24T14:42:38.317142+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":39,"num_read_kb":29,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":8,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-01-24T14:42:43.472479+0000","last_change":"2023-01-24T14:42:43.472889+0000","last_active":"2023-01-24T14:42:43.472479+0000","last_peered":"2023-01-24T14:42:43.472479+0000","last_clean":"2023-01-24T14:42:42.330702+0000","last_became_active":"2023-01-24T14:42:43.469425+0000","last_became_peered":"2023-01-24T14:42:43.469425+0000","last_unstale":"2023-01-24T14:42:43.472479+0000","last_undegraded":"2023-01-24T14:42:43.472479+0000","last_fullsized":"2023-01-24T14:42:43.472479+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T14:42:42.330702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T14:42:42.330702+0000","last_clean_scrub_stamp":"2023-01-24T14:42:42.330702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":8,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-01-24T14:42:43.411924+0000","last_change":"2023-01-24T14:42:43.412373+0000","last_active":"2023-01-24T14:42:43.411924+0000","last_peered":"2023-01-24T14:42:43.411924+0000","last_clean":"2023-01-24T14:42:42.330702+0000","last_became_active":"2023-01-24T14:42:43.409737+0000","last_became_peered":"2023-01-24T14:42:43.409737+0000","last_unstale":"2023-01-24T14:42:43.411924+0000","last_undegraded":"2023-01-24T14:42:43.411924+0000","last_fullsized":"2023-01-24T14:42:43.411924+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T14:42:42.330702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T14:42:42.330702+0000","last_clean_scrub_stamp":"2023-01-24T14:42:42.330702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-01-24T14:42:43.430910+0000","last_change":"2023-01-24T14:42:43.431336+0000","last_active":"2023-01-24T14:42:43.430910+0000","last_peered":"2023-01-24T14:42:43.430910+0000","last_clean":"2023-01-24T14:42:42.330702+0000","last_became_active":"2023-01-24T14:42:43.428361+0000","last_became_peered":"2023-01-24T14:42:43.428361+0000","last_unstale":"2023-01-24T14:42:43.430910+0000","last_undegraded":"2023-01-24T14:42:43.430910+0000","last_fullsized":"2023-01-24T14:42:43.430910+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T14:42:42.330702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T14:42:42.330702+0000","last_clean_scrub_stamp":"2023-01-24T14:42:42.330702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-01-24T14:42:43.400716+0000","last_change":"2023-01-24T14:42:43.400716+0000","last_active":"2023-01-24T14:42:43.400716+0000","last_peered":"2023-01-24T14:42:43.400716+0000","last_clean":"2023-01-24T14:42:43.400716+0000","last_became_active":"2023-01-24T14:42:43.396186+0000","last_became_peered":"2023-01-24T14:42:43.396186+0000","last_unstale":"2023-01-24T14:42:43.400716+0000","last_undegraded":"2023-01-24T14:42:43.400716+0000","last_fullsized":"2023-01-24T14:42:43.400716+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T14:42:42.330702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T14:42:42.330702+0000","last_clean_scrub_stamp":"2023-01-24T14:42:42.330702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":39,"num_read_kb":29,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705670,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":6080,"kb_used_data":636,"kb_used_omap":0,"kb_used_meta":5248,"kb_avail":104851520,"statfs":{"total":107374182400,"available":107367956480,"internally_reserved":0,"allocated":651264,"data_stored":610887,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5373952},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":9,"seq":38654705670,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":4928,"kb_used_data":56,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":104852672,"statfs":{"total":107374182400,"available":107369136128,"internally_reserved":0,"allocated":57344,"data_stored":20519,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":8,"seq":34359738375,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":6084,"kb_used_data":636,"kb_used_omap":0,"kb_used_meta":5312,"kb_avail":104851516,"statfs":{"total":107374182400,"available":107367952384,"internally_reserved":0,"allocated":651264,"data_stored":610887,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5439488},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":8,"seq":34359738375,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":4928,"kb_used_data":56,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":104852672,"statfs":{"total":107374182400,"available":107369136128,"internally_reserved":0,"allocated":57344,"data_stored":20519,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2023-01-24T14:42:45.510 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-01-24T14:42:45.842 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-01-24T14:42:45.844 INFO:teuthology.orchestra.run.smithi002.stderr:dumped all 2023-01-24T14:42:45.856 INFO:teuthology.orchestra.run.smithi002.stdout:{"pg_ready":true,"pg_map":{"version":9,"stamp":"2023-01-24T14:42:44.660421+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":39,"num_read_kb":29,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":419430400,"kb_used":22020,"kb_used_data":1384,"kb_used_omap":0,"kb_used_meta":20288,"kb_avail":419408380,"statfs":{"total":429496729600,"available":429474181120,"internally_reserved":0,"allocated":1417216,"data_stored":1262812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":20774912},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":8,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-01-24T14:42:43.462691+0000","last_change":"2023-01-24T14:42:43.463202+0000","last_active":"2023-01-24T14:42:43.462691+0000","last_peered":"2023-01-24T14:42:43.462691+0000","last_clean":"2023-01-24T14:42:42.330702+0000","last_became_active":"2023-01-24T14:42:43.459539+0000","last_became_peered":"2023-01-24T14:42:43.459539+0000","last_unstale":"2023-01-24T14:42:43.462691+0000","last_undegraded":"2023-01-24T14:42:43.462691+0000","last_fullsized":"2023-01-24T14:42:43.462691+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T14:42:42.330702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T14:42:42.330702+0000","last_clean_scrub_stamp":"2023-01-24T14:42:42.330702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-01-24T14:42:43.425383+0000","last_change":"2023-01-24T14:42:43.425383+0000","last_active":"2023-01-24T14:42:43.425383+0000","last_peered":"2023-01-24T14:42:43.425383+0000","last_clean":"2023-01-24T14:42:43.425383+0000","last_became_active":"2023-01-24T14:42:43.422741+0000","last_became_peered":"2023-01-24T14:42:43.422741+0000","last_unstale":"2023-01-24T14:42:43.425383+0000","last_undegraded":"2023-01-24T14:42:43.425383+0000","last_fullsized":"2023-01-24T14:42:43.425383+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T14:42:42.330702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T14:42:42.330702+0000","last_clean_scrub_stamp":"2023-01-24T14:42:42.330702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":8,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-01-24T14:42:43.439558+0000","last_change":"2023-01-24T14:42:43.439907+0000","last_active":"2023-01-24T14:42:43.439558+0000","last_peered":"2023-01-24T14:42:43.439558+0000","last_clean":"2023-01-24T14:42:42.330702+0000","last_became_active":"2023-01-24T14:42:43.436779+0000","last_became_peered":"2023-01-24T14:42:43.436779+0000","last_unstale":"2023-01-24T14:42:43.439558+0000","last_undegraded":"2023-01-24T14:42:43.439558+0000","last_fullsized":"2023-01-24T14:42:43.439558+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T14:42:42.330702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T14:42:42.330702+0000","last_clean_scrub_stamp":"2023-01-24T14:42:42.330702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-01-24T14:42:43.425664+0000","last_change":"2023-01-24T14:42:43.426068+0000","last_active":"2023-01-24T14:42:43.425664+0000","last_peered":"2023-01-24T14:42:43.425664+0000","last_clean":"2023-01-24T14:42:42.330702+0000","last_became_active":"2023-01-24T14:42:43.422382+0000","last_became_peered":"2023-01-24T14:42:43.422382+0000","last_unstale":"2023-01-24T14:42:43.425664+0000","last_undegraded":"2023-01-24T14:42:43.425664+0000","last_fullsized":"2023-01-24T14:42:43.425664+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T14:42:42.330702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T14:42:42.330702+0000","last_clean_scrub_stamp":"2023-01-24T14:42:42.330702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":108,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-01-24T14:42:42.355562+0000","last_change":"2023-01-24T14:42:39.395501+0000","last_active":"2023-01-24T14:42:42.355562+0000","last_peered":"2023-01-24T14:42:42.355562+0000","last_clean":"2023-01-24T14:42:42.355562+0000","last_became_active":"2023-01-24T14:42:39.392286+0000","last_became_peered":"2023-01-24T14:42:39.392286+0000","last_unstale":"2023-01-24T14:42:42.355562+0000","last_undegraded":"2023-01-24T14:42:42.355562+0000","last_fullsized":"2023-01-24T14:42:42.355562+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T14:42:38.317142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T14:42:38.317142+0000","last_clean_scrub_stamp":"2023-01-24T14:42:38.317142+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":39,"num_read_kb":29,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":8,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-01-24T14:42:43.472479+0000","last_change":"2023-01-24T14:42:43.472889+0000","last_active":"2023-01-24T14:42:43.472479+0000","last_peered":"2023-01-24T14:42:43.472479+0000","last_clean":"2023-01-24T14:42:42.330702+0000","last_became_active":"2023-01-24T14:42:43.469425+0000","last_became_peered":"2023-01-24T14:42:43.469425+0000","last_unstale":"2023-01-24T14:42:43.472479+0000","last_undegraded":"2023-01-24T14:42:43.472479+0000","last_fullsized":"2023-01-24T14:42:43.472479+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T14:42:42.330702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T14:42:42.330702+0000","last_clean_scrub_stamp":"2023-01-24T14:42:42.330702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":8,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-01-24T14:42:43.411924+0000","last_change":"2023-01-24T14:42:43.412373+0000","last_active":"2023-01-24T14:42:43.411924+0000","last_peered":"2023-01-24T14:42:43.411924+0000","last_clean":"2023-01-24T14:42:42.330702+0000","last_became_active":"2023-01-24T14:42:43.409737+0000","last_became_peered":"2023-01-24T14:42:43.409737+0000","last_unstale":"2023-01-24T14:42:43.411924+0000","last_undegraded":"2023-01-24T14:42:43.411924+0000","last_fullsized":"2023-01-24T14:42:43.411924+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T14:42:42.330702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T14:42:42.330702+0000","last_clean_scrub_stamp":"2023-01-24T14:42:42.330702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-01-24T14:42:43.430910+0000","last_change":"2023-01-24T14:42:43.431336+0000","last_active":"2023-01-24T14:42:43.430910+0000","last_peered":"2023-01-24T14:42:43.430910+0000","last_clean":"2023-01-24T14:42:42.330702+0000","last_became_active":"2023-01-24T14:42:43.428361+0000","last_became_peered":"2023-01-24T14:42:43.428361+0000","last_unstale":"2023-01-24T14:42:43.430910+0000","last_undegraded":"2023-01-24T14:42:43.430910+0000","last_fullsized":"2023-01-24T14:42:43.430910+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T14:42:42.330702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T14:42:42.330702+0000","last_clean_scrub_stamp":"2023-01-24T14:42:42.330702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-01-24T14:42:43.400716+0000","last_change":"2023-01-24T14:42:43.400716+0000","last_active":"2023-01-24T14:42:43.400716+0000","last_peered":"2023-01-24T14:42:43.400716+0000","last_clean":"2023-01-24T14:42:43.400716+0000","last_became_active":"2023-01-24T14:42:43.396186+0000","last_became_peered":"2023-01-24T14:42:43.396186+0000","last_unstale":"2023-01-24T14:42:43.400716+0000","last_undegraded":"2023-01-24T14:42:43.400716+0000","last_fullsized":"2023-01-24T14:42:43.400716+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T14:42:42.330702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T14:42:42.330702+0000","last_clean_scrub_stamp":"2023-01-24T14:42:42.330702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":39,"num_read_kb":29,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705670,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":6080,"kb_used_data":636,"kb_used_omap":0,"kb_used_meta":5248,"kb_avail":104851520,"statfs":{"total":107374182400,"available":107367956480,"internally_reserved":0,"allocated":651264,"data_stored":610887,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5373952},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":9,"seq":38654705670,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":4928,"kb_used_data":56,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":104852672,"statfs":{"total":107374182400,"available":107369136128,"internally_reserved":0,"allocated":57344,"data_stored":20519,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":8,"seq":34359738375,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":6084,"kb_used_data":636,"kb_used_omap":0,"kb_used_meta":5312,"kb_avail":104851516,"statfs":{"total":107374182400,"available":107367952384,"internally_reserved":0,"allocated":651264,"data_stored":610887,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5439488},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":8,"seq":34359738375,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":4928,"kb_used_data":56,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":104852672,"statfs":{"total":107374182400,"available":107369136128,"internally_reserved":0,"allocated":57344,"data_stored":20519,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2023-01-24T14:42:45.857 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-01-24T14:42:45.857 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2023-01-24T14:42:45.858 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2023-01-24T14:42:45.858 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-01-24T14:42:46.234 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-01-24T14:42:46.234 INFO:teuthology.orchestra.run.smithi002.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-01-24T14:42:46.248 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2023-01-24T14:42:46.248 INFO:teuthology.run_tasks:Running task workunit... 2023-01-24T14:42:46.260 INFO:tasks.workunit:Pulling workunits from ref 510284b66513490445619d1430aa869868c71a09 2023-01-24T14:42:46.261 INFO:tasks.workunit:Making a separate scratch dir for every client... 2023-01-24T14:42:46.261 DEBUG:teuthology.orchestra.run.smithi002:> stat -- /home/ubuntu/cephtest/mnt.0 2023-01-24T14:42:46.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-01-24T14:42:46.280 INFO:teuthology.orchestra.run.smithi002.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2023-01-24T14:42:46.280 DEBUG:teuthology.orchestra.run.smithi002:> mkdir -- /home/ubuntu/cephtest/mnt.0 2023-01-24T14:42:46.338 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2023-01-24T14:42:46.339 DEBUG:teuthology.orchestra.run.smithi002:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2023-01-24T14:42:46.397 INFO:tasks.workunit:timeout=1h 2023-01-24T14:42:46.397 INFO:tasks.workunit:cleanup=True 2023-01-24T14:42:46.398 DEBUG:teuthology.orchestra.run.smithi002:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 510284b66513490445619d1430aa869868c71a09 2023-01-24T14:42:46.456 INFO:tasks.workunit.client.0.smithi002.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2023-01-24T14:43:33.681 INFO:tasks.workunit.client.0.smithi002.stderr:Updating files: 98% (11450/11669) Updating files: 99% (11553/11669) Updating files: 100% (11669/11669) Updating files: 100% (11669/11669), done. 2023-01-24T14:43:34.188 INFO:tasks.workunit.client.0.smithi002.stderr:Note: switching to '510284b66513490445619d1430aa869868c71a09'. 2023-01-24T14:43:34.189 INFO:tasks.workunit.client.0.smithi002.stderr: 2023-01-24T14:43:34.189 INFO:tasks.workunit.client.0.smithi002.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2023-01-24T14:43:34.189 INFO:tasks.workunit.client.0.smithi002.stderr:changes and commit them, and you can discard any commits you make in this 2023-01-24T14:43:34.189 INFO:tasks.workunit.client.0.smithi002.stderr:state without impacting any branches by switching back to a branch. 2023-01-24T14:43:34.189 INFO:tasks.workunit.client.0.smithi002.stderr: 2023-01-24T14:43:34.190 INFO:tasks.workunit.client.0.smithi002.stderr:If you want to create a new branch to retain commits you create, you may 2023-01-24T14:43:34.190 INFO:tasks.workunit.client.0.smithi002.stderr:do so (now or later) by using -c with the switch command. Example: 2023-01-24T14:43:34.190 INFO:tasks.workunit.client.0.smithi002.stderr: 2023-01-24T14:43:34.190 INFO:tasks.workunit.client.0.smithi002.stderr: git switch -c 2023-01-24T14:43:34.190 INFO:tasks.workunit.client.0.smithi002.stderr: 2023-01-24T14:43:34.191 INFO:tasks.workunit.client.0.smithi002.stderr:Or undo this operation with: 2023-01-24T14:43:34.191 INFO:tasks.workunit.client.0.smithi002.stderr: 2023-01-24T14:43:34.191 INFO:tasks.workunit.client.0.smithi002.stderr: git switch - 2023-01-24T14:43:34.191 INFO:tasks.workunit.client.0.smithi002.stderr: 2023-01-24T14:43:34.191 INFO:tasks.workunit.client.0.smithi002.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2023-01-24T14:43:34.192 INFO:tasks.workunit.client.0.smithi002.stderr: 2023-01-24T14:43:34.192 INFO:tasks.workunit.client.0.smithi002.stderr:HEAD is now at 510284b6651 Merge pull request #49791 from idryomov/wip-doc-exclusive-lock-transitions 2023-01-24T14:43:34.192 DEBUG:teuthology.orchestra.run.smithi002:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2023-01-24T14:43:34.257 INFO:tasks.workunit.client.0.smithi002.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2023-01-24T14:43:34.257 INFO:tasks.workunit.client.0.smithi002.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2023-01-24T14:43:34.257 INFO:tasks.workunit.client.0.smithi002.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2023-01-24T14:43:34.800 INFO:tasks.workunit.client.0.smithi002.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2023-01-24T14:43:34.870 INFO:tasks.workunit.client.0.smithi002.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2023-01-24T14:43:34.911 INFO:tasks.workunit.client.0.smithi002.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2023-01-24T14:43:34.913 INFO:tasks.workunit.client.0.smithi002.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2023-01-24T14:43:34.913 INFO:tasks.workunit.client.0.smithi002.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2023-01-24T14:43:34.951 INFO:tasks.workunit.client.0.smithi002.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2023-01-24T14:43:34.954 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2023-01-24T14:43:34.955 DEBUG:teuthology.orchestra.run.smithi002:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2023-01-24T14:43:35.013 INFO:tasks.workunit:Running workunits matching rados/test_python.sh on client.0... 2023-01-24T14:43:35.015 INFO:tasks.workunit:Running workunit rados/test_python.sh... 2023-01-24T14:43:35.015 DEBUG:teuthology.orchestra.run.smithi002:workunit test rados/test_python.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=510284b66513490445619d1430aa869868c71a09 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 CRIMSON_COMPAT=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 1h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test_python.sh --eval-attr 'not (wait or tier or ec or bench or stats)' 2023-01-24T14:43:35.076 INFO:tasks.workunit.client.0.smithi002.stderr:+ ceph osd pool create rbd 2023-01-24T14:43:35.517 INFO:tasks.workunit.client.0.smithi002.stderr:pool 'rbd' already exists 2023-01-24T14:43:35.527 INFO:tasks.workunit.client.0.smithi002.stderr:++ dirname /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test_python.sh 2023-01-24T14:43:35.528 INFO:tasks.workunit.client.0.smithi002.stderr:+ python3 -m nose -v /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/../../../src/test/pybind/test_rados.py --eval-attr 'not (wait or tier or ec or bench or stats)' 2023-01-24T14:43:35.932 INFO:tasks.workunit.client.0.smithi002.stderr:test_rados.TestCommand.test_ceph_osd_pool_create_utf8 ... ok 2023-01-24T14:43:35.953 INFO:tasks.workunit.client.0.smithi002.stderr:test_rados.TestCommand.test_monmap_dump ... ok 2023-01-24T14:43:43.958 INFO:tasks.workunit.client.0.smithi002.stderr:test_rados.TestIoCtxSelfManagedSnaps.test ... ok 2023-01-24T14:43:47.976 INFO:tasks.workunit.client.0.smithi002.stderr:test_rados.TestIoctx.test_aio_append ... ok 2023-01-24T14:43:51.992 INFO:tasks.workunit.client.0.smithi002.stderr:test_rados.TestIoctx.test_aio_cmpext ... ok 2023-01-24T14:43:56.007 INFO:tasks.workunit.client.0.smithi002.stderr:test_rados.TestIoctx.test_aio_execute ... ok 2023-01-24T14:44:00.021 INFO:tasks.workunit.client.0.smithi002.stderr:test_rados.TestIoctx.test_aio_remove ... ok 2023-01-24T14:44:04.042 INFO:tasks.workunit.client.0.smithi002.stderr:test_rados.TestIoctx.test_aio_rmxattr ... ok 2023-01-24T14:44:08.054 INFO:tasks.workunit.client.0.smithi002.stderr:test_rados.TestIoctx.test_aio_setxattr ... ok 2023-01-24T14:44:12.068 INFO:tasks.workunit.client.0.smithi002.stderr:test_rados.TestIoctx.test_aio_stat ... ok 2023-01-24T14:44:16.083 INFO:tasks.workunit.client.0.smithi002.stderr:test_rados.TestIoctx.test_aio_write ... ok 2023-01-24T14:44:20.099 INFO:tasks.workunit.client.0.smithi002.stderr:test_rados.TestIoctx.test_aio_write_full ... ok 2023-01-24T14:44:24.112 INFO:tasks.workunit.client.0.smithi002.stderr:test_rados.TestIoctx.test_aio_write_no_comp_ref ... ok 2023-01-24T14:44:29.134 INFO:tasks.workunit.client.0.smithi002.stderr:test_rados.TestIoctx.test_aio_writesame ... ok 2023-01-24T14:44:31.882 INFO:tasks.workunit.client.0.smithi002.stderr:test_rados.TestIoctx.test_alignment ... ok 2023-01-24T14:44:36.883 INFO:tasks.workunit.client.0.smithi002.stderr:test_rados.TestIoctx.test_append ... ok 2023-01-24T14:44:45.923 INFO:tasks.workunit.client.0.smithi002.stderr:test_rados.TestIoctx.test_applications ... ok 2023-01-24T14:44:53.943 INFO:tasks.workunit.client.0.smithi002.stderr:test_rados.TestIoctx.test_clear_omap ... ok 2023-01-24T14:45:00.967 INFO:tasks.workunit.client.0.smithi002.stderr:test_rados.TestIoctx.test_cmpext ... ok 2023-01-24T14:45:06.888 INFO:tasks.workunit.client.0.smithi002.stderr:test_rados.TestIoctx.test_cmpext_op ... ok 2023-01-24T14:45:11.904 INFO:tasks.workunit.client.0.smithi002.stderr:test_rados.TestIoctx.test_create_snap ... ok 2023-01-24T14:45:17.927 INFO:tasks.workunit.client.0.smithi002.stderr:test_rados.TestIoctx.test_execute ... ok 2023-01-24T14:45:27.960 INFO:tasks.workunit.client.0.smithi002.stderr:test_rados.TestIoctx.test_execute_op ... ok 2023-01-24T14:45:30.968 INFO:tasks.workunit.client.0.smithi002.stderr:test_rados.TestIoctx.test_get_last_version ... ok 2023-01-24T14:45:36.982 INFO:tasks.workunit.client.0.smithi002.stderr:test_rados.TestIoctx.test_get_omap_keys ... ok 2023-01-24T14:45:42.901 INFO:tasks.workunit.client.0.smithi002.stderr:test_rados.TestIoctx.test_get_omap_vals_by_keys ... ok 2023-01-24T14:45:45.907 INFO:tasks.workunit.client.0.smithi002.stderr:test_rados.TestIoctx.test_get_pool_id ... ok 2023-01-24T14:45:48.918 INFO:tasks.workunit.client.0.smithi002.stderr:test_rados.TestIoctx.test_get_pool_name ... ok 2023-01-24T14:45:51.897 INFO:tasks.workunit.client.0.smithi002.stderr:test_rados.TestIoctx.test_get_stats ... ok 2023-01-24T14:46:11.037 INFO:tasks.workunit.client.0.smithi002.stderr:test_rados.TestIoctx.test_list_ns_objects ... ok 2023-01-24T14:46:16.903 INFO:tasks.workunit.client.0.smithi002.stderr:test_rados.TestIoctx.test_list_objects ... ok 2023-01-24T14:46:32.989 INFO:tasks.workunit.client.0.smithi002.stderr:test_rados.TestIoctx.test_list_objects_empty ... ok 2023-01-24T14:46:38.913 INFO:tasks.workunit.client.0.smithi002.stderr:test_rados.TestIoctx.test_list_snaps ... ok 2023-01-24T14:46:41.921 INFO:tasks.workunit.client.0.smithi002.stderr:test_rados.TestIoctx.test_list_snaps_empty ... ok 2023-01-24T14:46:49.939 INFO:tasks.workunit.client.0.smithi002.stderr:test_rados.TestIoctx.test_locator ... ok 2023-01-24T14:47:05.979 INFO:tasks.workunit.client.0.smithi002.stderr:test_rados.TestIoctx.test_lock ... ok 2023-01-24T14:47:09.924 INFO:tasks.workunit.client.0.smithi002.stderr:test_rados.TestIoctx.test_lookup_snap ... ok 2023-01-24T14:47:17.946 INFO:tasks.workunit.client.0.smithi002.stderr:test_rados.TestIoctx.test_obj_xattrs ... ok 2023-01-24T14:47:25.974 INFO:tasks.workunit.client.0.smithi002.stderr:test_rados.TestIoctx.test_omap_cmp ... ok 2023-01-24T14:47:34.927 INFO:tasks.workunit.client.0.smithi002.stderr:test_rados.TestIoctx.test_operate_aio_write_op ... ok 2023-01-24T14:47:52.972 INFO:tasks.workunit.client.0.smithi002.stderr:test_rados.TestIoctx.test_remove_omap_range2 ... ok 2023-01-24T14:47:57.927 INFO:tasks.workunit.client.0.smithi002.stderr:test_rados.TestIoctx.test_remove_snap ... ok 2023-01-24T14:48:00.935 INFO:tasks.workunit.client.0.smithi002.stderr:test_rados.TestIoctx.test_service_daemon ... ok 2023-01-24T14:48:23.935 INFO:tasks.workunit.client.0.smithi002.stderr:test_rados.TestIoctx.test_set_omap ... ok 2023-01-24T14:48:37.932 INFO:tasks.workunit.client.0.smithi002.stderr:test_rados.TestIoctx.test_set_omap_aio ... ok 2023-01-24T14:49:11.942 INFO:tasks.workunit.client.0.smithi002.stderr:test_rados.TestIoctx.test_snap_read ... ok 2023-01-24T14:49:34.954 INFO:tasks.workunit.client.0.smithi002.stderr:test_rados.TestIoctx.test_snap_rollback ... ok 2023-01-24T14:49:49.985 INFO:tasks.workunit.client.0.smithi002.stderr:test_rados.TestIoctx.test_snap_rollback_removed ... ok 2023-01-24T14:49:54.002 INFO:tasks.workunit.client.0.smithi002.stderr:test_rados.TestIoctx.test_snap_timestamp ... ok 2023-01-24T14:50:11.032 INFO:tasks.workunit.client.0.smithi002.stderr:test_rados.TestIoctx.test_trunc ... ok 2023-01-24T14:50:37.956 INFO:tasks.workunit.client.0.smithi002.stderr:test_rados.TestIoctx.test_write ... ok 2023-01-24T14:51:13.971 INFO:tasks.workunit.client.0.smithi002.stderr:test_rados.TestIoctx.test_write_full ... ok 2023-01-24T14:51:30.006 INFO:tasks.workunit.client.0.smithi002.stderr:test_rados.TestIoctx.test_write_ops ... ok 2023-01-24T14:51:45.980 INFO:tasks.workunit.client.0.smithi002.stderr:test_rados.TestIoctx.test_write_zeros ... ok 2023-01-24T14:52:12.975 INFO:tasks.workunit.client.0.smithi002.stderr:test_rados.TestIoctx.test_writesame ... ok 2023-01-24T14:52:29.005 INFO:tasks.workunit.client.0.smithi002.stderr:test_rados.TestIoctx.test_writesame_op ... ok 2023-01-24T14:52:42.025 INFO:tasks.ceph.osd.1.smithi002.stderr:Reactor stalled for 53 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x375fa433 0x3760be55 0x3760c8a4 0x378ccc9e 0x378cd474 0x3687c5ba 0x3687f128 0x4449fbfa 0x444a049a 0x444a0a55 0x4449f7fa 0x3671ebc7 0x378e7f29 0x364b52af 0x364b571b 0x362c6842 0x362dcdef 0x362dda11 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T14:52:42.026 INFO:tasks.ceph.osd.1.smithi002.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff81b1f69e 0xffffffff81b21b06 0xffffffff823f08e9 0xffffffff823f0ed5 0xffffffff81b2604a 0xffffffff81a043ab 0xffffffff824000a9 2023-01-24T14:52:42.026 INFO:tasks.ceph.osd.1.smithi002.stderr:kernel callstack: 2023-01-24T14:52:42.026 INFO:tasks.ceph.osd.1.smithi002.stderr:kernel callstack: 2023-01-24T14:52:42.027 INFO:tasks.ceph.osd.1.smithi002.stderr:kernel callstack: 2023-01-24T14:52:42.027 INFO:tasks.ceph.osd.1.smithi002.stderr:kernel callstack: 2023-01-24T14:52:42.027 INFO:tasks.ceph.osd.1.smithi002.stderr:kernel callstack: 2023-01-24T14:52:42.027 INFO:tasks.ceph.osd.1.smithi002.stderr:kernel callstack: 2023-01-24T14:52:42.027 INFO:tasks.ceph.osd.1.smithi002.stderr:kernel callstack: 2023-01-24T14:52:42.028 INFO:tasks.ceph.osd.1.smithi002.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff823df837 0xffffffff81cf6712 0xffffffff81cf9378 0xffffffff81cfbac6 0xffffffff81d173d4 0xffffffff81cd96c4 0xffffffff81cdc2f3 0xffffffff81cdc62a 0xffffffff81a7c8b0 0xffffffff81a7cb47 0xffffffff8240114e 2023-01-24T14:52:42.028 INFO:tasks.ceph.osd.1.smithi002.stderr:kernel callstack: 2023-01-24T14:52:42.028 INFO:tasks.ceph.osd.1.smithi002.stderr:kernel callstack: 2023-01-24T14:52:42.028 INFO:tasks.ceph.osd.1.smithi002.stderr:kernel callstack: 2023-01-24T14:52:42.029 INFO:tasks.ceph.osd.1.smithi002.stderr:kernel callstack: 2023-01-24T14:52:42.029 INFO:tasks.ceph.osd.1.smithi002.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff823f6e91 0xffffffff81db4a78 0xffffffff81db5161 0xffffffff81db666a 0xffffffff81db6e4e 0xffffffff81a043ab 0xffffffff824000a9 2023-01-24T14:52:42.029 INFO:tasks.ceph.osd.1.smithi002.stderr:kernel callstack: 2023-01-24T14:52:42.029 INFO:tasks.ceph.osd.1.smithi002.stderr:kernel callstack: 2023-01-24T14:52:42.029 INFO:tasks.ceph.osd.1.smithi002.stderr:kernel callstack: 2023-01-24T14:52:42.030 INFO:tasks.ceph.osd.1.smithi002.stderr:kernel callstack: 2023-01-24T14:52:42.030 INFO:tasks.ceph.osd.1.smithi002.stderr:kernel callstack: 2023-01-24T14:52:42.030 INFO:tasks.ceph.osd.1.smithi002.stderr:kernel callstack: 2023-01-24T14:52:42.030 INFO:tasks.ceph.osd.1.smithi002.stderr:kernel callstack: 2023-01-24T14:52:42.031 INFO:tasks.ceph.osd.1.smithi002.stderr:kernel callstack: 2023-01-24T14:52:42.031 INFO:tasks.ceph.osd.1.smithi002.stderr:kernel callstack: 2023-01-24T14:52:42.031 INFO:tasks.ceph.osd.1.smithi002.stderr:kernel callstack: 2023-01-24T14:52:42.031 INFO:tasks.ceph.osd.1.smithi002.stderr:kernel callstack: 2023-01-24T14:52:42.031 INFO:tasks.ceph.osd.1.smithi002.stderr:kernel callstack: 2023-01-24T14:52:42.032 INFO:tasks.ceph.osd.1.smithi002.stderr:kernel callstack: 2023-01-24T14:52:42.032 INFO:tasks.ceph.osd.1.smithi002.stderr:kernel callstack: 2023-01-24T14:52:42.032 INFO:tasks.ceph.osd.1.smithi002.stderr:kernel callstack: 2023-01-24T14:52:42.032 INFO:tasks.ceph.osd.1.smithi002.stderr:kernel callstack: 2023-01-24T14:52:42.032 INFO:tasks.ceph.osd.1.smithi002.stderr:kernel callstack: 2023-01-24T14:52:42.033 INFO:tasks.ceph.osd.1.smithi002.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff81d3bb2f 0xffffffff81d3bcad 0xffffffff81d41e41 0xffffffff81d45082 0xffffffff81ca74a0 0xffffffff81ce4e84 0xffffffff81ce50ec 0xffffffff81cd827a 0xffffffff81cfd957 0xffffffff81cff106 0xffffffff81a043ab 0xffffffff824000a9 2023-01-24T14:52:42.033 INFO:tasks.ceph.osd.1.smithi002.stderr:kernel callstack: 2023-01-24T14:52:42.033 INFO:tasks.ceph.osd.1.smithi002.stderr:kernel callstack: 2023-01-24T14:52:42.033 INFO:tasks.ceph.osd.1.smithi002.stderr:kernel callstack: 2023-01-24T14:52:42.033 INFO:tasks.ceph.osd.1.smithi002.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff81cd9377 0xffffffff81cdc2f3 0xffffffff81cdc62a 0xffffffff81a7c8b0 0xffffffff81a7cb47 0xffffffff8240114e 2023-01-24T14:52:42.034 INFO:tasks.ceph.osd.1.smithi002.stderr:kernel callstack: 2023-01-24T14:52:42.034 INFO:tasks.ceph.osd.1.smithi002.stderr:kernel callstack: 2023-01-24T14:52:42.034 INFO:tasks.ceph.osd.1.smithi002.stderr:kernel callstack: 2023-01-24T14:52:42.034 INFO:tasks.ceph.osd.1.smithi002.stderr:kernel callstack: 2023-01-24T14:52:42.034 INFO:tasks.ceph.osd.1.smithi002.stderr:kernel callstack: 2023-01-24T14:52:42.035 INFO:tasks.ceph.osd.1.smithi002.stderr:kernel callstack: 2023-01-24T14:52:42.035 INFO:tasks.ceph.osd.1.smithi002.stderr:kernel callstack: 2023-01-24T14:52:42.035 INFO:tasks.ceph.osd.1.smithi002.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff81d42ee2 0xffffffff81d43ce0 0xffffffff81d44e2f 0xffffffff81cd96e2 0xffffffff81cdc2f3 0xffffffff81cdc62a 0xffffffff81a7c8b0 0xffffffff81a7cb47 0xffffffff8240114e 2023-01-24T14:52:42.035 INFO:tasks.ceph.osd.1.smithi002.stderr:kernel callstack: 2023-01-24T14:52:42.035 INFO:tasks.ceph.osd.1.smithi002.stderr:kernel callstack: 2023-01-24T14:52:42.036 INFO:tasks.ceph.osd.1.smithi002.stderr:kernel callstack: 2023-01-24T14:52:42.036 INFO:tasks.ceph.osd.1.smithi002.stderr:kernel callstack: 2023-01-24T14:52:42.036 INFO:tasks.ceph.osd.1.smithi002.stderr:kernel callstack: 2023-01-24T14:52:42.036 INFO:tasks.ceph.osd.1.smithi002.stderr:kernel callstack: 2023-01-24T14:52:42.036 INFO:tasks.ceph.osd.1.smithi002.stderr:kernel callstack: 2023-01-24T14:52:42.037 INFO:tasks.ceph.osd.1.smithi002.stderr:kernel callstack: 2023-01-24T14:52:42.037 INFO:tasks.ceph.osd.1.smithi002.stderr:kernel callstack: 2023-01-24T14:52:42.037 INFO:tasks.ceph.osd.1.smithi002.stderr:kernel callstack: 2023-01-24T14:52:42.037 INFO:tasks.ceph.osd.1.smithi002.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff81d173d4 0xffffffff81cd96c4 0xffffffff81cdc2f3 0xffffffff81cdc62a 0xffffffff81a7c8b0 0xffffffff81a7cb47 0xffffffff8240114e 2023-01-24T14:52:42.037 INFO:tasks.ceph.osd.1.smithi002.stderr:kernel callstack: 2023-01-24T14:52:42.038 INFO:tasks.ceph.osd.1.smithi002.stderr:kernel callstack: 2023-01-24T14:52:42.038 INFO:tasks.ceph.osd.1.smithi002.stderr:kernel callstack: 2023-01-24T14:52:42.038 INFO:tasks.ceph.osd.1.smithi002.stderr:kernel callstack: 2023-01-24T14:52:42.038 INFO:tasks.ceph.osd.1.smithi002.stderr:kernel callstack: 2023-01-24T14:52:42.038 INFO:tasks.ceph.osd.1.smithi002.stderr:kernel callstack: 2023-01-24T14:52:42.039 INFO:tasks.ceph.osd.1.smithi002.stderr:kernel callstack: 2023-01-24T14:52:42.039 INFO:tasks.ceph.osd.1.smithi002.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff823df837 0xffffffff81cf6712 0xffffffff81cf9378 0xffffffff81cfbac6 0xffffffff81d173d4 0xffffffff81cd96c4 0xffffffff81cdc2f3 0xffffffff81cdc62a 0xffffffff81a7c8b0 0xffffffff81a7cb47 0xffffffff8240114e 2023-01-24T14:52:42.039 INFO:tasks.ceph.osd.1.smithi002.stderr:kernel callstack: 2023-01-24T14:52:42.039 INFO:tasks.ceph.osd.1.smithi002.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff823df839 0xffffffff81cf6712 0xffffffff81cf9378 0xffffffff81cfbac6 0xffffffff81a7df84 0xffffffff81cd6006 0xffffffff81cd9b2a 0xffffffff81cdc2f3 0xffffffff81cdc62a 0xffffffff81a7c8b0 0xffffffff81a7cb47 0xffffffff8240114e 2023-01-24T14:52:42.039 INFO:tasks.ceph.osd.1.smithi002.stderr:kernel callstack: 2023-01-24T14:52:42.039 INFO:tasks.ceph.osd.1.smithi002.stderr:kernel callstack: 2023-01-24T14:52:42.040 INFO:tasks.ceph.osd.1.smithi002.stderr:kernel callstack: 2023-01-24T14:52:42.040 INFO:tasks.ceph.osd.1.smithi002.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff81cd795b 0xffffffff81cd8140 0xffffffff81cdfc28 0xffffffff81ce1e6d 0xffffffff81ce2158 0xffffffff81ce21e7 0xffffffff81a043ab 0xffffffff824000a9 2023-01-24T14:52:42.040 INFO:tasks.ceph.osd.1.smithi002.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff81a03974 0xffffffff81a044c4 0xffffffff824000a9 2023-01-24T14:52:42.040 INFO:tasks.ceph.osd.1.smithi002.stderr:kernel callstack: 2023-01-24T14:52:42.040 INFO:tasks.ceph.osd.1.smithi002.stderr:kernel callstack: 2023-01-24T14:52:42.041 INFO:tasks.ceph.osd.1.smithi002.stderr:kernel callstack: 2023-01-24T14:52:42.041 INFO:tasks.ceph.osd.1.smithi002.stderr:kernel callstack: 2023-01-24T14:52:42.041 INFO:tasks.ceph.osd.1.smithi002.stderr:kernel callstack: 2023-01-24T14:52:42.041 INFO:tasks.ceph.osd.1.smithi002.stderr:kernel callstack: 2023-01-24T14:52:42.041 INFO:tasks.ceph.osd.1.smithi002.stderr:kernel callstack: 2023-01-24T14:52:42.042 INFO:tasks.ceph.osd.1.smithi002.stderr:kernel callstack: 2023-01-24T14:52:42.042 INFO:tasks.ceph.osd.1.smithi002.stderr:kernel callstack: 2023-01-24T14:52:42.042 INFO:tasks.ceph.osd.1.smithi002.stderr:kernel callstack: 2023-01-24T14:52:42.042 INFO:tasks.ceph.osd.1.smithi002.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff823f6e91 0xffffffff81ca74e4 0xffffffff81ce4e84 0xffffffff81ce50ec 0xffffffff81cdfc51 0xffffffff81ce1e6d 0xffffffff81ce2158 0xffffffff81ce21e7 0xffffffff81a043ab 0xffffffff824000a9 2023-01-24T14:52:42.042 INFO:tasks.ceph.osd.1.smithi002.stderr:kernel callstack: 2023-01-24T14:52:42.043 INFO:tasks.ceph.osd.1.smithi002.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff81a7c919 0xffffffff81a7cb47 0xffffffff8240114e 2023-01-24T14:52:42.043 INFO:tasks.ceph.osd.1.smithi002.stderr:kernel callstack: 2023-01-24T14:52:42.043 INFO:tasks.ceph.osd.1.smithi002.stderr:kernel callstack: 2023-01-24T14:52:42.043 INFO:tasks.ceph.osd.1.smithi002.stderr:kernel callstack: 2023-01-24T14:52:42.043 INFO:tasks.ceph.osd.1.smithi002.stderr:kernel callstack: 0xffffffffffffff80 0xffffffffc070ad35 0xffffffffc070e972 0xffffffffc070ecd7 0xffffffffc0714099 0xffffffff81d95291 0xffffffff81da3d67 0xffffffffc0712f65 0xffffffff81c9671e 0xffffffff81c9b9c2 0xffffffffc06fd8e5 0xffffffff81d5d462 0xffffffff81d60c55 0xffffffff81d60edf 0xffffffff81a043ab 0xffffffff824000a9 2023-01-24T14:52:42.044 INFO:tasks.ceph.osd.1.smithi002.stderr:kernel callstack: 2023-01-24T14:52:42.044 INFO:tasks.ceph.osd.1.smithi002.stderr:kernel callstack: 2023-01-24T14:52:42.044 INFO:tasks.ceph.osd.1.smithi002.stderr:kernel callstack: 2023-01-24T14:52:42.044 INFO:tasks.ceph.osd.1.smithi002.stderr:kernel callstack: 2023-01-24T14:52:42.044 INFO:tasks.ceph.osd.1.smithi002.stderr:kernel callstack: 2023-01-24T14:52:42.045 INFO:tasks.ceph.osd.1.smithi002.stderr:kernel callstack: 2023-01-24T14:52:42.045 INFO:tasks.ceph.osd.1.smithi002.stderr:kernel callstack: 2023-01-24T14:52:42.045 INFO:tasks.ceph.osd.1.smithi002.stderr:kernel callstack: 2023-01-24T14:52:42.045 INFO:tasks.ceph.osd.1.smithi002.stderr:kernel callstack: 2023-01-24T14:52:42.045 INFO:tasks.ceph.osd.1.smithi002.stderr:kernel callstack: 2023-01-24T14:52:42.045 INFO:tasks.ceph.osd.1.smithi002.stderr:kernel callstack: 2023-01-24T14:52:42.046 INFO:tasks.ceph.osd.1.smithi002.stderr:kernel callstack: 2023-01-24T14:52:42.046 INFO:tasks.ceph.osd.1.smithi002.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff81c79100 0xffffffff81e2eb5e 0xffffffff81d60c17 0xffffffff81d60edf 0xffffffff81a043ab 0xffffffff824000a9 2023-01-24T14:52:42.046 INFO:tasks.ceph.osd.1.smithi002.stderr:kernel callstack: 2023-01-24T14:52:42.046 INFO:tasks.ceph.osd.1.smithi002.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff81a7c919 0xffffffff81a7cb47 0xffffffff8240114e 2023-01-24T14:52:42.046 INFO:tasks.ceph.osd.1.smithi002.stderr:kernel callstack: 0xffffffffffffff80 0xffffffffc06f1231 0xffffffff81d95291 0xffffffff81da3d67 0xffffffffc0712f65 0xffffffff81c9671e 0xffffffff81c9b9c2 0xffffffffc06fd8e5 0xffffffff81d5d462 0xffffffff81d60c55 0xffffffff81d60edf 0xffffffff81a043ab 0xffffffff824000a9 2023-01-24T14:52:42.047 INFO:tasks.ceph.osd.1.smithi002.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff81a7c919 0xffffffff81a7cb47 0xffffffff8240114e 2023-01-24T14:52:42.047 INFO:tasks.ceph.osd.1.smithi002.stderr:kernel callstack: 2023-01-24T14:52:42.047 INFO:tasks.ceph.osd.1.smithi002.stderr:kernel callstack: 2023-01-24T14:52:42.047 INFO:tasks.ceph.osd.1.smithi002.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff81c98e07 0xffffffff81c9ae6f 0xffffffffc071162e 0xffffffff81c966d2 0xffffffff81c9b9c2 0xffffffffc06fd8e5 0xffffffff81d5d462 0xffffffff81d60c55 0xffffffff81d60edf 0xffffffff81a043ab 0xffffffff824000a9 2023-01-24T14:52:42.047 INFO:tasks.ceph.osd.1.smithi002.stderr:kernel callstack: 2023-01-24T14:52:42.048 INFO:tasks.ceph.osd.1.smithi002.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff81cf4bc6 0xffffffff81cf7a46 0xffffffff81ca74aa 0xffffffff81ce4e84 0xffffffff81ce50ec 0xffffffff81cdfc51 0xffffffff81ce1e6d 0xffffffff81ce2158 0xffffffff81ce21e7 0xffffffff81a043ab 0xffffffff824000a9 2023-01-24T14:52:42.048 INFO:tasks.ceph.osd.1.smithi002.stderr:kernel callstack: 2023-01-24T14:52:42.048 INFO:tasks.ceph.osd.1.smithi002.stderr:kernel callstack: 2023-01-24T14:52:42.048 INFO:tasks.ceph.osd.1.smithi002.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff81d21055 0xffffffff81d25a5f 0xffffffff81ced39f 0xffffffff81ce1ad4 0xffffffff81ce6529 0xffffffff81ce6734 0xffffffff81ce68fb 0xffffffff81a043ab 0xffffffff824000a9 2023-01-24T14:52:42.049 INFO:tasks.ceph.osd.1.smithi002.stderr:kernel callstack: 2023-01-24T14:52:42.049 INFO:tasks.ceph.osd.1.smithi002.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff81b46ee4 0xffffffffc0575a26 0xffffffffc0576811 0xffffffffc06f1236 0xffffffffc0712f81 0xffffffff81c9671e 0xffffffff81c9b9c2 0xffffffffc06fd8e5 0xffffffff81d5d462 0xffffffff81d60c55 0xffffffff81d60edf 0xffffffff81a043ab 0xffffffff824000a9 2023-01-24T14:52:42.049 INFO:tasks.ceph.osd.1.smithi002.stderr:kernel callstack: 2023-01-24T14:52:42.049 INFO:tasks.ceph.osd.1.smithi002.stderr:kernel callstack: 2023-01-24T14:52:42.049 INFO:tasks.ceph.osd.1.smithi002.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff81cf862c 0xffffffff81cfbac6 0xffffffff81d173d4 0xffffffff81cd96c4 0xffffffff81cdc2f3 0xffffffff81cdc62a 0xffffffff81a7c8b0 0xffffffff81a7cb47 0xffffffff8240114e 2023-01-24T14:52:42.050 INFO:tasks.ceph.osd.1.smithi002.stderr:kernel callstack: 2023-01-24T14:52:42.050 INFO:tasks.ceph.osd.1.smithi002.stderr:kernel callstack: 2023-01-24T14:52:42.050 INFO:tasks.ceph.osd.1.smithi002.stderr:kernel callstack: 2023-01-24T14:52:42.050 INFO:tasks.ceph.osd.1.smithi002.stderr:kernel callstack: 2023-01-24T14:52:42.050 INFO:tasks.ceph.osd.1.smithi002.stderr:kernel callstack: 2023-01-24T14:52:42.051 INFO:tasks.ceph.osd.1.smithi002.stderr:kernel callstack: 2023-01-24T14:52:42.051 INFO:tasks.ceph.osd.1.smithi002.stderr:kernel callstack: 2023-01-24T14:52:42.051 INFO:tasks.ceph.osd.1.smithi002.stderr:kernel callstack: 2023-01-24T14:52:42.051 INFO:tasks.ceph.osd.1.smithi002.stderr:kernel callstack: 2023-01-24T14:52:42.051 INFO:tasks.ceph.osd.1.smithi002.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff81d8149e 0xffffffff821f98c2 0xffffffff821fcdc7 0xffffffff81a043ab 0xffffffff824000a9 2023-01-24T14:52:42.052 INFO:tasks.ceph.osd.1.smithi002.stderr:kernel callstack: 2023-01-24T14:52:42.052 INFO:tasks.ceph.osd.1.smithi002.stderr:kernel callstack: 2023-01-24T14:52:42.052 INFO:tasks.ceph.osd.1.smithi002.stderr:kernel callstack: 2023-01-24T14:52:42.052 INFO:tasks.ceph.osd.1.smithi002.stderr:kernel callstack: 2023-01-24T14:52:42.052 INFO:tasks.ceph.osd.1.smithi002.stderr:kernel callstack: 2023-01-24T14:52:42.053 INFO:tasks.ceph.osd.1.smithi002.stderr:kernel callstack: 2023-01-24T14:52:42.053 INFO:tasks.ceph.osd.1.smithi002.stderr:kernel callstack: 2023-01-24T14:52:42.053 INFO:tasks.ceph.osd.1.smithi002.stderr:kernel callstack: 2023-01-24T14:52:42.053 INFO:tasks.ceph.osd.1.smithi002.stderr:kernel callstack: 2023-01-24T14:52:42.053 INFO:tasks.ceph.osd.1.smithi002.stderr:kernel callstack: 2023-01-24T14:52:42.053 INFO:tasks.ceph.osd.1.smithi002.stderr:kernel callstack: 2023-01-24T14:52:42.054 INFO:tasks.ceph.osd.1.smithi002.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff81cdbf59 0xffffffff81cdc62a 0xffffffff81a7c8b0 0xffffffff81a7cb47 0xffffffff8240114e 2023-01-24T14:52:45.984 INFO:tasks.workunit.client.0.smithi002.stderr:test_rados.TestIoctx.test_xattrs ... ok 2023-01-24T14:53:36.991 INFO:tasks.workunit.client.0.smithi002.stderr:test_rados.TestIoctx.test_xattrs_op ... ok 2023-01-24T14:53:40.000 INFO:tasks.workunit.client.0.smithi002.stderr:test_rados.TestIoctx2.test_get_last_version ... ok 2023-01-24T14:53:43.009 INFO:tasks.workunit.client.0.smithi002.stderr:test_rados.TestIoctx2.test_get_stats ... ok 2023-01-24T14:54:12.110 INFO:tasks.ceph.osd.2.smithi039.stderr:Reactor stalled for 56 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x383b1e11 0x38390ac5 0x376fc9bb 0x375f9c0c 0x3760be55 0x3760c8a4 0x378ccc9e 0x378cd474 0x3687c5ba 0x3687f128 0x4449fbfa 0x444a049a 0x444a0a55 0x4449f7fa 0x3671ebc7 0x378e7f29 0x364b52af 0x364b571b 0x362c6842 0x362dcdef 0x362dda11 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T14:54:12.111 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff87f1f69e 0xffffffff87f21b06 0xffffffff887f08e9 0xffffffff887f0ed5 0xffffffff87f2604a 0xffffffff87e043ab 0xffffffff888000a9 2023-01-24T14:54:12.111 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:12.111 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:12.112 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:12.112 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:12.112 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff880e03a0 0xffffffff880e0b79 0xffffffff880e1bc3 0xffffffff880e6529 0xffffffff880e6734 0xffffffff880e68fb 0xffffffff87e043ab 0xffffffff888000a9 2023-01-24T14:54:12.112 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:12.112 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:12.113 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:12.113 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:12.113 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:12.113 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:12.113 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff88117360 0xffffffff880d96c4 0xffffffff880dc2f3 0xffffffff880dc62a 0xffffffff87e7c8b0 0xffffffff87e7cb47 0xffffffff8880114e 2023-01-24T14:54:12.114 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff87e7fca9 0xffffffff880e4ef9 0xffffffff880e50ec 0xffffffff880d827a 0xffffffff880fd957 0xffffffff880ff106 0xffffffff87e043ab 0xffffffff888000a9 2023-01-24T14:54:12.114 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:12.114 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:12.114 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:12.114 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:12.115 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:12.115 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:12.115 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:12.116 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:12.116 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:12.116 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:12.116 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:12.116 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:12.117 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:12.117 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:12.117 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:12.117 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:12.118 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:12.118 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:12.118 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:12.118 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:12.118 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff87e7c919 0xffffffff87e7cb47 0xffffffff8880114e 2023-01-24T14:54:12.119 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:12.119 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:12.119 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff87e7c919 0xffffffff87e7cb47 0xffffffff8880114e 2023-01-24T14:54:12.119 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:12.119 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:12.120 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:12.120 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:12.120 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:12.120 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:12.121 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:12.121 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:12.121 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:12.121 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff887df837 0xffffffff880f6712 0xffffffff880f9378 0xffffffff880fbac6 0xffffffff881173d4 0xffffffff880d96c4 0xffffffff880dc2f3 0xffffffff880dc62a 0xffffffff87e7c8b0 0xffffffff87e7cb47 0xffffffff8880114e 2023-01-24T14:54:12.121 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:12.121 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:12.122 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:12.122 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff87f56be9 0xffffffffc04ba8f0 0xffffffff8815d462 0xffffffff88160c55 0xffffffff88160edf 0xffffffff87e043ab 0xffffffff888000a9 2023-01-24T14:54:12.122 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:12.122 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:12.123 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:12.123 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:12.123 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:12.123 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:12.123 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:12.124 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:12.124 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:12.124 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:12.124 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:12.124 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:12.125 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff87e7c919 0xffffffff87e7cb47 0xffffffff8880114e 2023-01-24T14:54:12.125 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:12.125 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:12.125 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:12.125 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:12.125 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:12.126 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:12.126 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff880d409d 0xffffffff880d9715 0xffffffff880dc2f3 0xffffffff880dc62a 0xffffffff87e7c8b0 0xffffffff87e7cb47 0xffffffff8880114e 2023-01-24T14:54:12.126 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:12.126 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff87fc0735 0xffffffff8814087d 0xffffffff881417de 0xffffffff880ec1e2 0xffffffff880d75c9 0xffffffff880d8140 0xffffffff880dfc28 0xffffffff880e1e6d 0xffffffff880e2158 0xffffffff880e21e7 0xffffffff87e043ab 0xffffffff888000a9 2023-01-24T14:54:12.126 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:12.127 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:12.127 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:12.127 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:12.127 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff887f6e91 0xffffffff880a7541 0xffffffff880e4e84 0xffffffff880e50ec 0xffffffff880d827a 0xffffffff880fd957 0xffffffff880ff106 0xffffffff87e043ab 0xffffffff888000a9 2023-01-24T14:54:12.127 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:12.128 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:12.128 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:12.128 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff887df839 0xffffffff880f6712 0xffffffff880f9378 0xffffffff880fbac6 0xffffffff881173d4 0xffffffff880d96c4 0xffffffff880dc2f3 0xffffffff880dc62a 0xffffffff87e7c8b0 0xffffffff87e7cb47 0xffffffff8880114e 2023-01-24T14:54:12.128 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:12.128 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:12.129 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:12.129 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff887f6e91 0xffffffff880a7541 0xffffffff880e4e84 0xffffffff880e50ec 0xffffffff880dfc51 0xffffffff880e1e6d 0xffffffff880e2158 0xffffffff880e21e7 0xffffffff87e043ab 0xffffffff888000a9 2023-01-24T14:54:12.129 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:12.129 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:12.129 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:12.130 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:12.130 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:12.130 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:12.130 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:12.130 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:12.131 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:12.131 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:12.131 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff887df839 0xffffffff880f6712 0xffffffff880f9378 0xffffffff880fbac6 0xffffffff881173d4 0xffffffff880d96c4 0xffffffff880dc2f3 0xffffffff880dc62a 0xffffffff87e7c8b0 0xffffffff87e7cb47 0xffffffff8880114e 2023-01-24T14:54:12.131 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:12.131 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:12.132 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff880a1484 0xffffffffc04ce65d 0xffffffff880966d2 0xffffffff8809b9c2 0xffffffffc04ba8e5 0xffffffff8815d462 0xffffffff88160c55 0xffffffff88160edf 0xffffffff87e043ab 0xffffffff888000a9 2023-01-24T14:54:12.132 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff880cd8ca 0xffffffff880ed1f2 0xffffffff880e098e 0xffffffff880e1278 0xffffffff880e641b 0xffffffff880e6734 0xffffffff880e68fb 0xffffffff87e043ab 0xffffffff888000a9 2023-01-24T14:54:12.132 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff87e7c919 0xffffffff87e7cb47 0xffffffff8880114e 2023-01-24T14:54:12.132 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:12.133 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:12.133 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:12.133 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:12.133 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:12.133 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:12.134 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:12.134 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff880f6722 0xffffffff880f9378 0xffffffff880fbac6 0xffffffff881173d4 0xffffffff880d96c4 0xffffffff880dc2f3 0xffffffff880dc62a 0xffffffff87e7c8b0 0xffffffff87e7cb47 0xffffffff8880114e 2023-01-24T14:54:12.134 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:12.134 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:12.134 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:12.135 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:12.135 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:12.135 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:12.135 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:12.135 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffffc04c6226 0xffffffffc04ce5cf 0xffffffff880966d2 0xffffffff8809b9c2 0xffffffffc04ba8e5 0xffffffff8815d462 0xffffffff88160c55 0xffffffff88160edf 0xffffffff87e043ab 0xffffffff888000a9 2023-01-24T14:54:12.136 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:12.136 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff880cd835 0xffffffff880ed1f2 0xffffffff880d5f3f 0xffffffff880dfc49 0xffffffff880e1e6d 0xffffffff880e2158 0xffffffff880e21e7 0xffffffff87e043ab 0xffffffff888000a9 2023-01-24T14:54:12.136 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:12.136 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:12.136 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff880f862c 0xffffffff880fbac6 0xffffffff881173d4 0xffffffff880d96c4 0xffffffff880dc2f3 0xffffffff880dc62a 0xffffffff87e7c8b0 0xffffffff87e7cb47 0xffffffff8880114e 2023-01-24T14:54:12.137 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:12.137 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff881206d2 0xffffffff88126d06 0xffffffff880e09b0 0xffffffff880e1278 0xffffffff880e641b 0xffffffff880e6734 0xffffffff880e68fb 0xffffffff87e043ab 0xffffffff888000a9 2023-01-24T14:54:12.137 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:12.137 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:12.137 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:12.138 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:12.138 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:12.138 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff880e283e 0xffffffff880e63b9 0xffffffff880e6734 0xffffffff880e68fb 0xffffffff87e043ab 0xffffffff888000a9 2023-01-24T14:54:12.138 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff887df837 0xffffffff880f6712 0xffffffff880f9378 0xffffffff880fbac6 0xffffffff881173d4 0xffffffff880d96c4 0xffffffff880dc2f3 0xffffffff880dc62a 0xffffffff87e7c8b0 0xffffffff87e7cb47 0xffffffff8880114e 2023-01-24T14:54:12.138 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:12.139 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:12.139 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:12.139 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:12.139 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:12.139 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:12.139 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:26.006 INFO:tasks.workunit.client.0.smithi002.stderr:test_rados.TestObject.test_read ... ok 2023-01-24T14:54:42.989 INFO:tasks.ceph.osd.2.smithi039.stderr:Reactor stalled for 34 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x3b64e0b8 0x3b6513c1 0x35f19188 0x3602c6e5 0x3602ef36 0x3602f68b 0x45189fad 0x4518acc4 0x376fcd55 0x375f9c0c 0x3760be55 0x3760c8a4 0x378ccc9e 0x378cd474 0x3687c5ba 0x3687f128 0x4449fbfa 0x444a049a 0x444a0a55 0x4449f7fa 0x3671ebc7 0x378e7f29 0x364b52af 0x364b571b 0x362c6842 0x362dcdef 0x362dda11 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T14:54:42.989 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:42.989 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:42.990 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:42.990 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:42.990 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:42.990 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:42.990 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:42.991 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:42.991 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:42.991 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff8811740e 0xffffffff880d96c4 0xffffffff880dc2f3 0xffffffff880dc62a 0xffffffff87e7c8b0 0xffffffff87e7cb47 0xffffffff8880114e 2023-01-24T14:54:42.991 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:42.992 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff880f7b7b 0xffffffff880a74aa 0xffffffff880e4e84 0xffffffff880e50ec 0xffffffff880dfc51 0xffffffff880e1e6d 0xffffffff880e2158 0xffffffff880e21e7 0xffffffff87e043ab 0xffffffff888000a9 2023-01-24T14:54:42.992 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:42.992 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:42.992 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff88143d12 0xffffffff88144e2f 0xffffffff880d96e2 0xffffffff880dc2f3 0xffffffff880dc62a 0xffffffff87e7c8b0 0xffffffff87e7cb47 0xffffffff8880114e 2023-01-24T14:54:42.993 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff87e7c919 0xffffffff87e7cb47 0xffffffff8880114e 2023-01-24T14:54:42.993 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:42.993 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:42.996 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff87e7c919 0xffffffff87e7cb47 0xffffffff8880114e 2023-01-24T14:54:42.996 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:42.996 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:42.997 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:42.997 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff87e7c919 0xffffffff87e7cb47 0xffffffff8880114e 2023-01-24T14:54:42.997 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:42.997 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:42.997 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:42.998 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:42.998 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:42.998 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff880dfe3a 0xffffffff880e66cd 0xffffffff880e68fb 0xffffffff87e043ab 0xffffffff888000a9 2023-01-24T14:54:42.998 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:42.998 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff887f7180 0xffffffffc04c76bb 0xffffffffc04cb972 0xffffffffc04cbcd7 0xffffffffc04d1099 0xffffffff88195291 0xffffffff8817e06a 0xffffffff8817e5b5 0xffffffff8809b95c 0xffffffffc04ba8e5 0xffffffff8815d462 0xffffffff88160c55 0xffffffff88160edf 0xffffffff87e043ab 0xffffffff888000a9 2023-01-24T14:54:42.999 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:42.999 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff880961c7 0xffffffffc04ce642 0xffffffff880966d2 0xffffffff8809b9c2 0xffffffffc04ba8e5 0xffffffff8815d462 0xffffffff88160c55 0xffffffff88160edf 0xffffffff87e043ab 0xffffffff888000a9 2023-01-24T14:54:42.999 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:42.999 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:42.999 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff880e191a 0xffffffff87e2b49a 0xffffffff880dfd03 0xffffffff880e3c83 0xffffffff880bbed4 0xffffffff880e1369 0xffffffff87e043ab 0xffffffff888000a9 2023-01-24T14:54:43.000 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:43.000 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:43.000 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:43.000 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:43.001 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff87e7c919 0xffffffff87e7cb47 0xffffffff8880114e 2023-01-24T14:54:43.001 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:43.001 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:43.001 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffffc04cbc65 0xffffffffc04d1099 0xffffffff88195291 0xffffffff881a3d67 0xffffffffc04cff65 0xffffffff8809671e 0xffffffff8809b9c2 0xffffffffc04ba8e5 0xffffffff8815d462 0xffffffff88160c55 0xffffffff88160edf 0xffffffff87e043ab 0xffffffff888000a9 2023-01-24T14:54:43.001 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:43.002 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:43.002 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff880f862c 0xffffffff880fbac6 0xffffffff881173d4 0xffffffff880d96c4 0xffffffff880dc2f3 0xffffffff880dc62a 0xffffffff87e7c8b0 0xffffffff87e7cb47 0xffffffff8880114e 2023-01-24T14:54:43.002 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:43.002 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:43.002 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:43.003 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:43.003 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff880e53e4 0xffffffff880e6245 0xffffffff880e646b 0xffffffff880e6734 0xffffffff880e68fb 0xffffffff87e043ab 0xffffffff888000a9 2023-01-24T14:54:43.003 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff880dc6be 0xffffffff87e7c8b0 0xffffffff87e7cb47 0xffffffff8880114e 2023-01-24T14:54:43.003 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:43.004 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff880f862c 0xffffffff880fbac6 0xffffffff881173d4 0xffffffff880d96c4 0xffffffff880dc2f3 0xffffffff880dc62a 0xffffffff87e7c8b0 0xffffffff87e7cb47 0xffffffff8880114e 2023-01-24T14:54:43.004 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:43.004 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff887df837 0xffffffff880f6712 0xffffffff880f9378 0xffffffff880fbac6 0xffffffff881173d4 0xffffffff880d96c4 0xffffffff880dc2f3 0xffffffff880dc62a 0xffffffff87e7c8b0 0xffffffff87e7cb47 0xffffffff8880114e 2023-01-24T14:54:43.004 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:43.005 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:43.005 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:43.005 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:43.005 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:43.005 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:43.006 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:43.006 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:43.006 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:43.006 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:43.006 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:43.007 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:43.007 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:43.007 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:43.007 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:43.007 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:43.008 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:43.008 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:43.008 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff87e7c919 0xffffffff87e7cb47 0xffffffff8880114e 2023-01-24T14:54:43.008 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:43.008 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:43.009 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff880f8305 0xffffffff880fbac6 0xffffffff881173d4 0xffffffff880d96c4 0xffffffff880dc2f3 0xffffffff880dc62a 0xffffffff87e7c8b0 0xffffffff87e7cb47 0xffffffff8880114e 2023-01-24T14:54:43.009 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:43.009 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:43.009 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff887df837 0xffffffff880f6712 0xffffffff880f9378 0xffffffff880fbac6 0xffffffff881173d4 0xffffffff880d96c4 0xffffffff880dc2f3 0xffffffff880dc62a 0xffffffff87e7c8b0 0xffffffff87e7cb47 0xffffffff8880114e 2023-01-24T14:54:43.009 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:43.010 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:43.010 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:43.010 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:43.010 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:43.010 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:43.010 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff880dc16e 0xffffffff880dc62a 0xffffffff87e7c8b0 0xffffffff87e7cb47 0xffffffff8880114e 2023-01-24T14:54:43.011 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:43.011 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:43.011 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:43.011 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff887f704e 0xffffffffc031c4b4 0xffffffffc031cdce 0xffffffffc04ae12e 0xffffffffc04ce704 0xffffffff880966d2 0xffffffff8809b9c2 0xffffffffc04ba8e5 0xffffffff8815d462 0xffffffff88160c55 0xffffffff88160edf 0xffffffff87e043ab 0xffffffff888000a9 2023-01-24T14:54:43.011 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff881417d0 0xffffffff880ebfcb 0xffffffff880d97c4 0xffffffff880dc2f3 0xffffffff880dc62a 0xffffffff87e7c8b0 0xffffffff87e7cb47 0xffffffff8880114e 2023-01-24T14:54:43.012 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:43.012 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:43.012 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:43.012 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:43.013 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff87e0436e 0xffffffff888000a9 2023-01-24T14:54:43.013 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:43.013 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:43.013 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:43.013 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:43.014 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:43.014 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:43.014 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:43.014 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:43.014 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:43.014 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:43.015 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:43.015 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:43.015 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:43.015 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff880f862c 0xffffffff880fbac6 0xffffffff881173d4 0xffffffff880d96c4 0xffffffff880dc2f3 0xffffffff880dc62a 0xffffffff87e7c8b0 0xffffffff87e7cb47 0xffffffff8880114e 2023-01-24T14:54:43.015 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:43.016 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:43.016 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:43.016 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:43.016 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:43.016 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff880d75c4 0xffffffff880d8140 0xffffffff880dfc28 0xffffffff880e1e6d 0xffffffff880e2158 0xffffffff880e21e7 0xffffffff87e043ab 0xffffffff888000a9 2023-01-24T14:54:43.017 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:43.017 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff8809b954 0xffffffffc04ba8e5 0xffffffff8815d462 0xffffffff88160c55 0xffffffff88160edf 0xffffffff87e043ab 0xffffffff888000a9 2023-01-24T14:54:43.017 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:43.018 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:43.018 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:43.018 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff8816256b 0xffffffff88160cf9 0xffffffff88160edf 0xffffffff87e043ab 0xffffffff888000a9 2023-01-24T14:54:43.018 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:43.018 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:43.019 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff880dc5d4 0xffffffff87e7c8b0 0xffffffff87e7cb47 0xffffffff8880114e 2023-01-24T14:54:43.019 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:43.019 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:43.019 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:54:43.019 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T14:55:19.126 INFO:tasks.ceph.osd.0.smithi002.stderr:Reactor stalled for 37 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x35f98ba6 0x3602f6dc 0x45189bed 0x4518acc4 0x383b4383 0x38390611 0x376fc9bb 0x375f9c0c 0x3760be55 0x3760c8a4 0x378ccc9e 0x378cd474 0x3687c5ba 0x3687f128 0x4449fbfa 0x444a049a 0x444a0a55 0x4449f7fa 0x3671ebc7 0x378e7f29 0x364b52af 0x364b571b 0x362c6842 0x362dcdef 0x362dda11 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T14:55:19.127 INFO:tasks.ceph.osd.0.smithi002.stderr:kernel callstack: 2023-01-24T14:55:19.127 INFO:tasks.ceph.osd.0.smithi002.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff81b1f69e 0xffffffff81b21b06 0xffffffff823f08e9 0xffffffff823f0ed5 0xffffffff81b2604a 0xffffffff81a043ab 0xffffffff824000a9 2023-01-24T14:55:19.127 INFO:tasks.ceph.osd.0.smithi002.stderr:kernel callstack: 2023-01-24T14:55:19.127 INFO:tasks.ceph.osd.0.smithi002.stderr:kernel callstack: 2023-01-24T14:55:19.128 INFO:tasks.ceph.osd.0.smithi002.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff81e2b06e 0xffffffff81e2ee12 0xffffffff81cdfd7b 0xffffffff81ce3c83 0xffffffff81cbbed4 0xffffffff81ce1369 0xffffffff81a043ab 0xffffffff824000a9 2023-01-24T14:55:19.128 INFO:tasks.ceph.osd.0.smithi002.stderr:kernel callstack: 2023-01-24T14:55:19.128 INFO:tasks.ceph.osd.0.smithi002.stderr:kernel callstack: 2023-01-24T14:55:19.128 INFO:tasks.ceph.osd.0.smithi002.stderr:kernel callstack: 2023-01-24T14:55:19.128 INFO:tasks.ceph.osd.0.smithi002.stderr:kernel callstack: 2023-01-24T14:55:19.129 INFO:tasks.ceph.osd.0.smithi002.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff81d44e61 0xffffffff81cd96e2 0xffffffff81cdc2f3 0xffffffff81cdc62a 0xffffffff81a7c8b0 0xffffffff81a7cb47 0xffffffff8240114e 2023-01-24T14:55:19.129 INFO:tasks.ceph.osd.0.smithi002.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff81cdfe21 0xffffffff81a7c83a 0xffffffff81a7cb47 0xffffffff8240114e 2023-01-24T14:55:19.129 INFO:tasks.ceph.osd.0.smithi002.stderr:kernel callstack: 2023-01-24T14:55:19.129 INFO:tasks.ceph.osd.0.smithi002.stderr:kernel callstack: 2023-01-24T14:55:19.129 INFO:tasks.ceph.osd.0.smithi002.stderr:kernel callstack: 2023-01-24T14:55:19.130 INFO:tasks.ceph.osd.0.smithi002.stderr:kernel callstack: 2023-01-24T14:55:19.130 INFO:tasks.ceph.osd.0.smithi002.stderr:kernel callstack: 2023-01-24T14:55:19.130 INFO:tasks.ceph.osd.0.smithi002.stderr:kernel callstack: 2023-01-24T14:55:19.130 INFO:tasks.ceph.osd.0.smithi002.stderr:kernel callstack: 2023-01-24T14:55:19.130 INFO:tasks.ceph.osd.0.smithi002.stderr:kernel callstack: 2023-01-24T14:55:19.131 INFO:tasks.ceph.osd.0.smithi002.stderr:kernel callstack: 2023-01-24T14:55:19.131 INFO:tasks.ceph.osd.0.smithi002.stderr:kernel callstack: 2023-01-24T14:55:19.131 INFO:tasks.ceph.osd.0.smithi002.stderr:kernel callstack: 2023-01-24T14:55:19.131 INFO:tasks.ceph.osd.0.smithi002.stderr:kernel callstack: 2023-01-24T14:55:19.131 INFO:tasks.ceph.osd.0.smithi002.stderr:kernel callstack: 2023-01-24T14:55:19.131 INFO:tasks.ceph.osd.0.smithi002.stderr:kernel callstack: 2023-01-24T14:55:19.132 INFO:tasks.ceph.osd.0.smithi002.stderr:kernel callstack: 2023-01-24T14:55:19.132 INFO:tasks.ceph.osd.0.smithi002.stderr:kernel callstack: 2023-01-24T14:55:19.132 INFO:tasks.ceph.osd.0.smithi002.stderr:kernel callstack: 2023-01-24T14:55:19.132 INFO:tasks.ceph.osd.0.smithi002.stderr:kernel callstack: 2023-01-24T14:55:19.132 INFO:tasks.ceph.osd.0.smithi002.stderr:kernel callstack: 2023-01-24T14:55:19.133 INFO:tasks.ceph.osd.0.smithi002.stderr:kernel callstack: 2023-01-24T14:55:19.133 INFO:tasks.ceph.osd.0.smithi002.stderr:kernel callstack: 2023-01-24T14:55:19.133 INFO:tasks.ceph.osd.0.smithi002.stderr:kernel callstack: 2023-01-24T14:55:19.133 INFO:tasks.ceph.osd.0.smithi002.stderr:kernel callstack: 2023-01-24T14:55:19.133 INFO:tasks.ceph.osd.0.smithi002.stderr:kernel callstack: 2023-01-24T14:55:19.134 INFO:tasks.ceph.osd.0.smithi002.stderr:kernel callstack: 2023-01-24T14:55:19.134 INFO:tasks.ceph.osd.0.smithi002.stderr:kernel callstack: 2023-01-24T14:55:19.134 INFO:tasks.ceph.osd.0.smithi002.stderr:kernel callstack: 2023-01-24T14:55:19.134 INFO:tasks.ceph.osd.0.smithi002.stderr:kernel callstack: 2023-01-24T14:55:19.134 INFO:tasks.ceph.osd.0.smithi002.stderr:kernel callstack: 2023-01-24T14:55:19.135 INFO:tasks.ceph.osd.0.smithi002.stderr:kernel callstack: 2023-01-24T14:55:19.135 INFO:tasks.ceph.osd.0.smithi002.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff81a7c919 0xffffffff81a7cb47 0xffffffff8240114e 2023-01-24T14:55:19.135 INFO:tasks.ceph.osd.0.smithi002.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff81b3bde7 0xffffffff826000dc 0xffffffff81afaaa5 0xffffffff81afaaba 0xffffffff82402744 0xffffffff82401c6f 2023-01-24T14:55:19.135 INFO:tasks.ceph.osd.0.smithi002.stderr:kernel callstack: 2023-01-24T14:55:19.135 INFO:tasks.ceph.osd.0.smithi002.stderr:kernel callstack: 2023-01-24T14:55:19.136 INFO:tasks.ceph.osd.0.smithi002.stderr:kernel callstack: 2023-01-24T14:55:19.136 INFO:tasks.ceph.osd.0.smithi002.stderr:kernel callstack: 2023-01-24T14:55:19.136 INFO:tasks.ceph.osd.0.smithi002.stderr:kernel callstack: 2023-01-24T14:55:19.136 INFO:tasks.ceph.osd.0.smithi002.stderr:kernel callstack: 2023-01-24T14:55:19.136 INFO:tasks.ceph.osd.0.smithi002.stderr:kernel callstack: 2023-01-24T14:55:19.137 INFO:tasks.ceph.osd.0.smithi002.stderr:kernel callstack: 2023-01-24T14:55:19.137 INFO:tasks.ceph.osd.0.smithi002.stderr:kernel callstack: 2023-01-24T14:55:19.137 INFO:tasks.ceph.osd.0.smithi002.stderr:kernel callstack: 2023-01-24T14:55:19.137 INFO:tasks.ceph.osd.0.smithi002.stderr:kernel callstack: 2023-01-24T14:55:19.137 INFO:tasks.ceph.osd.0.smithi002.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff823df837 0xffffffff81cf6712 0xffffffff81cf9378 0xffffffff81cfbac6 0xffffffff81d173d4 0xffffffff81cd96c4 0xffffffff81cdc2f3 0xffffffff81cdc62a 0xffffffff81a7c8b0 0xffffffff81a7cb47 0xffffffff8240114e 2023-01-24T14:55:19.137 INFO:tasks.ceph.osd.0.smithi002.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff81d7e503 0xffffffff81c9b95c 0xffffffffc06fd8e5 0xffffffff81d5d462 0xffffffff81d60c55 0xffffffff81d60edf 0xffffffff81a043ab 0xffffffff824000a9 2023-01-24T14:55:19.138 INFO:tasks.ceph.osd.0.smithi002.stderr:kernel callstack: 2023-01-24T14:55:19.138 INFO:tasks.ceph.osd.0.smithi002.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff81d43d12 0xffffffff81d44e2f 0xffffffff81cd96e2 0xffffffff81cdc2f3 0xffffffff81cdc62a 0xffffffff81a7c8b0 0xffffffff81a7cb47 0xffffffff8240114e 2023-01-24T14:55:19.138 INFO:tasks.ceph.osd.0.smithi002.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff81a7c919 0xffffffff81a7cb47 0xffffffff8240114e 2023-01-24T14:55:19.138 INFO:tasks.ceph.osd.0.smithi002.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff81d42ee9 0xffffffff81d43ce0 0xffffffff81d44e2f 0xffffffff81cd96e2 0xffffffff81cdc2f3 0xffffffff81cdc62a 0xffffffff81a7c8b0 0xffffffff81a7cb47 0xffffffff8240114e 2023-01-24T14:55:19.138 INFO:tasks.ceph.osd.0.smithi002.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff81d41747 0xffffffff81d417de 0xffffffff81cebfcb 0xffffffff81cd97c4 0xffffffff81cdc2f3 0xffffffff81cdc62a 0xffffffff81a7c8b0 0xffffffff81a7cb47 0xffffffff8240114e 2023-01-24T14:55:19.139 INFO:tasks.ceph.osd.0.smithi002.stderr:kernel callstack: 2023-01-24T14:55:19.139 INFO:tasks.ceph.osd.0.smithi002.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff81d43d12 0xffffffff81d44e2f 0xffffffff81cd96e2 0xffffffff81cdc2f3 0xffffffff81cdc62a 0xffffffff81a7c8b0 0xffffffff81a7cb47 0xffffffff8240114e 2023-01-24T14:55:19.139 INFO:tasks.ceph.osd.0.smithi002.stderr:kernel callstack: 2023-01-24T14:55:19.139 INFO:tasks.ceph.osd.0.smithi002.stderr:kernel callstack: 2023-01-24T14:55:19.139 INFO:tasks.ceph.osd.0.smithi002.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff81cf862c 0xffffffff81cfbac6 0xffffffff81d173d4 0xffffffff81cd96c4 0xffffffff81cdc2f3 0xffffffff81cdc62a 0xffffffff81a7c8b0 0xffffffff81a7cb47 0xffffffff8240114e 2023-01-24T14:55:19.140 INFO:tasks.ceph.osd.0.smithi002.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff81c96503 0xffffffffc06fd1cb 0xffffffffc06fd8c9 0xffffffff81d5d462 0xffffffff81d60c55 0xffffffff81d60edf 0xffffffff81a043ab 0xffffffff824000a9 2023-01-24T14:55:19.140 INFO:tasks.ceph.osd.0.smithi002.stderr:kernel callstack: 2023-01-24T14:55:19.140 INFO:tasks.ceph.osd.0.smithi002.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff81a0436e 0xffffffff824000a9 2023-01-24T14:55:19.140 INFO:tasks.ceph.osd.0.smithi002.stderr:kernel callstack: 2023-01-24T14:55:19.140 INFO:tasks.ceph.osd.0.smithi002.stderr:kernel callstack: 2023-01-24T14:55:19.141 INFO:tasks.ceph.osd.0.smithi002.stderr:kernel callstack: 2023-01-24T14:55:19.141 INFO:tasks.ceph.osd.0.smithi002.stderr:kernel callstack: 2023-01-24T14:55:19.141 INFO:tasks.ceph.osd.0.smithi002.stderr:kernel callstack: 2023-01-24T14:55:19.141 INFO:tasks.ceph.osd.0.smithi002.stderr:kernel callstack: 2023-01-24T14:55:19.141 INFO:tasks.ceph.osd.0.smithi002.stderr:kernel callstack: 2023-01-24T14:55:19.141 INFO:tasks.ceph.osd.0.smithi002.stderr:kernel callstack: 2023-01-24T14:55:19.142 INFO:tasks.ceph.osd.0.smithi002.stderr:kernel callstack: 2023-01-24T14:55:19.142 INFO:tasks.ceph.osd.0.smithi002.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff81cec1c4 0xffffffff81cd75c9 0xffffffff81cd8140 0xffffffff81cdfc28 0xffffffff81ce1e6d 0xffffffff81ce2158 0xffffffff81ce21e7 0xffffffff81a043ab 0xffffffff824000a9 2023-01-24T14:55:19.142 INFO:tasks.ceph.osd.0.smithi002.stderr:kernel callstack: 2023-01-24T14:55:19.142 INFO:tasks.ceph.osd.0.smithi002.stderr:kernel callstack: 2023-01-24T14:55:19.142 INFO:tasks.ceph.osd.0.smithi002.stderr:kernel callstack: 2023-01-24T14:55:19.143 INFO:tasks.ceph.osd.0.smithi002.stderr:kernel callstack: 2023-01-24T14:55:19.143 INFO:tasks.ceph.osd.0.smithi002.stderr:kernel callstack: 2023-01-24T14:55:19.143 INFO:tasks.ceph.osd.0.smithi002.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff81a7c919 0xffffffff81a7cb47 0xffffffff8240114e 2023-01-24T14:55:19.143 INFO:tasks.ceph.osd.0.smithi002.stderr:kernel callstack: 2023-01-24T14:55:19.143 INFO:tasks.ceph.osd.0.smithi002.stderr:kernel callstack: 2023-01-24T14:55:19.144 INFO:tasks.ceph.osd.0.smithi002.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff81d9510b 0xffffffff81da3d67 0xffffffffc0712f65 0xffffffff81c9671e 0xffffffff81c9b9c2 0xffffffffc06fd8e5 0xffffffff81d5d462 0xffffffff81d60c55 0xffffffff81d60edf 0xffffffff81a043ab 0xffffffff824000a9 2023-01-24T14:55:19.144 INFO:tasks.ceph.osd.0.smithi002.stderr:kernel callstack: 2023-01-24T14:55:19.144 INFO:tasks.ceph.osd.0.smithi002.stderr:kernel callstack: 2023-01-24T14:55:19.144 INFO:tasks.ceph.osd.0.smithi002.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff81ce27fe 0xffffffff81ce63b9 0xffffffff81ce6734 0xffffffff81ce68fb 0xffffffff81a043ab 0xffffffff824000a9 2023-01-24T14:55:19.145 INFO:tasks.ceph.osd.0.smithi002.stderr:kernel callstack: 0xffffffffffffff80 0xffffffffc048f2c7 0xffffffffc070a28a 0xffffffffc070a435 0xffffffffc070aad2 0xffffffffc070e972 0xffffffffc070ecd7 0xffffffffc0714099 0xffffffff81d95291 0xffffffff81d7e06a 0xffffffff81d7e5b5 0xffffffff81c9b95c 0xffffffffc06fd8e5 0xffffffff81d5d462 0xffffffff81d60c55 0xffffffff81d60edf 0xffffffff81a043ab 0xffffffff824000a9 2023-01-24T14:55:19.145 INFO:tasks.ceph.osd.0.smithi002.stderr:kernel callstack: 2023-01-24T14:55:19.145 INFO:tasks.ceph.osd.0.smithi002.stderr:kernel callstack: 2023-01-24T14:55:19.145 INFO:tasks.ceph.osd.0.smithi002.stderr:kernel callstack: 2023-01-24T14:55:19.145 INFO:tasks.ceph.osd.0.smithi002.stderr:kernel callstack: 2023-01-24T14:55:19.146 INFO:tasks.ceph.osd.0.smithi002.stderr:kernel callstack: 2023-01-24T14:55:19.146 INFO:tasks.ceph.osd.0.smithi002.stderr:kernel callstack: 2023-01-24T14:55:19.146 INFO:tasks.ceph.osd.0.smithi002.stderr:kernel callstack: 2023-01-24T14:55:19.146 INFO:tasks.ceph.osd.0.smithi002.stderr:kernel callstack: 2023-01-24T14:55:19.146 INFO:tasks.ceph.osd.0.smithi002.stderr:kernel callstack: 2023-01-24T14:55:19.147 INFO:tasks.ceph.osd.0.smithi002.stderr:kernel callstack: 2023-01-24T14:55:19.147 INFO:tasks.ceph.osd.0.smithi002.stderr:kernel callstack: 2023-01-24T14:55:19.147 INFO:tasks.ceph.osd.0.smithi002.stderr:kernel callstack: 0xffffffffffffff80 0xffffffffc05767fc 0xffffffffc06f1236 0xffffffff81d95291 0xffffffff81d7e06a 0xffffffff81d7e5b5 0xffffffff81c9b95c 0xffffffffc06fd8e5 0xffffffff81d5d462 0xffffffff81d60c55 0xffffffff81d60edf 0xffffffff81a043ab 0xffffffff824000a9 2023-01-24T14:55:19.147 INFO:tasks.ceph.osd.0.smithi002.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff81ca2fbe 0xffffffff81c96759 0xffffffff81c9b9c2 0xffffffffc06fd8e5 0xffffffff81d5d462 0xffffffff81d60c55 0xffffffff81d60edf 0xffffffff81a043ab 0xffffffff824000a9 2023-01-24T14:55:19.147 INFO:tasks.ceph.osd.0.smithi002.stderr:kernel callstack: 2023-01-24T14:55:19.147 INFO:tasks.ceph.osd.0.smithi002.stderr:kernel callstack: 2023-01-24T14:55:19.148 INFO:tasks.ceph.osd.0.smithi002.stderr:kernel callstack: 2023-01-24T14:55:19.148 INFO:tasks.ceph.osd.0.smithi002.stderr:kernel callstack: 2023-01-24T14:55:19.148 INFO:tasks.ceph.osd.0.smithi002.stderr:kernel callstack: 2023-01-24T14:55:19.148 INFO:tasks.ceph.osd.0.smithi002.stderr:kernel callstack: 0xffffffffffffff80 0xffffffffc048f2ff 2023-01-24T14:55:19.148 INFO:tasks.ceph.osd.0.smithi002.stderr:kernel callstack: 2023-01-24T14:55:19.149 INFO:tasks.ceph.osd.0.smithi002.stderr:kernel callstack: 2023-01-24T14:55:19.149 INFO:tasks.ceph.osd.0.smithi002.stderr:kernel callstack: 2023-01-24T14:55:19.149 INFO:tasks.ceph.osd.0.smithi002.stderr:kernel callstack: 0xffffffffffffff80 0xffffffffc06f179f 0xffffffffc070ab05 0xffffffffc070e972 0xffffffffc070ecd7 0xffffffffc0714099 0xffffffff81d95291 0xffffffff81da3d67 0xffffffffc0712f65 0xffffffff81c9671e 0xffffffff81c9b9c2 0xffffffffc06fd8e5 0xffffffff81d5d462 0xffffffff81d60c55 0xffffffff81d60edf 0xffffffff81a043ab 0xffffffff824000a9 2023-01-24T14:55:19.149 INFO:tasks.ceph.osd.0.smithi002.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff81e3c314 0xffffffff81e2eb5e 0xffffffff81d60c17 0xffffffff81d60edf 0xffffffff81a043ab 0xffffffff824000a9 2023-01-24T14:55:19.149 INFO:tasks.ceph.osd.0.smithi002.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff81a0436e 0xffffffff824000a9 2023-01-24T14:55:19.150 INFO:tasks.ceph.osd.0.smithi002.stderr:kernel callstack: 2023-01-24T14:55:19.150 INFO:tasks.ceph.osd.0.smithi002.stderr:kernel callstack: 2023-01-24T14:55:19.150 INFO:tasks.ceph.osd.0.smithi002.stderr:kernel callstack: 2023-01-24T14:55:19.150 INFO:tasks.ceph.osd.0.smithi002.stderr:kernel callstack: 0xffffffffffffff80 0xffffffffc070a1ea 0xffffffffc070a435 0xffffffffc070aad2 0xffffffffc070e972 0xffffffffc070ecd7 0xffffffffc0714099 0xffffffff81d95291 0xffffffff81da3d67 0xffffffffc0712f65 0xffffffff81c9671e 0xffffffff81c9b9c2 0xffffffffc06fd8e5 0xffffffff81d5d462 0xffffffff81d60c55 0xffffffff81d60edf 0xffffffff81a043ab 0xffffffff824000a9 2023-01-24T14:55:19.150 INFO:tasks.ceph.osd.0.smithi002.stderr:kernel callstack: 2023-01-24T14:55:19.151 INFO:tasks.ceph.osd.0.smithi002.stderr:kernel callstack: 2023-01-24T14:55:19.151 INFO:tasks.ceph.osd.0.smithi002.stderr:kernel callstack: 2023-01-24T14:55:19.151 INFO:tasks.ceph.osd.0.smithi002.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff81a0436e 0xffffffff824000a9 2023-01-24T14:55:19.151 INFO:tasks.ceph.osd.0.smithi002.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff81a7c919 0xffffffff81a7cb47 0xffffffff8240114e 2023-01-24T14:55:19.151 INFO:tasks.ceph.osd.0.smithi002.stderr:kernel callstack: 2023-01-24T14:55:19.152 INFO:tasks.ceph.osd.0.smithi002.stderr:kernel callstack: 2023-01-24T15:09:27.272 INFO:tasks.workunit.client.0.smithi002.stderr:test_rados.TestObject.test_seek ... ok 2023-01-24T15:09:28.941 INFO:tasks.ceph.osd.2.smithi039.stderr:================================================================= 2023-01-24T15:09:28.941 INFO:tasks.ceph.osd.2.smithi039.stderr:==102207==ERROR: AddressSanitizer: stack-buffer-overflow on address 0x7f26f91fd730 at pc 0x7f272b132107 bp 0x7f26f91fd6f0 sp 0x7f26f91fce98 2023-01-24T15:09:28.942 INFO:tasks.ceph.osd.2.smithi039.stderr:WRITE of size 24 at 0x7f26f91fd730 thread T0 2023-01-24T15:09:29.001 INFO:tasks.ceph.osd.2.smithi039.stderr:Reactor stalled for 76 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x2912991f5e89 0x2912991f5edd 0x2912991f5edd 0x2912991f5edd 0x291299206184 0x291299206360 0x169dd6 0x43be19cb 0x2912992065d4 0x2912991f5b19 0x2912991f5be5 0x2912991e70a5 0x2912991e8f2a 0xd61c0 0x32402 0xbd8a7 0xbd134 0x54128 0xba5b3 0xc014c 0x3602ee27 0x3602f68b 0x36859bea 0x4518a0ba 0x4518acc4 0x37f3e601 0x36ace54a 0x39336153 0x3933c2b8 0x397cc422 0x36ea196c 0x36ea2c8c 0x394405e9 0x36b13350 0x36b1446c 0x445f3ec8 2023-01-24T15:09:29.001 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T15:09:29.001 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff880f8638 0xffffffff880fbac6 0xffffffff881173d4 0xffffffff880d96c4 0xffffffff880dc2f3 0xffffffff880dc62a 0xffffffff87e7c8b0 0xffffffff87e7cb47 0xffffffff8880114e 2023-01-24T15:09:29.001 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T15:09:29.002 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff880f862c 0xffffffff880fbac6 0xffffffff881173d4 0xffffffff880d96c4 0xffffffff880dc2f3 0xffffffff880dc62a 0xffffffff87e7c8b0 0xffffffff87e7cb47 0xffffffff8880114e 2023-01-24T15:09:29.002 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff887df837 0xffffffff880f6712 0xffffffff880f9378 0xffffffff880fbac6 0xffffffff881173d4 0xffffffff880d96c4 0xffffffff880dc2f3 0xffffffff880dc62a 0xffffffff87e7c8b0 0xffffffff87e7cb47 0xffffffff8880114e 2023-01-24T15:09:29.002 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T15:09:29.002 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T15:09:29.003 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T15:09:29.003 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff88143ca7 0xffffffff88144e2f 0xffffffff880d96e2 0xffffffff880dc2f3 0xffffffff880dc62a 0xffffffff87e7c8b0 0xffffffff87e7cb47 0xffffffff8880114e 2023-01-24T15:09:29.003 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff87e7c919 0xffffffff87e7cb47 0xffffffff8880114e 2023-01-24T15:09:29.003 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T15:09:29.003 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff87e7c919 0xffffffff87e7cb47 0xffffffff8880114e 2023-01-24T15:09:29.004 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T15:09:29.004 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff880d93d4 0xffffffff880dc2f3 0xffffffff880dc62a 0xffffffff87e7c8b0 0xffffffff87e7cb47 0xffffffff8880114e 2023-01-24T15:09:29.004 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T15:09:29.004 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T15:09:29.004 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T15:09:29.005 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff87e7c919 0xffffffff87e7cb47 0xffffffff8880114e 2023-01-24T15:09:29.005 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T15:09:29.005 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T15:09:29.005 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T15:09:29.005 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T15:09:29.006 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff880f7b7b 0xffffffff880a74aa 0xffffffff880e4e84 0xffffffff880e50ec 0xffffffff880dfc51 0xffffffff880e1e6d 0xffffffff880e2158 0xffffffff880e21e7 0xffffffff87e043ab 0xffffffff888000a9 2023-01-24T15:09:29.006 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T15:09:29.006 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff887df837 0xffffffff880f6712 0xffffffff880f9378 0xffffffff880fbac6 0xffffffff881173d4 0xffffffff880d96c4 0xffffffff880dc2f3 0xffffffff880dc62a 0xffffffff87e7c8b0 0xffffffff87e7cb47 0xffffffff8880114e 2023-01-24T15:09:29.006 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T15:09:29.006 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T15:09:29.007 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T15:09:29.007 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T15:09:29.007 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff87e7c919 0xffffffff87e7cb47 0xffffffff8880114e 2023-01-24T15:09:29.007 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T15:09:29.007 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T15:09:29.008 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffffc04ae0c5 0xffffffffc04ce704 0xffffffff880966d2 0xffffffff8809b9c2 0xffffffffc04ba8e5 0xffffffff8815d462 0xffffffff88160c55 0xffffffff88160edf 0xffffffff87e043ab 0xffffffff888000a9 2023-01-24T15:09:29.008 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T15:09:29.008 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T15:09:29.008 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T15:09:29.008 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T15:09:29.009 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T15:09:29.009 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T15:09:29.009 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T15:09:29.009 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff87e7c919 0xffffffff87e7cb47 0xffffffff8880114e 2023-01-24T15:09:29.009 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff87e7c919 0xffffffff87e7cb47 0xffffffff8880114e 2023-01-24T15:09:29.010 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T15:09:29.010 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff880d97f4 0xffffffff880dc2f3 0xffffffff880dc62a 0xffffffff87e7c8b0 0xffffffff87e7cb47 0xffffffff8880114e 2023-01-24T15:09:29.010 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T15:09:29.010 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff880e55bf 0xffffffff880e6245 0xffffffff880e646b 0xffffffff880e6734 0xffffffff880e68fb 0xffffffff87e043ab 0xffffffff888000a9 2023-01-24T15:09:29.010 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T15:09:29.011 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T15:09:29.011 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T15:09:29.011 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T15:09:29.011 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T15:09:29.011 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff87e7c919 0xffffffff87e7cb47 0xffffffff8880114e 2023-01-24T15:09:29.012 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T15:09:29.012 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T15:09:29.012 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffffc04ba770 0xffffffff8815d462 0xffffffff88160c55 0xffffffff88160edf 0xffffffff87e043ab 0xffffffff888000a9 2023-01-24T15:09:29.012 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T15:09:29.012 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T15:09:29.013 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T15:09:29.013 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T15:09:29.013 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T15:09:29.013 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T15:09:29.013 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T15:09:29.014 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T15:09:29.014 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T15:09:29.014 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff887cc7f1 0xffffffff8817f3c6 0xffffffff8817abc5 0xffffffff8817b77d 0xffffffff8817b9cf 0xffffffff881619bf 0xffffffff87f177ea 0xffffffff87e03d1f 0xffffffff87e044ec 0xffffffff888000a9 2023-01-24T15:09:29.014 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T15:09:29.014 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T15:09:29.015 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T15:09:29.015 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T15:09:29.015 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T15:09:29.015 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T15:09:29.015 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff87e7c919 0xffffffff87e7cb47 0xffffffff8880114e 2023-01-24T15:09:29.016 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T15:09:29.016 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T15:09:29.016 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff87e7c919 0xffffffff87e7cb47 0xffffffff8880114e 2023-01-24T15:09:29.016 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff88107979 0xffffffff880d96f7 0xffffffff880dc2f3 0xffffffff880dc62a 0xffffffff87e7c8b0 0xffffffff87e7cb47 0xffffffff8880114e 2023-01-24T15:09:29.016 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T15:09:29.017 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T15:09:29.017 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T15:09:29.017 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T15:09:29.017 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T15:09:29.017 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffffc031d686 0xffffffffc04ae236 0xffffffffc04cff81 0xffffffff8809671e 0xffffffff8809b9c2 0xffffffffc04ba8e5 0xffffffff8815d462 0xffffffff88160c55 0xffffffff88160edf 0xffffffff87e043ab 0xffffffff888000a9 2023-01-24T15:09:29.018 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffffc04cff03 0xffffffff8809671e 0xffffffff8809b9c2 0xffffffffc04ba8e5 0xffffffff8815d462 0xffffffff88160c55 0xffffffff88160edf 0xffffffff87e043ab 0xffffffff888000a9 2023-01-24T15:09:29.018 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff88143d12 0xffffffff88144e2f 0xffffffff880d96e2 0xffffffff880dc2f3 0xffffffff880dc62a 0xffffffff87e7c8b0 0xffffffff87e7cb47 0xffffffff8880114e 2023-01-24T15:09:29.018 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T15:09:29.018 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T15:09:29.018 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T15:09:29.019 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T15:09:29.019 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T15:09:29.019 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T15:09:29.019 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff887f6e91 0xffffffff880a8b21 0xffffffff880a8c48 0xffffffff880d97cf 0xffffffff880dc2f3 0xffffffff880dc62a 0xffffffff87e7c8b0 0xffffffff87e7cb47 0xffffffff8880114e 2023-01-24T15:09:29.019 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff88143d1d 0xffffffff88144e2f 0xffffffff880d96e2 0xffffffff880dc2f3 0xffffffff880dc62a 0xffffffff87e7c8b0 0xffffffff87e7cb47 0xffffffff8880114e 2023-01-24T15:09:29.020 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff887df837 0xffffffff880f6712 0xffffffff880f9378 0xffffffff880fbac6 0xffffffff881173d4 0xffffffff880d96c4 0xffffffff880dc2f3 0xffffffff880dc62a 0xffffffff87e7c8b0 0xffffffff87e7cb47 0xffffffff8880114e 2023-01-24T15:09:29.020 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T15:09:29.020 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff882681f7 0xffffffffc04c736b 0xffffffffc04c7435 0xffffffffc04c7ad2 0xffffffffc04cb972 0xffffffffc04cbcd7 0xffffffffc04d1099 0xffffffff88195291 0xffffffff881a3d67 0xffffffffc04cff65 0xffffffff8809671e 0xffffffff8809b9c2 0xffffffffc04ba8e5 0xffffffff8815d462 0xffffffff88160c55 0xffffffff88160edf 0xffffffff87e043ab 0xffffffff888000a9 2023-01-24T15:09:29.020 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T15:09:29.020 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff880f4bc6 0xffffffff880f7a46 0xffffffff880a74aa 0xffffffff880e4e84 0xffffffff880e50ec 0xffffffff880dfc51 0xffffffff880e1e6d 0xffffffff880e2158 0xffffffff880e21e7 0xffffffff87e043ab 0xffffffff888000a9 2023-01-24T15:09:29.021 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T15:09:29.021 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T15:09:29.021 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T15:09:29.021 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T15:09:29.021 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T15:09:29.021 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T15:09:29.022 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T15:09:29.022 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T15:09:29.022 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffffc015e318 2023-01-24T15:09:29.022 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T15:09:29.022 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T15:09:29.023 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T15:09:29.023 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T15:09:29.023 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T15:09:29.023 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T15:09:29.023 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T15:09:29.024 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T15:09:29.024 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T15:09:29.024 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T15:09:29.024 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T15:09:29.024 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff880f7b7b 0xffffffff880a74aa 0xffffffff880e4e84 0xffffffff880e50ec 0xffffffff880d827a 0xffffffff880fd957 0xffffffff880ff106 0xffffffff87e043ab 0xffffffff888000a9 2023-01-24T15:09:29.025 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T15:09:29.025 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T15:09:29.025 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T15:09:29.025 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T15:09:29.117 INFO:tasks.ceph.osd.2.smithi039.stderr:Reactor stalled for 193 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x291299201cee 0x291299202813 0x2912992056cf 0x291299206360 0x169dd6 0x43be19cb 0x2912992065d4 0x2912991f5b19 0x2912991f5be5 0x2912991e70a5 0x2912991e8f2a 0xd61c0 0x32402 0xbd8a7 0xbd134 0x54128 0xba5b3 0xc014c 0x3602ee27 0x3602f68b 0x36859bea 0x4518a0ba 0x4518acc4 0x37f3e601 0x36ace54a 0x39336153 0x3933c2b8 0x397cc422 0x36ea196c 0x36ea2c8c 0x394405e9 0x36b13350 0x36b1446c 0x445f3ec8 2023-01-24T15:09:29.118 INFO:tasks.ceph.osd.2.smithi039.stderr:kernel callstack: 2023-01-24T15:09:29.213 INFO:tasks.ceph.osd.2.smithi039.stderr: #0 0x7f272b132106 in __interceptor_sigaltstack.part.0 (/lib64/libasan.so.6+0x54106) 2023-01-24T15:09:29.214 INFO:tasks.ceph.osd.2.smithi039.stderr: #1 0x7f272b1985b3 in __asan::PlatformUnpoisonStacks() (/lib64/libasan.so.6+0xba5b3) 2023-01-24T15:09:29.214 INFO:tasks.ceph.osd.2.smithi039.stderr: #2 0x7f272b19e14c in __asan_handle_no_return (/lib64/libasan.so.6+0xc014c) 2023-01-24T15:09:29.216 INFO:tasks.ceph.osd.2.smithi039.stderr: #3 0x5614c8000e27 in void fmt::v9::detail::vformat_to(fmt::v9::detail::buffer&, fmt::v9::basic_string_view, fmt::v9::basic_format_args::type, char>::value, fmt::v9::appender, std::back_insert_iterator::type> > >::type, fmt::v9::type_identity::type> >, fmt::v9::detail::locale_ref) (/usr/bin/ceph-osd+0x3602ee27) 2023-01-24T15:09:29.216 INFO:tasks.ceph.osd.2.smithi039.stderr: #4 0x5614c800168b in seastar::internal::log_buf::inserter_iterator fmt::v9::vformat_to(seastar::internal::log_buf::inserter_iterator, fmt::v9::basic_string_view, fmt::v9::basic_format_args >) (/usr/bin/ceph-osd+0x3602f68b) 2023-01-24T15:09:29.216 INFO:tasks.ceph.osd.2.smithi039.stderr: #5 0x5614c882bbea in seastar::logger::lambda_log_writer(seastar::log_level, seastar::logger::format_info, crimson::osd::ClientRequest&)::{lambda(seastar::internal::log_buf::inserter_iterator)#1}>::operator()(seastar::internal::log_buf::inserter_iterator) (/usr/bin/ceph-osd+0x36859bea) 2023-01-24T15:09:29.216 INFO:tasks.ceph.osd.2.smithi039.stderr: #6 0x5614d715c0ba in seastar::logger::do_log(seastar::log_level, seastar::logger::log_writer&)::{lambda(seastar::internal::log_buf::inserter_iterator)#1}::operator()(seastar::internal::log_buf::inserter_iterator) const (/usr/bin/ceph-osd+0x4518a0ba) 2023-01-24T15:09:29.217 INFO:tasks.ceph.osd.2.smithi039.stderr: #7 0x5614d715ccc4 in seastar::logger::do_log(seastar::log_level, seastar::logger::log_writer&) (/usr/bin/ceph-osd+0x4518acc4) 2023-01-24T15:09:29.217 INFO:tasks.ceph.osd.2.smithi039.stderr: #8 0x5614c9f10601 in crimson::osd::ClientRequest::Orderer::clear_and_cancel() (/usr/bin/ceph-osd+0x37f3e601) 2023-01-24T15:09:29.217 INFO:tasks.ceph.osd.2.smithi039.stderr: #9 0x5614c8aa054a in crimson::osd::PG::on_change(ceph::os::Transaction&) (/usr/bin/ceph-osd+0x36ace54a) 2023-01-24T15:09:29.217 INFO:tasks.ceph.osd.2.smithi039.stderr: #10 0x5614cb308153 in PeeringState::start_peering_interval(crimson::local_shared_foreign_ptr >, std::vector > const&, int, std::vector > const&, int, ceph::os::Transaction&) (/usr/bin/ceph-osd+0x39336153) 2023-01-24T15:09:29.217 INFO:tasks.ceph.osd.2.smithi039.stderr: #11 0x5614cb30e2b8 in PeeringState::Reset::react(PeeringState::AdvMap const&) (/usr/bin/ceph-osd+0x3933c2b8) 2023-01-24T15:09:29.218 INFO:tasks.ceph.osd.2.smithi039.stderr: #12 0x5614cb79e422 in boost::statechart::simple_state, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const&, void const*) (/usr/bin/ceph-osd+0x397cc422) 2023-01-24T15:09:29.218 INFO:tasks.ceph.osd.2.smithi039.stderr: #13 0x5614c8e7396c in boost::statechart::state_machine, boost::statechart::null_exception_translator>::send_event(boost::statechart::event_base const&) (/usr/bin/ceph-osd+0x36ea196c) 2023-01-24T15:09:29.218 INFO:tasks.ceph.osd.2.smithi039.stderr: #14 0x5614c8e74c8c in boost::statechart::state_machine, boost::statechart::null_exception_translator>::process_queued_events() (/usr/bin/ceph-osd+0x36ea2c8c) 2023-01-24T15:09:29.218 INFO:tasks.ceph.osd.2.smithi039.stderr: #15 0x5614cb4125e9 in PeeringState::advance_map(crimson::local_shared_foreign_ptr >, crimson::local_shared_foreign_ptr >, std::vector >&, int, std::vector >&, int, PeeringCtx&) (/usr/bin/ceph-osd+0x394405e9) 2023-01-24T15:09:29.219 INFO:tasks.ceph.osd.2.smithi039.stderr: #16 0x5614c8ae5350 in crimson::osd::PG::handle_advance_map(crimson::local_shared_foreign_ptr >, PeeringCtx&)::{lambda()#1}::operator()() const (/usr/bin/ceph-osd+0x36b13350) 2023-01-24T15:09:29.219 INFO:tasks.ceph.osd.2.smithi039.stderr: #17 0x5614c8ae646c in seastar::noncopyable_function::direct_vtable_for >, PeeringCtx&)::{lambda()#1}>(seastar::thread_attributes, crimson::osd::PG::handle_advance_map(crimson::local_shared_foreign_ptr >, PeeringCtx&)::{lambda()#1}&&)::{lambda()#1}>::call(seastar::noncopyable_function const*) (/usr/bin/ceph-osd+0x36b1446c) 2023-01-24T15:09:29.219 INFO:tasks.ceph.osd.2.smithi039.stderr: #18 0x5614d65c5ec8 in seastar::thread_context::main() (/usr/bin/ceph-osd+0x445f3ec8) 2023-01-24T15:09:29.219 INFO:tasks.ceph.osd.2.smithi039.stderr: 2023-01-24T15:09:29.219 INFO:tasks.ceph.osd.2.smithi039.stderr:0x7f26f91fd730 is located 243504 bytes inside of 262144-byte region [0x7f26f91c2000,0x7f26f9202000) 2023-01-24T15:09:29.220 INFO:tasks.ceph.osd.2.smithi039.stderr:allocated by thread T0 here: 2023-01-24T15:09:29.220 INFO:tasks.ceph.osd.2.smithi039.stderr: #0 0x7f272b1934a7 in aligned_alloc (/lib64/libasan.so.6+0xb54a7) 2023-01-24T15:09:29.220 INFO:tasks.ceph.osd.2.smithi039.stderr: #1 0x5614d65c670c in seastar::thread_context::make_stack(unsigned long) (/usr/bin/ceph-osd+0x445f470c) 2023-01-24T15:09:29.221 INFO:tasks.ceph.osd.2.smithi039.stderr: #2 0xe5d0deaf () 2023-01-24T15:09:29.221 INFO:tasks.ceph.osd.2.smithi039.stderr: 2023-01-24T15:09:29.221 INFO:tasks.ceph.osd.2.smithi039.stderr:SUMMARY: AddressSanitizer: stack-buffer-overflow (/lib64/libasan.so.6+0x54106) in __interceptor_sigaltstack.part.0 2023-01-24T15:09:29.221 INFO:tasks.ceph.osd.2.smithi039.stderr:Shadow bytes around the buggy address: 2023-01-24T15:09:29.222 INFO:tasks.ceph.osd.2.smithi039.stderr: 0x0fe55f237a90: 00 00 00 00 00 00 00 00 00 00 00 00 f1 f1 f1 f1 2023-01-24T15:09:29.222 INFO:tasks.ceph.osd.2.smithi039.stderr: 0x0fe55f237aa0: f1 f1 01 f2 00 f2 f2 f2 00 f2 f2 f2 00 00 f2 f2 2023-01-24T15:09:29.222 INFO:tasks.ceph.osd.2.smithi039.stderr: 0x0fe55f237ab0: 00 00 f2 f2 00 00 f2 f2 00 00 f2 f2 00 00 f2 f2 2023-01-24T15:09:29.222 INFO:tasks.ceph.osd.2.smithi039.stderr: 0x0fe55f237ac0: 00 00 00 00 f2 f2 f2 f2 00 00 00 00 f2 f2 f2 f2 2023-01-24T15:09:29.222 INFO:tasks.ceph.osd.2.smithi039.stderr: 0x0fe55f237ad0: 00 00 00 00 f2 f2 f2 f2 00 00 00 00 f2 f2 f2 f2 2023-01-24T15:09:29.223 INFO:tasks.ceph.osd.2.smithi039.stderr:=>0x0fe55f237ae0: 00 00 f2 f2 00 00[f3]f3 00 00 00 00 00 00 00 00 2023-01-24T15:09:29.223 INFO:tasks.ceph.osd.2.smithi039.stderr: 0x0fe55f237af0: 00 00 00 00 00 00 00 00 f1 f1 f1 f1 f1 f1 01 f2 2023-01-24T15:09:29.223 INFO:tasks.ceph.osd.2.smithi039.stderr: 0x0fe55f237b00: 00 f2 f2 f2 00 f2 f2 f2 00 f2 f2 f2 00 f2 f2 f2 2023-01-24T15:09:29.223 INFO:tasks.ceph.osd.2.smithi039.stderr: 0x0fe55f237b10: 00 f2 f2 f2 00 f2 f2 f2 00 f2 f2 f2 00 f2 f2 f2 2023-01-24T15:09:29.223 INFO:tasks.ceph.osd.2.smithi039.stderr: 0x0fe55f237b20: 00 f2 f2 f2 00 f2 f2 f2 00 00 f2 f2 00 00 f2 f2 2023-01-24T15:09:29.223 INFO:tasks.ceph.osd.2.smithi039.stderr: 0x0fe55f237b30: 00 00 f2 f2 00 00 f2 f2 00 00 f2 f2 00 00 f2 f2 2023-01-24T15:09:29.224 INFO:tasks.ceph.osd.2.smithi039.stderr:Shadow byte legend (one shadow byte represents 8 application bytes): 2023-01-24T15:09:29.224 INFO:tasks.ceph.osd.2.smithi039.stderr: Addressable: 00 2023-01-24T15:09:29.224 INFO:tasks.ceph.osd.2.smithi039.stderr: Partially addressable: 01 02 03 04 05 06 07 2023-01-24T15:09:29.224 INFO:tasks.ceph.osd.2.smithi039.stderr: Heap left redzone: fa 2023-01-24T15:09:29.224 INFO:tasks.ceph.osd.2.smithi039.stderr: Freed heap region: fd 2023-01-24T15:09:29.224 INFO:tasks.ceph.osd.2.smithi039.stderr: Stack left redzone: f1 2023-01-24T15:09:29.225 INFO:tasks.ceph.osd.2.smithi039.stderr: Stack mid redzone: f2 2023-01-24T15:09:29.225 INFO:tasks.ceph.osd.2.smithi039.stderr: Stack right redzone: f3 2023-01-24T15:09:29.225 INFO:tasks.ceph.osd.2.smithi039.stderr: Stack after return: f5 2023-01-24T15:09:29.225 INFO:tasks.ceph.osd.2.smithi039.stderr: Stack use after scope: f8 2023-01-24T15:09:29.225 INFO:tasks.ceph.osd.2.smithi039.stderr: Global redzone: f9 2023-01-24T15:09:29.226 INFO:tasks.ceph.osd.2.smithi039.stderr: Global init order: f6 2023-01-24T15:09:29.226 INFO:tasks.ceph.osd.2.smithi039.stderr: Poisoned by user: f7 2023-01-24T15:09:29.226 INFO:tasks.ceph.osd.2.smithi039.stderr: Container overflow: fc 2023-01-24T15:09:29.226 INFO:tasks.ceph.osd.2.smithi039.stderr: Array cookie: ac 2023-01-24T15:09:29.227 INFO:tasks.ceph.osd.2.smithi039.stderr: Intra object redzone: bb 2023-01-24T15:09:29.227 INFO:tasks.ceph.osd.2.smithi039.stderr: ASan internal: fe 2023-01-24T15:09:29.227 INFO:tasks.ceph.osd.2.smithi039.stderr: Left alloca redzone: ca 2023-01-24T15:09:29.227 INFO:tasks.ceph.osd.2.smithi039.stderr: Right alloca redzone: cb 2023-01-24T15:09:29.227 INFO:tasks.ceph.osd.2.smithi039.stderr: Shadow gap: cc 2023-01-24T15:09:29.227 INFO:tasks.ceph.osd.2.smithi039.stderr:==102207==ABORTING 2023-01-24T15:09:29.492 INFO:tasks.ceph.osd.2.smithi039.stderr:daemon-helper: command failed with exit status 1 2023-01-24T15:09:32.439 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~0s 2023-01-24T15:09:37.943 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~6s 2023-01-24T15:09:43.448 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~11s 2023-01-24T15:09:48.953 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~17s 2023-01-24T15:09:54.457 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~22s 2023-01-24T15:09:57.401 INFO:tasks.ceph.osd.3.smithi039.stderr:Reactor stalled for 42 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x2c5f5 0x2857c 0xb62c3 0x3839210b 0x376fc9bb 0x375f9c0c 0x3760be55 0x3760c8a4 0x378ccc9e 0x378cd474 0x3687c5ba 0x3687f128 0x4449fbfa 0x444a049a 0x444a0a55 0x4449f7fa 0x3671ebc7 0x378e7f29 0x364b52af 0x364b571b 0x362c6842 0x362dcdef 0x362dda11 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:09:57.401 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:09:57.402 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff887f0682 0xffffffff887f0ed5 0xffffffff87f2604a 0xffffffff87e043ab 0xffffffff888000a9 2023-01-24T15:09:57.402 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:09:57.402 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff87fd69ba 0xffffffff87e03f2f 0xffffffff87e0449d 0xffffffff888000a9 2023-01-24T15:09:57.402 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff87e7c919 0xffffffff87e7cb47 0xffffffff8880114e 2023-01-24T15:09:57.403 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:09:57.403 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:09:57.403 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:09:57.403 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:09:57.403 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:09:57.404 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:09:57.404 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:09:57.404 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff87e7c919 0xffffffff87e7cb47 0xffffffff8880114e 2023-01-24T15:09:57.404 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff880f862c 0xffffffff880fbac6 0xffffffff881173d4 0xffffffff880d96c4 0xffffffff880dc2f3 0xffffffff880dc62a 0xffffffff87e7c8b0 0xffffffff87e7cb47 0xffffffff8880114e 2023-01-24T15:09:57.404 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:09:57.405 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:09:57.405 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:09:57.405 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:09:57.405 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:09:57.405 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:09:57.406 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:09:57.406 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:09:57.406 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:09:57.406 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:09:57.406 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:09:57.407 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:09:57.407 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:09:57.407 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff887df837 0xffffffff880f6712 0xffffffff880f9378 0xffffffff880fbac6 0xffffffff881173d4 0xffffffff880d96c4 0xffffffff880dc2f3 0xffffffff880dc62a 0xffffffff87e7c8b0 0xffffffff87e7cb47 0xffffffff8880114e 2023-01-24T15:09:57.407 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:09:57.407 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:09:57.407 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:09:57.408 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:09:57.408 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff88160c6e 0xffffffff88160edf 0xffffffff87e043ab 0xffffffff888000a9 2023-01-24T15:09:57.408 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff880e10c7 0xffffffff880e36d3 0xffffffff880e3edb 0xffffffff880bbed4 0xffffffff880e1369 0xffffffff87e043ab 0xffffffff888000a9 2023-01-24T15:09:57.408 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:09:57.409 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:09:57.409 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:09:57.409 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:09:57.409 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:09:57.409 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:09:57.410 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:09:57.410 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:09:57.410 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:09:57.410 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:09:57.410 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff880ec202 0xffffffff880d75c9 0xffffffff880d8140 0xffffffff880dfc28 0xffffffff880e1e6d 0xffffffff880e2158 0xffffffff880e21e7 0xffffffff87e043ab 0xffffffff888000a9 2023-01-24T15:09:57.411 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff880dbee6 0xffffffff880dc62a 0xffffffff87e7c8b0 0xffffffff87e7cb47 0xffffffff8880114e 2023-01-24T15:09:57.411 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:09:57.411 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:09:57.411 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:09:57.411 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:09:57.411 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:09:57.412 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff88160ccf 0xffffffff88160edf 0xffffffff87e043ab 0xffffffff888000a9 2023-01-24T15:09:57.412 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff880f9378 0xffffffff880fbac6 0xffffffff881173d4 0xffffffff880d96c4 0xffffffff880dc2f3 0xffffffff880dc62a 0xffffffff87e7c8b0 0xffffffff87e7cb47 0xffffffff8880114e 2023-01-24T15:09:57.412 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:09:57.412 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:09:57.412 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:09:57.413 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:09:57.413 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:09:57.413 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:09:57.413 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:09:57.414 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:09:57.414 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:09:57.414 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:09:57.414 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff880f7b7b 0xffffffff880a74aa 0xffffffff880e4e84 0xffffffff880e50ec 0xffffffff880dfc51 0xffffffff880e1e6d 0xffffffff880e2158 0xffffffff880e21e7 0xffffffff87e043ab 0xffffffff888000a9 2023-01-24T15:09:57.414 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:09:57.415 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:09:57.415 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:09:57.415 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:09:57.415 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:09:57.415 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:09:57.416 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:09:57.416 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:09:57.416 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:09:57.416 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:09:57.416 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:09:57.417 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:09:57.417 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:09:57.417 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:09:57.417 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:09:57.417 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:09:57.418 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:09:57.418 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:09:57.418 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:09:57.418 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:09:57.418 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff880f8330 0xffffffff880fbac6 0xffffffff881173d4 0xffffffff880d96c4 0xffffffff880dc2f3 0xffffffff880dc62a 0xffffffff87e7c8b0 0xffffffff87e7cb47 0xffffffff8880114e 2023-01-24T15:09:57.419 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff880f827f 0xffffffff880fbac6 0xffffffff881173d4 0xffffffff880d96c4 0xffffffff880dc2f3 0xffffffff880dc62a 0xffffffff87e7c8b0 0xffffffff87e7cb47 0xffffffff8880114e 2023-01-24T15:09:57.419 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:09:57.419 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:09:57.419 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:09:57.419 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:09:57.420 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:09:57.420 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff880dfe3a 0xffffffff880e1cc2 0xffffffff880e2158 0xffffffff880e21e7 0xffffffff87e043ab 0xffffffff888000a9 2023-01-24T15:09:57.420 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffffc015e2de 2023-01-24T15:09:57.420 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:09:57.420 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:09:57.421 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:09:57.421 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:09:57.421 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:09:57.421 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:09:57.422 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:09:57.422 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff87e7c919 0xffffffff87e7cb47 0xffffffff8880114e 2023-01-24T15:09:57.422 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:09:57.422 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff887f6e91 0xffffffff880a7541 0xffffffff880e4e84 0xffffffff880e50ec 0xffffffff880dfc51 0xffffffff880e1e6d 0xffffffff880e2158 0xffffffff880e21e7 0xffffffff87e043ab 0xffffffff888000a9 2023-01-24T15:09:57.422 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:09:57.423 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:09:57.423 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:09:57.423 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:09:57.423 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff87e7c919 0xffffffff87e7cb47 0xffffffff8880114e 2023-01-24T15:09:57.423 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffffc031d7f5 0xffffffffc04ae236 0xffffffffc04cff81 0xffffffff8809671e 0xffffffff8809b9c2 0xffffffffc04ba8e5 0xffffffff8815d462 0xffffffff88160c55 0xffffffff88160edf 0xffffffff87e043ab 0xffffffff888000a9 2023-01-24T15:09:57.424 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:09:57.424 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:09:57.424 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:09:57.424 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff880e6197 0xffffffff880e646b 0xffffffff880e6734 0xffffffff880e68fb 0xffffffff87e043ab 0xffffffff888000a9 2023-01-24T15:09:57.424 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:09:57.425 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:09:57.425 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:09:57.425 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:09:57.425 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:09:57.425 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:09:57.426 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:09:57.426 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:09:57.426 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:09:57.426 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:09:57.426 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:09:57.427 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:09:57.427 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:09:57.427 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:09:57.427 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:09:57.427 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:09:57.428 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:09:57.428 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:09:57.428 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:09:57.428 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:09:57.428 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:09:57.429 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff887df837 0xffffffff880f6712 0xffffffff880f9378 0xffffffff880fbac6 0xffffffff881173d4 0xffffffff880d96c4 0xffffffff880dc2f3 0xffffffff880dc62a 0xffffffff87e7c8b0 0xffffffff87e7cb47 0xffffffff8880114e 2023-01-24T15:09:57.429 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff887df837 0xffffffff880f6712 0xffffffff880f9378 0xffffffff880fbac6 0xffffffff881173d4 0xffffffff880d96c4 0xffffffff880dc2f3 0xffffffff880dc62a 0xffffffff87e7c8b0 0xffffffff87e7cb47 0xffffffff8880114e 2023-01-24T15:09:57.429 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:09:57.429 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:09:57.429 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff880ccfb0 0xffffffff880dfe06 0xffffffff880e1a02 0xffffffff880fd401 0xffffffff880ff106 0xffffffff87e043ab 0xffffffff888000a9 2023-01-24T15:09:57.430 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff880e3edb 0xffffffff880bbed4 0xffffffff880e1369 0xffffffff87e043ab 0xffffffff888000a9 2023-01-24T15:09:57.430 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff880a8c70 0xffffffff880d97cf 0xffffffff880dc2f3 0xffffffff880dc62a 0xffffffff87e7c8b0 0xffffffff87e7cb47 0xffffffff8880114e 2023-01-24T15:09:57.430 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff87e7c919 0xffffffff87e7cb47 0xffffffff8880114e 2023-01-24T15:09:57.430 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:09:57.430 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:09:57.431 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:09:57.431 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:09:57.431 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:09:57.431 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:09:57.431 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:09:59.962 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~28s 2023-01-24T15:10:05.467 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~33s 2023-01-24T15:10:09.329 INFO:tasks.workunit.client.0.smithi002.stderr:test_rados.TestObject.test_write ... ok 2023-01-24T15:10:09.349 INFO:tasks.workunit.client.0.smithi002.stderr:test_rados.TestRados.test_annotations ... ok 2023-01-24T15:10:10.335 INFO:tasks.workunit.client.0.smithi002.stderr:test_rados.TestRados.test_blocklist_add ... ok 2023-01-24T15:10:10.607 INFO:tasks.ceph.osd.3.smithi039.stderr:Reactor stalled for 36 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0xccf03 0x35f6fe47 0x35f94012 0x11bda3 0x3837f398 0x3c1816a1 0x3780398d 0x37804104 0x37806c18 0x35f19188 0x3602c6e5 0x3602ef36 0x3602f68b 0x37769cbf 0x4518a0ba 0x4518acc4 0x376fcd55 0x375f9c0c 0x3760be55 0x3760c8a4 0x378ccc9e 0x378cd474 0x3687c5ba 0x3687f128 0x4449fbfa 0x444a049a 0x444a0a55 0x4449f7fa 0x3671ebc7 0x378e7f29 0x364b52af 0x364b571b 0x362c6842 0x362dcdef 0x362dda11 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:10:10.607 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:10.608 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:10.608 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:10.608 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:10.609 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:10.609 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:10.609 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:10.609 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:10.609 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:10.610 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:10.610 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:10.610 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:10.610 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff88143d12 0xffffffff88144e2f 0xffffffff880d96e2 0xffffffff880dc2f3 0xffffffff880dc62a 0xffffffff87e7c8b0 0xffffffff87e7cb47 0xffffffff8880114e 2023-01-24T15:10:10.610 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:10.611 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff887df837 0xffffffff880f6712 0xffffffff880f9378 0xffffffff880fbac6 0xffffffff881173d4 0xffffffff880d96c4 0xffffffff880dc2f3 0xffffffff880dc62a 0xffffffff87e7c8b0 0xffffffff87e7cb47 0xffffffff8880114e 2023-01-24T15:10:10.611 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:10.611 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:10.611 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:10.611 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:10.612 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:10.612 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:10.612 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff87e0436e 0xffffffff888000a9 2023-01-24T15:10:10.612 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:10.613 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:10.613 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:10.614 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:10.614 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:10.614 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:10.615 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff887df837 0xffffffff880f6712 0xffffffff880f9378 0xffffffff880fbac6 0xffffffff881173d4 0xffffffff880d96c4 0xffffffff880dc2f3 0xffffffff880dc62a 0xffffffff87e7c8b0 0xffffffff87e7cb47 0xffffffff8880114e 2023-01-24T15:10:10.615 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:10.615 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:10.616 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:10.616 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:10.616 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:10.616 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:10.617 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:10.617 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:10.617 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:10.617 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:10.617 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:10.618 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:10.618 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:10.618 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff880f7b7b 0xffffffff880a74aa 0xffffffff880e4e84 0xffffffff880e50ec 0xffffffff880d827a 0xffffffff880fd957 0xffffffff880ff106 0xffffffff87e043ab 0xffffffff888000a9 2023-01-24T15:10:10.618 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:10.618 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:10.619 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff880bbe00 0xffffffff880e1369 0xffffffff87e043ab 0xffffffff888000a9 2023-01-24T15:10:10.619 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:10.619 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:10.619 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffffc04ba7b7 0xffffffff8815d462 0xffffffff88160c55 0xffffffff88160edf 0xffffffff87e043ab 0xffffffff888000a9 2023-01-24T15:10:10.619 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:10.620 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff87e7c919 0xffffffff87e7cb47 0xffffffff8880114e 2023-01-24T15:10:10.620 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff87e7c919 0xffffffff87e7cb47 0xffffffff8880114e 2023-01-24T15:10:10.620 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff88143d12 0xffffffff88144e2f 0xffffffff880d4b7c 0xffffffff880d844d 0xffffffff880dc2f3 0xffffffff880dc62a 0xffffffff87e7c8b0 0xffffffff87e7cb47 0xffffffff8880114e 2023-01-24T15:10:10.620 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:10.620 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:10.621 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:10.621 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:10.621 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:10.621 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:10.621 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:10.622 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:10.622 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:10.622 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:10.622 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff880d753a 0xffffffff880d8140 0xffffffff880dfc28 0xffffffff880e1e6d 0xffffffff880e2158 0xffffffff880e21e7 0xffffffff87e043ab 0xffffffff888000a9 2023-01-24T15:10:10.622 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff880a8c09 0xffffffff880d97cf 0xffffffff880dc2f3 0xffffffff880dc62a 0xffffffff87e7c8b0 0xffffffff87e7cb47 0xffffffff8880114e 2023-01-24T15:10:10.623 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:10.623 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:10.623 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:10.623 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:10.624 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:10.624 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:10.624 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff87e7c919 0xffffffff87e7cb47 0xffffffff8880114e 2023-01-24T15:10:10.624 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff880f862c 0xffffffff880fbac6 0xffffffff881173d4 0xffffffff880d96c4 0xffffffff880dc2f3 0xffffffff880dc62a 0xffffffff87e7c8b0 0xffffffff87e7cb47 0xffffffff8880114e 2023-01-24T15:10:10.625 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:10.625 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:10.625 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:10.626 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff887df837 0xffffffff880f6712 0xffffffff880f9378 0xffffffff880fbac6 0xffffffff881173d4 0xffffffff880d96c4 0xffffffff880dc2f3 0xffffffff880dc62a 0xffffffff87e7c8b0 0xffffffff87e7cb47 0xffffffff8880114e 2023-01-24T15:10:10.626 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff87e7c919 0xffffffff87e7cb47 0xffffffff8880114e 2023-01-24T15:10:10.626 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff87f56833 0xffffffff87e7c822 0xffffffff87e7cb47 0xffffffff8880114e 2023-01-24T15:10:10.627 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff87e7c919 0xffffffff87e7cb47 0xffffffff8880114e 2023-01-24T15:10:10.627 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff87e7c919 0xffffffff87e7cb47 0xffffffff8880114e 2023-01-24T15:10:10.627 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:10.628 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:10.628 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:10.629 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:10.629 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:10.629 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:10.630 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff87e7c919 0xffffffff87e7cb47 0xffffffff8880114e 2023-01-24T15:10:10.630 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffffc04c7d49 0xffffffffc04cb972 0xffffffffc04cbcd7 0xffffffffc04d1099 0xffffffff88195291 0xffffffff881a3d67 0xffffffffc04cff65 0xffffffff8809671e 0xffffffff8809b9c2 0xffffffffc04ba8e5 0xffffffff8815d462 0xffffffff88160c55 0xffffffff88160edf 0xffffffff87e043ab 0xffffffff888000a9 2023-01-24T15:10:10.630 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:10.631 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:10.631 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:10.631 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff880f8640 0xffffffff880fbac6 0xffffffff881173d4 0xffffffff880d96c4 0xffffffff880dc2f3 0xffffffff880dc62a 0xffffffff87e7c8b0 0xffffffff87e7cb47 0xffffffff8880114e 2023-01-24T15:10:10.632 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:10.632 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:10.633 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:10.633 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff880d753a 0xffffffff880d8140 0xffffffff880dfc28 0xffffffff880e1e6d 0xffffffff880e2158 0xffffffff880e21e7 0xffffffff87e043ab 0xffffffff888000a9 2023-01-24T15:10:10.633 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff88142f14 0xffffffff88143ce0 0xffffffff88144e2f 0xffffffff880d96e2 0xffffffff880dc2f3 0xffffffff880dc62a 0xffffffff87e7c8b0 0xffffffff87e7cb47 0xffffffff8880114e 2023-01-24T15:10:10.634 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:10.634 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff88142ee7 0xffffffff88143ce0 0xffffffff88144e2f 0xffffffff880d96e2 0xffffffff880dc2f3 0xffffffff880dc62a 0xffffffff87e7c8b0 0xffffffff87e7cb47 0xffffffff8880114e 2023-01-24T15:10:10.634 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:10.634 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:10.634 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff887df837 0xffffffff880f6712 0xffffffff880f9378 0xffffffff880fbac6 0xffffffff881173d4 0xffffffff880d96c4 0xffffffff880dc2f3 0xffffffff880dc62a 0xffffffff87e7c8b0 0xffffffff87e7cb47 0xffffffff8880114e 2023-01-24T15:10:10.635 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:10.635 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:10.635 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:10.635 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:10.636 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:10.636 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff88160f23 0xffffffff87e043ab 0xffffffff888000a9 2023-01-24T15:10:10.636 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:10.636 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:10.637 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:10.637 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:10.637 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffffc04c7670 0xffffffffc04cb972 0xffffffffc04cbcd7 0xffffffffc04d1099 0xffffffff88195291 0xffffffff881a3d67 0xffffffffc04cff65 0xffffffff8809671e 0xffffffff8809b9c2 0xffffffffc04ba8e5 0xffffffff8815d462 0xffffffff88160c55 0xffffffff88160edf 0xffffffff87e043ab 0xffffffff888000a9 2023-01-24T15:10:10.637 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff880f862c 0xffffffff880fbac6 0xffffffff881173d4 0xffffffff880d96c4 0xffffffff880dc2f3 0xffffffff880dc62a 0xffffffff87e7c8b0 0xffffffff87e7cb47 0xffffffff8880114e 2023-01-24T15:10:10.637 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:10.638 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff87e7c919 0xffffffff87e7cb47 0xffffffff8880114e 2023-01-24T15:10:10.638 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:10.638 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:10.638 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:10.638 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff887f6e91 0xffffffff880a8b21 0xffffffff880a8c48 0xffffffff880d97cf 0xffffffff880dc2f3 0xffffffff880dc62a 0xffffffff87e7c8b0 0xffffffff87e7cb47 0xffffffff8880114e 2023-01-24T15:10:10.639 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:10.639 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff87e7fca9 0xffffffff880e4ef9 0xffffffff880e50ec 0xffffffff880d827a 0xffffffff880fd957 0xffffffff880ff106 0xffffffff87e043ab 0xffffffff888000a9 2023-01-24T15:10:10.639 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:10.972 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~39s 2023-01-24T15:10:12.216 INFO:tasks.workunit.client.0.smithi002.stderr:test_rados.TestRados.test_create ... ok 2023-01-24T15:10:14.223 INFO:tasks.workunit.client.0.smithi002.stderr:test_rados.TestRados.test_create_utf8 ... ok 2023-01-24T15:10:16.228 INFO:tasks.workunit.client.0.smithi002.stderr:test_rados.TestRados.test_eexist ... ok 2023-01-24T15:10:16.250 INFO:tasks.workunit.client.0.smithi002.stderr:test_rados.TestRados.test_get_fsid ... ok 2023-01-24T15:10:16.476 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~44s 2023-01-24T15:10:19.281 INFO:tasks.ceph.osd.3.smithi039.stderr:Reactor stalled for 43 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0xae1f6 0x3760c8b0 0x378ccc9e 0x378cd474 0x3687c5ba 0x3687f128 0x4449fbfa 0x444a049a 0x444a0a55 0x4449f7fa 0x3671ebc7 0x378e7f29 0x364b52af 0x364b571b 0x362c6842 0x362dcdef 0x362dda11 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:10:19.282 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:19.282 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:19.282 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:19.283 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:19.283 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff887f6e91 0xffffffff880a8b21 0xffffffff880a8c48 0xffffffff880d97cf 0xffffffff880dc2f3 0xffffffff880dc62a 0xffffffff87e7c8b0 0xffffffff87e7cb47 0xffffffff8880114e 2023-01-24T15:10:19.283 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffffc04c7414 0xffffffffc04c7ad2 0xffffffffc04cb972 0xffffffffc04cbcd7 0xffffffffc04d1099 0xffffffff88195291 0xffffffff881a3d67 0xffffffffc04cff65 0xffffffff8809671e 0xffffffff8809b9c2 0xffffffffc04ba8e5 0xffffffff8815d462 0xffffffff88160c55 0xffffffff88160edf 0xffffffff87e043ab 0xffffffff888000a9 2023-01-24T15:10:19.283 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:19.283 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff8813f486 0xffffffff88144e1f 0xffffffff880d96e2 0xffffffff880dc2f3 0xffffffff880dc62a 0xffffffff87e7c8b0 0xffffffff87e7cb47 0xffffffff8880114e 2023-01-24T15:10:19.284 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:19.284 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:19.284 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff880d795b 0xffffffff880d8140 0xffffffff880dfc28 0xffffffff880e1e6d 0xffffffff880e2158 0xffffffff880e21e7 0xffffffff87e043ab 0xffffffff888000a9 2023-01-24T15:10:19.284 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:19.284 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:19.285 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:19.285 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:19.285 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:19.285 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:19.285 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:19.286 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:19.286 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:19.286 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:19.286 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:19.286 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:19.287 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff887df837 0xffffffff880f6712 0xffffffff880f9378 0xffffffff880fbac6 0xffffffff881173d4 0xffffffff880d96c4 0xffffffff880dc2f3 0xffffffff880dc62a 0xffffffff87e7c8b0 0xffffffff87e7cb47 0xffffffff8880114e 2023-01-24T15:10:19.287 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff8816256b 0xffffffff88160cf9 0xffffffff88160edf 0xffffffff87e043ab 0xffffffff888000a9 2023-01-24T15:10:19.287 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:19.287 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:19.287 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:19.288 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:19.288 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:19.288 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:19.288 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:19.288 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:19.289 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:19.289 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:19.289 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:19.289 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:19.289 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:19.290 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:19.290 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:19.290 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:19.290 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:19.290 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:19.291 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:19.291 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:19.291 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:19.291 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff87e7c919 0xffffffff87e7cb47 0xffffffff8880114e 2023-01-24T15:10:19.291 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:19.291 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:19.292 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:19.292 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff87e7c919 0xffffffff87e7cb47 0xffffffff8880114e 2023-01-24T15:10:19.292 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff887f7040 0xffffffffc031c4b4 0xffffffffc031cdce 0xffffffffc04ae12e 0xffffffffc04ce704 0xffffffff880966d2 0xffffffff8809b9c2 0xffffffffc04ba8e5 0xffffffff8815d462 0xffffffff88160c55 0xffffffff88160edf 0xffffffff87e043ab 0xffffffff888000a9 2023-01-24T15:10:19.292 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:19.293 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:19.293 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff88141640 0xffffffff881206f1 0xffffffff88126d06 0xffffffff880ed23d 0xffffffff880e098e 0xffffffff880e1278 0xffffffff880e641b 0xffffffff880e6734 0xffffffff880e68fb 0xffffffff87e043ab 0xffffffff888000a9 2023-01-24T15:10:19.293 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:19.293 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:19.293 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff88160f30 0xffffffff87e043ab 0xffffffff888000a9 2023-01-24T15:10:19.294 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:19.294 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff87e0436e 0xffffffff888000a9 2023-01-24T15:10:19.294 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:19.294 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:19.294 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff880dc47e 0xffffffff880dc62a 0xffffffff87e7c8b0 0xffffffff87e7cb47 0xffffffff8880114e 2023-01-24T15:10:19.294 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:19.295 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff87f7dd46 0xffffffff887f0fc5 0xffffffff880fbb54 0xffffffff881173d4 0xffffffff880d96c4 0xffffffff880dc2f3 0xffffffff880dc62a 0xffffffff87e7c8b0 0xffffffff87e7cb47 0xffffffff8880114e 2023-01-24T15:10:19.295 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:19.295 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:19.295 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:19.295 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:19.296 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:19.296 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:19.296 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:19.296 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:19.296 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff87e0436e 0xffffffff888000a9 2023-01-24T15:10:19.297 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:19.297 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff88a01440 2023-01-24T15:10:19.297 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:19.297 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:19.297 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:19.298 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff880fbb05 0xffffffff881173d4 0xffffffff880d96c4 0xffffffff880dc2f3 0xffffffff880dc62a 0xffffffff87e7c8b0 0xffffffff87e7cb47 0xffffffff8880114e 2023-01-24T15:10:19.298 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:19.298 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:19.298 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:19.298 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:19.298 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:19.299 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff887df837 0xffffffff880f6712 0xffffffff880f9378 0xffffffff880fbac6 0xffffffff881173d4 0xffffffff880d96c4 0xffffffff880dc2f3 0xffffffff880dc62a 0xffffffff87e7c8b0 0xffffffff87e7cb47 0xffffffff8880114e 2023-01-24T15:10:19.299 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:19.299 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:19.299 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:19.299 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:19.300 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:19.300 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:19.300 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff887df837 0xffffffff880f6712 0xffffffff880f9378 0xffffffff880fbac6 0xffffffff881173d4 0xffffffff880d96c4 0xffffffff880dc2f3 0xffffffff880dc62a 0xffffffff87e7c8b0 0xffffffff87e7cb47 0xffffffff8880114e 2023-01-24T15:10:19.300 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:19.300 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:19.301 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:19.301 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:19.301 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:19.301 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:19.301 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:19.302 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:19.302 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:19.302 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:19.302 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff87f56ba9 0xffffffff87e7c8c4 0xffffffff87e7cb47 0xffffffff8880114e 2023-01-24T15:10:19.302 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:19.303 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:19.303 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:19.303 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:19.303 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:19.303 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:19.304 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff887df837 0xffffffff880f6712 0xffffffff880f9378 0xffffffff880fbac6 0xffffffff881173d4 0xffffffff880d96c4 0xffffffff880dc2f3 0xffffffff880dc62a 0xffffffff87e7c8b0 0xffffffff87e7cb47 0xffffffff8880114e 2023-01-24T15:10:19.304 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:19.304 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:19.304 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:19.304 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff880e03a0 0xffffffff880e0b79 0xffffffff880e1bc3 0xffffffff880e6529 0xffffffff880e6734 0xffffffff880e68fb 0xffffffff87e043ab 0xffffffff888000a9 2023-01-24T15:10:19.305 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:19.305 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:19.305 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:19.305 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff880d753a 0xffffffff880d8140 0xffffffff880dfc28 0xffffffff880e1e6d 0xffffffff880e2158 0xffffffff880e21e7 0xffffffff87e043ab 0xffffffff888000a9 2023-01-24T15:10:19.305 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:19.305 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:19.306 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff887f7180 0xffffffff880d976f 0xffffffff880dc2f3 0xffffffff880dc62a 0xffffffff87e7c8b0 0xffffffff87e7cb47 0xffffffff8880114e 2023-01-24T15:10:19.306 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff87e7c919 0xffffffff87e7cb47 0xffffffff8880114e 2023-01-24T15:10:19.306 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:19.306 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:19.306 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:19.307 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:19.307 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:19.307 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:19.307 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:19.307 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:19.307 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff87e7fca9 0xffffffff880e4ef9 0xffffffff880e50ec 0xffffffff880d827a 0xffffffff880fd957 0xffffffff880ff106 0xffffffff87e043ab 0xffffffff888000a9 2023-01-24T15:10:19.308 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:19.308 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff887df837 0xffffffff880f6712 0xffffffff880f9378 0xffffffff880fbac6 0xffffffff881173d4 0xffffffff880d96c4 0xffffffff880dc2f3 0xffffffff880dc62a 0xffffffff87e7c8b0 0xffffffff87e7cb47 0xffffffff8880114e 2023-01-24T15:10:19.308 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:19.308 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:19.308 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:19.309 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:19.309 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:19.309 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:21.980 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~50s 2023-01-24T15:10:24.227 INFO:tasks.workunit.client.0.smithi002.stderr:test_rados.TestRados.test_list_pools ... ok 2023-01-24T15:10:24.248 INFO:tasks.workunit.client.0.smithi002.stderr:test_rados.TestRados.test_monitor_log ... ok 2023-01-24T15:10:24.271 INFO:tasks.workunit.client.0.smithi002.stderr:test_rados.TestRados.test_ping_monitor ... ok 2023-01-24T15:10:26.231 INFO:tasks.workunit.client.0.smithi002.stderr:test_rados.TestRados.test_pool_lookup_utf8 ... ok 2023-01-24T15:10:26.242 INFO:tasks.workunit.client.0.smithi002.stderr:test_rados.TestRadosStateError.test_configuring ... ok 2023-01-24T15:10:26.259 INFO:tasks.workunit.client.0.smithi002.stderr:test_rados.TestRadosStateError.test_connected ... ok 2023-01-24T15:10:26.277 INFO:tasks.workunit.client.0.smithi002.stderr:test_rados.TestRadosStateError.test_shutdown ... ok 2023-01-24T15:10:27.483 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~55s 2023-01-24T15:10:32.987 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~61s 2023-01-24T15:10:38.492 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~66s 2023-01-24T15:10:43.997 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~72s 2023-01-24T15:10:49.501 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~77s 2023-01-24T15:10:55.006 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~83s 2023-01-24T15:10:57.311 INFO:tasks.workunit.client.0.smithi002.stderr:test_rados.TestWatchNotify.test ... ok 2023-01-24T15:10:59.151 INFO:tasks.ceph.osd.3.smithi039.stderr:Reactor stalled for 37 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x3600083d 0x36029d50 0x3602b6aa 0x3602ef36 0x3602f68b 0x45189bed 0x4518acc4 0x376fcd55 0x375f9c0c 0x3760be55 0x3760c8a4 0x378ccc9e 0x378cd474 0x3687c5ba 0x3687f128 0x4449fbfa 0x444a049a 0x444a0a55 0x4449f7fa 0x3671ebc7 0x378e7f29 0x364b52af 0x364b571b 0x362c6842 0x362dcdef 0x362dda11 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:10:59.151 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:59.152 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:59.152 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:59.153 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff88142ed3 0xffffffff88143ce0 0xffffffff88144e2f 0xffffffff880d96e2 0xffffffff880dc2f3 0xffffffff880dc62a 0xffffffff87e7c8b0 0xffffffff87e7cb47 0xffffffff8880114e 2023-01-24T15:10:59.153 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:59.153 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:59.154 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:59.154 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff8822ee69 0xffffffff880e67f5 0xffffffff880e68fb 0xffffffff87e043ab 0xffffffff888000a9 2023-01-24T15:10:59.154 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:59.155 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff887f6e91 0xffffffff880a8b21 0xffffffff880a8c48 0xffffffff880d97cf 0xffffffff880dc2f3 0xffffffff880dc62a 0xffffffff87e7c8b0 0xffffffff87e7cb47 0xffffffff8880114e 2023-01-24T15:10:59.155 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:59.155 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:59.156 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:59.156 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:59.157 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:59.157 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:59.158 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff88126c4e 0xffffffff880ed23d 0xffffffff880e098e 0xffffffff880e1278 0xffffffff880e641b 0xffffffff880e6734 0xffffffff880e68fb 0xffffffff87e043ab 0xffffffff888000a9 2023-01-24T15:10:59.158 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff880d795b 0xffffffff880d8140 0xffffffff880dfc28 0xffffffff880e1e6d 0xffffffff880e2158 0xffffffff880e21e7 0xffffffff87e043ab 0xffffffff888000a9 2023-01-24T15:10:59.158 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:59.159 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:59.159 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:59.159 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:59.160 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:59.160 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:59.161 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:59.161 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff880d753a 0xffffffff880d8140 0xffffffff880dfc28 0xffffffff880e1e6d 0xffffffff880e2158 0xffffffff880e21e7 0xffffffff87e043ab 0xffffffff888000a9 2023-01-24T15:10:59.161 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:59.162 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:59.162 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:59.162 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff87f56c27 0xffffffff880e207e 0xffffffff880e2158 0xffffffff880e21e7 0xffffffff87e043ab 0xffffffff888000a9 2023-01-24T15:10:59.163 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff887df837 0xffffffff880f6712 0xffffffff880f9378 0xffffffff880fbac6 0xffffffff881173d4 0xffffffff880d96c4 0xffffffff880dc2f3 0xffffffff880dc62a 0xffffffff87e7c8b0 0xffffffff87e7cb47 0xffffffff8880114e 2023-01-24T15:10:59.163 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff87e7c919 0xffffffff87e7cb47 0xffffffff8880114e 2023-01-24T15:10:59.164 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:59.164 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff880d753a 0xffffffff880d8140 0xffffffff880dfc28 0xffffffff880e1e6d 0xffffffff880e2158 0xffffffff880e21e7 0xffffffff87e043ab 0xffffffff888000a9 2023-01-24T15:10:59.164 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:59.165 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:59.165 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:59.165 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:59.166 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffffc04c7eb3 0xffffffffc04cba22 0xffffffffc04cbcb1 0xffffffffc04d1099 0xffffffff88195291 0xffffffff881a3d67 0xffffffffc04cff65 0xffffffff8809671e 0xffffffff8809b9c2 0xffffffffc04ba8e5 0xffffffff8815d462 0xffffffff88160c55 0xffffffff88160edf 0xffffffff87e043ab 0xffffffff888000a9 2023-01-24T15:10:59.166 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff887df837 0xffffffff880f6712 0xffffffff880f9378 0xffffffff880fbac6 0xffffffff881173d4 0xffffffff880d96c4 0xffffffff880dc2f3 0xffffffff880dc62a 0xffffffff87e7c8b0 0xffffffff87e7cb47 0xffffffff8880114e 2023-01-24T15:10:59.166 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:59.167 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff880e5c2b 0xffffffff880e6245 0xffffffff880e646b 0xffffffff880e6734 0xffffffff880e68fb 0xffffffff87e043ab 0xffffffff888000a9 2023-01-24T15:10:59.167 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:59.167 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff87e7c919 0xffffffff87e7cb47 0xffffffff8880114e 2023-01-24T15:10:59.168 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:59.168 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:59.168 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:59.169 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff880dfe2b 0xffffffff880e1cc2 0xffffffff880e2158 0xffffffff880e21e7 0xffffffff87e043ab 0xffffffff888000a9 2023-01-24T15:10:59.169 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff881a33f0 0xffffffff881a36c4 0xffffffffc04c7f44 0xffffffffc04cba22 0xffffffffc04cbcb1 0xffffffffc04d1099 0xffffffff88195291 0xffffffff881a3d67 0xffffffffc04cff65 0xffffffff8809671e 0xffffffff8809b9c2 0xffffffffc04ba8e5 0xffffffff8815d462 0xffffffff88160c55 0xffffffff88160edf 0xffffffff87e043ab 0xffffffff888000a9 2023-01-24T15:10:59.169 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:59.169 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:59.170 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:59.170 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff880d406e 0xffffffff880d9715 0xffffffff880dc2f3 0xffffffff880dc62a 0xffffffff87e7c8b0 0xffffffff87e7cb47 0xffffffff8880114e 2023-01-24T15:10:59.170 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:59.170 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:59.171 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:59.171 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:59.171 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:59.171 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:59.171 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:59.172 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:59.172 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:59.172 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff887df837 0xffffffff880f6712 0xffffffff880f9378 0xffffffff880fbac6 0xffffffff881173d4 0xffffffff880d96c4 0xffffffff880dc2f3 0xffffffff880dc62a 0xffffffff87e7c8b0 0xffffffff87e7cb47 0xffffffff8880114e 2023-01-24T15:10:59.172 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:59.173 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff880f862c 0xffffffff880fbac6 0xffffffff880f4aba 0xffffffff880e4c87 0xffffffff880d75f5 0xffffffff880d8140 0xffffffff880dfc28 0xffffffff880e1e6d 0xffffffff880e2158 0xffffffff880e21e7 0xffffffff87e043ab 0xffffffff888000a9 2023-01-24T15:10:59.173 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:59.173 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:59.173 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:59.174 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:59.174 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:59.174 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:59.174 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:59.175 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff887df837 0xffffffff880f6712 0xffffffff880f9378 0xffffffff880fbac6 0xffffffff881173d4 0xffffffff880d96c4 0xffffffff880dc2f3 0xffffffff880dc62a 0xffffffff87e7c8b0 0xffffffff87e7cb47 0xffffffff8880114e 2023-01-24T15:10:59.175 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:59.175 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:59.175 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:59.175 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:59.176 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff88143d12 0xffffffff88144e2f 0xffffffff880d96e2 0xffffffff880dc2f3 0xffffffff880dc62a 0xffffffff87e7c8b0 0xffffffff87e7cb47 0xffffffff8880114e 2023-01-24T15:10:59.176 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:59.176 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:59.176 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:59.176 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:59.177 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:59.177 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:59.177 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:59.177 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:59.177 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff87e7c919 0xffffffff87e7cb47 0xffffffff8880114e 2023-01-24T15:10:59.177 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:59.178 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff87e0436e 0xffffffff888000a9 2023-01-24T15:10:59.178 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:59.178 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff87e7c919 0xffffffff87e7cb47 0xffffffff8880114e 2023-01-24T15:10:59.178 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff880fba48 0xffffffff881173d4 0xffffffff880d96c4 0xffffffff880dc2f3 0xffffffff880dc62a 0xffffffff87e7c8b0 0xffffffff87e7cb47 0xffffffff8880114e 2023-01-24T15:10:59.179 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:59.179 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:59.179 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff87e7c919 0xffffffff87e7cb47 0xffffffff8880114e 2023-01-24T15:10:59.179 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff88125a24 0xffffffff87ef078a 0xffffffff880e1a89 0xffffffff880e6529 0xffffffff880e6734 0xffffffff880e68fb 0xffffffff87e043ab 0xffffffff888000a9 2023-01-24T15:10:59.179 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:59.180 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff880f7b1d 0xffffffff880a74aa 0xffffffff880e4e84 0xffffffff880e50ec 0xffffffff880dfc51 0xffffffff880e1e6d 0xffffffff880e2158 0xffffffff880e21e7 0xffffffff87e043ab 0xffffffff888000a9 2023-01-24T15:10:59.180 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff880eb13d 0xffffffff880d97c4 0xffffffff880dc2f3 0xffffffff880dc62a 0xffffffff87e7c8b0 0xffffffff87e7cb47 0xffffffff8880114e 2023-01-24T15:10:59.180 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:59.181 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:59.181 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:59.181 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:59.182 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:59.183 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:59.183 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:59.183 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:59.184 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:59.184 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:59.184 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:59.185 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:59.185 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:59.185 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:59.185 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:59.186 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:59.186 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:59.186 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff88143d12 0xffffffff88144e2f 0xffffffff880d96e2 0xffffffff880dc2f3 0xffffffff880dc62a 0xffffffff87e7c8b0 0xffffffff87e7cb47 0xffffffff8880114e 2023-01-24T15:10:59.186 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:59.187 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:59.187 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:10:59.187 INFO:tasks.ceph.osd.3.smithi039.stderr:kernel callstack: 2023-01-24T15:11:00.510 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~88s 2023-01-24T15:11:06.015 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~94s 2023-01-24T15:11:10.341 INFO:tasks.workunit.client.0.smithi002.stderr:test_rados.TestWatchNotify.test_aio_notify ... ok 2023-01-24T15:11:10.351 INFO:tasks.workunit.client.0.smithi002.stderr:test_rados.test_rados_init_error ... ok 2023-01-24T15:11:10.422 INFO:tasks.workunit.client.0.smithi002.stderr:test_rados.test_rados_init ... ok 2023-01-24T15:11:10.440 INFO:tasks.workunit.client.0.smithi002.stderr:test_rados.test_ioctx_context_manager ... ok 2023-01-24T15:11:10.452 INFO:tasks.workunit.client.0.smithi002.stderr:test_rados.test_parse_argv ... ok 2023-01-24T15:11:10.458 INFO:tasks.workunit.client.0.smithi002.stderr:test_rados.test_parse_argv_empty_str ... ok 2023-01-24T15:11:10.458 INFO:tasks.workunit.client.0.smithi002.stderr: 2023-01-24T15:11:10.459 INFO:tasks.workunit.client.0.smithi002.stderr:---------------------------------------------------------------------- 2023-01-24T15:11:10.459 INFO:tasks.workunit.client.0.smithi002.stderr:Ran 83 tests in 1654.596s 2023-01-24T15:11:10.459 INFO:tasks.workunit.client.0.smithi002.stderr: 2023-01-24T15:11:10.459 INFO:tasks.workunit.client.0.smithi002.stderr:OK 2023-01-24T15:11:10.478 INFO:tasks.workunit.client.0.smithi002.stderr:+ exit 0 2023-01-24T15:11:10.479 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-01-24T15:11:10.479 DEBUG:teuthology.orchestra.run.smithi002:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-01-24T15:11:10.552 INFO:tasks.workunit:Stopping ["rados/test_python.sh --eval-attr 'not (wait or tier or ec or bench or stats)'"] on client.0... 2023-01-24T15:11:10.553 DEBUG:teuthology.orchestra.run.smithi002:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2023-01-24T15:11:10.874 DEBUG:teuthology.parallel:result is None 2023-01-24T15:11:10.876 DEBUG:teuthology.orchestra.run.smithi002:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2023-01-24T15:11:10.907 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2023-01-24T15:11:10.907 DEBUG:teuthology.orchestra.run.smithi002:> rmdir -- /home/ubuntu/cephtest/mnt.0 2023-01-24T15:11:10.966 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.0/client.0 2023-01-24T15:11:10.966 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-01-24T15:11:10.977 INFO:tasks.ceph.ceph_manager.ceph:Canceling any pending splits or merges... 2023-01-24T15:11:10.978 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-01-24T15:11:11.352 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-01-24T15:11:11.353 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":540,"fsid":"3808514d-c43d-4c1f-b296-499f8064f677","created":"2023-01-24T14:42:31.895770+0000","modified":"2023-01-24T15:11:10.336304+0000","last_up_change":"2023-01-24T15:09:56.216195+0000","last_in_change":"2023-01-24T14:42:33.243250+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":74,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-01-24T14:42:37.919219+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-01-24T14:42:41.363171+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"é»…","create_time":"2023-01-24T14:43:35.909451+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"5e1a8f93-714b-43f5-abb0-337ed2dff81e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":534,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6803","nonce":102589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":102589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":102589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":102589}]},"public_addr":"172.21.15.2:6803/102589","cluster_addr":"172.21.15.2:6802/102589","heartbeat_back_addr":"172.21.15.2:6804/102589","heartbeat_front_addr":"172.21.15.2:6806/102589","state":["exists","up"]},{"osd":1,"uuid":"ef33d692-a846-4999-8754-c88dc16a9e47","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":535,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6800","nonce":102590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6801","nonce":102590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6807","nonce":102590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6805","nonce":102590}]},"public_addr":"172.21.15.2:6800/102590","cluster_addr":"172.21.15.2:6801/102590","heartbeat_back_addr":"172.21.15.2:6807/102590","heartbeat_front_addr":"172.21.15.2:6805/102590","state":["exists","up"]},{"osd":2,"uuid":"b03e5360-6d49-4073-804f-90be8c39d8eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":453,"down_at":498,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6803","nonce":102207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":102207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6807","nonce":102207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":102207}]},"public_addr":"172.21.15.39:6803/102207","cluster_addr":"172.21.15.39:6802/102207","heartbeat_back_addr":"172.21.15.39:6807/102207","heartbeat_front_addr":"172.21.15.39:6806/102207","state":["exists"]},{"osd":3,"uuid":"4a337027-3e3d-4d6a-914e-2d8b4ff89ddc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":534,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6801","nonce":102208}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":102208}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6805","nonce":102208}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":102208}]},"public_addr":"172.21.15.39:6801/102208","cluster_addr":"172.21.15.39:6800/102208","heartbeat_back_addr":"172.21.15.39:6805/102208","heartbeat_front_addr":"172.21.15.39:6804/102208","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"2023-01-24T15:09:56.216195+0000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":4,"snaps":[{"begin":2,"length":2}]},{"pool":22,"snaps":[{"begin":1,"length":1}]},{"pool":43,"snaps":[{"begin":1,"length":1}]},{"pool":47,"snaps":[{"begin":1,"length":1}]},{"pool":48,"snaps":[{"begin":1,"length":1}]},{"pool":49,"snaps":[{"begin":1,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-01-24T15:11:11.368 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-01-24T15:11:11.369 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-01-24T15:11:11.519 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~99s 2023-01-24T15:11:11.712 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-01-24T15:11:11.714 INFO:teuthology.orchestra.run.smithi002.stderr:dumped all 2023-01-24T15:11:11.726 INFO:teuthology.orchestra.run.smithi002.stdout:{"pg_ready":true,"pg_map":{"version":1389,"stamp":"2023-01-24T15:11:11.391081+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":39,"num_read_kb":29,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":41,"acting":41,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":419430400,"kb_used":111600,"kb_used_data":9808,"kb_used_omap":0,"kb_used_meta":101760,"kb_avail":419318800,"statfs":{"total":429496729600,"available":429382451200,"internally_reserved":0,"allocated":10043392,"data_stored":8123935,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":104202240},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-1,"num_write_kb":-1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.157534"},"pg_stats":[{"pgid":"3.f","version":"0'0","reported_seq":397,"reported_epoch":536,"state":"active+clean","last_fresh":"2023-01-24T15:11:01.451955+0000","last_change":"2023-01-24T14:43:37.151318+0000","last_active":"2023-01-24T15:11:01.451955+0000","last_peered":"2023-01-24T15:11:01.451955+0000","last_clean":"2023-01-24T15:11:01.451955+0000","last_became_active":"2023-01-24T14:43:37.134254+0000","last_became_peered":"2023-01-24T14:43:37.134254+0000","last_unstale":"2023-01-24T15:11:01.451955+0000","last_undegraded":"2023-01-24T15:11:01.451955+0000","last_fullsized":"2023-01-24T15:11:01.451955+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T14:43:35.924477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T14:43:35.924477+0000","last_clean_scrub_stamp":"2023-01-24T14:43:35.924477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":28,"reported_epoch":538,"state":"active+undersized","last_fresh":"2023-01-24T15:11:04.568949+0000","last_change":"2023-01-24T15:10:01.868687+0000","last_active":"2023-01-24T15:11:04.568949+0000","last_peered":"2023-01-24T15:11:04.568949+0000","last_clean":"2023-01-24T14:43:35.924477+0000","last_became_active":"2023-01-24T15:10:01.868687+0000","last_became_peered":"2023-01-24T15:10:01.868687+0000","last_unstale":"2023-01-24T15:11:04.568949+0000","last_undegraded":"2023-01-24T15:11:04.568949+0000","last_fullsized":"2023-01-24T15:10:00.258846+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T14:43:35.924477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T14:43:35.924477+0000","last_clean_scrub_stamp":"2023-01-24T14:43:35.924477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":393,"reported_epoch":538,"state":"active+clean","last_fresh":"2023-01-24T15:11:08.147866+0000","last_change":"2023-01-24T14:43:37.258438+0000","last_active":"2023-01-24T15:11:08.147866+0000","last_peered":"2023-01-24T15:11:08.147866+0000","last_clean":"2023-01-24T15:11:08.147866+0000","last_became_active":"2023-01-24T14:43:37.248775+0000","last_became_peered":"2023-01-24T14:43:37.248775+0000","last_unstale":"2023-01-24T15:11:08.147866+0000","last_undegraded":"2023-01-24T15:11:08.147866+0000","last_fullsized":"2023-01-24T15:11:08.147866+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T14:43:35.924477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T14:43:35.924477+0000","last_clean_scrub_stamp":"2023-01-24T14:43:35.924477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":397,"reported_epoch":536,"state":"active+clean","last_fresh":"2023-01-24T15:11:00.367196+0000","last_change":"2023-01-24T14:43:37.149037+0000","last_active":"2023-01-24T15:11:00.367196+0000","last_peered":"2023-01-24T15:11:00.367196+0000","last_clean":"2023-01-24T15:11:00.367196+0000","last_became_active":"2023-01-24T14:43:37.127163+0000","last_became_peered":"2023-01-24T14:43:37.127163+0000","last_unstale":"2023-01-24T15:11:00.367196+0000","last_undegraded":"2023-01-24T15:11:00.367196+0000","last_fullsized":"2023-01-24T15:11:00.367196+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T14:43:35.924477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T14:43:35.924477+0000","last_clean_scrub_stamp":"2023-01-24T14:43:35.924477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":360,"reported_epoch":539,"state":"active+clean","last_fresh":"2023-01-24T15:11:08.105890+0000","last_change":"2023-01-24T14:43:37.181952+0000","last_active":"2023-01-24T15:11:08.105890+0000","last_peered":"2023-01-24T15:11:08.105890+0000","last_clean":"2023-01-24T15:11:08.105890+0000","last_became_active":"2023-01-24T14:43:37.166010+0000","last_became_peered":"2023-01-24T14:43:37.166010+0000","last_unstale":"2023-01-24T15:11:08.105890+0000","last_undegraded":"2023-01-24T15:11:08.105890+0000","last_fullsized":"2023-01-24T15:11:08.105890+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T14:43:35.924477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T14:43:35.924477+0000","last_clean_scrub_stamp":"2023-01-24T14:43:35.924477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":397,"reported_epoch":536,"state":"active+clean","last_fresh":"2023-01-24T15:11:01.461527+0000","last_change":"2023-01-24T14:43:37.208750+0000","last_active":"2023-01-24T15:11:01.461527+0000","last_peered":"2023-01-24T15:11:01.461527+0000","last_clean":"2023-01-24T15:11:01.461527+0000","last_became_active":"2023-01-24T14:43:37.193886+0000","last_became_peered":"2023-01-24T14:43:37.193886+0000","last_unstale":"2023-01-24T15:11:01.461527+0000","last_undegraded":"2023-01-24T15:11:01.461527+0000","last_fullsized":"2023-01-24T15:11:01.461527+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T14:43:35.924477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T14:43:35.924477+0000","last_clean_scrub_stamp":"2023-01-24T14:43:35.924477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":397,"reported_epoch":536,"state":"active+clean","last_fresh":"2023-01-24T15:10:59.857138+0000","last_change":"2023-01-24T14:43:37.245196+0000","last_active":"2023-01-24T15:10:59.857138+0000","last_peered":"2023-01-24T15:10:59.857138+0000","last_clean":"2023-01-24T15:10:59.857138+0000","last_became_active":"2023-01-24T14:43:37.233660+0000","last_became_peered":"2023-01-24T14:43:37.233660+0000","last_unstale":"2023-01-24T15:10:59.857138+0000","last_undegraded":"2023-01-24T15:10:59.857138+0000","last_fullsized":"2023-01-24T15:10:59.857138+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T14:43:35.924477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T14:43:35.924477+0000","last_clean_scrub_stamp":"2023-01-24T14:43:35.924477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":393,"reported_epoch":538,"state":"active+clean","last_fresh":"2023-01-24T15:11:08.774830+0000","last_change":"2023-01-24T14:43:37.273633+0000","last_active":"2023-01-24T15:11:08.774830+0000","last_peered":"2023-01-24T15:11:08.774830+0000","last_clean":"2023-01-24T15:11:08.774830+0000","last_became_active":"2023-01-24T14:43:37.267461+0000","last_became_peered":"2023-01-24T14:43:37.267461+0000","last_unstale":"2023-01-24T15:11:08.774830+0000","last_undegraded":"2023-01-24T15:11:08.774830+0000","last_fullsized":"2023-01-24T15:11:08.774830+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T14:43:35.924477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T14:43:35.924477+0000","last_clean_scrub_stamp":"2023-01-24T14:43:35.924477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":35,"reported_epoch":539,"state":"active+undersized","last_fresh":"2023-01-24T15:11:08.631228+0000","last_change":"2023-01-24T15:09:58.515020+0000","last_active":"2023-01-24T15:11:08.631228+0000","last_peered":"2023-01-24T15:11:08.631228+0000","last_clean":"2023-01-24T14:42:42.330702+0000","last_became_active":"2023-01-24T15:09:58.515020+0000","last_became_peered":"2023-01-24T15:09:58.515020+0000","last_unstale":"2023-01-24T15:11:08.631228+0000","last_undegraded":"2023-01-24T15:11:08.631228+0000","last_fullsized":"2023-01-24T15:09:58.124818+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T14:42:42.330702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T14:42:42.330702+0000","last_clean_scrub_stamp":"2023-01-24T14:42:42.330702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":392,"reported_epoch":538,"state":"active+clean","last_fresh":"2023-01-24T15:11:04.771745+0000","last_change":"2023-01-24T14:43:37.209033+0000","last_active":"2023-01-24T15:11:04.771745+0000","last_peered":"2023-01-24T15:11:04.771745+0000","last_clean":"2023-01-24T15:11:04.771745+0000","last_became_active":"2023-01-24T14:43:37.197571+0000","last_became_peered":"2023-01-24T14:43:37.197571+0000","last_unstale":"2023-01-24T15:11:04.771745+0000","last_undegraded":"2023-01-24T15:11:04.771745+0000","last_fullsized":"2023-01-24T15:11:04.771745+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T14:43:35.924477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T14:43:35.924477+0000","last_clean_scrub_stamp":"2023-01-24T14:43:35.924477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":404,"reported_epoch":536,"state":"active+undersized","last_fresh":"2023-01-24T15:11:01.693427+0000","last_change":"2023-01-24T15:10:04.691115+0000","last_active":"2023-01-24T15:11:01.693427+0000","last_peered":"2023-01-24T15:11:01.693427+0000","last_clean":"2023-01-24T15:09:53.938902+0000","last_became_active":"2023-01-24T15:10:04.691115+0000","last_became_peered":"2023-01-24T15:10:04.691115+0000","last_unstale":"2023-01-24T15:11:01.693427+0000","last_undegraded":"2023-01-24T15:11:01.693427+0000","last_fullsized":"2023-01-24T15:10:03.477405+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T14:42:42.330702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T14:42:42.330702+0000","last_clean_scrub_stamp":"2023-01-24T14:42:42.330702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0],"acting":[0],"avail_no_missing":["0"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":26,"reported_epoch":536,"state":"active+undersized","last_fresh":"2023-01-24T15:11:00.156339+0000","last_change":"2023-01-24T15:10:04.707850+0000","last_active":"2023-01-24T15:11:00.156339+0000","last_peered":"2023-01-24T15:11:00.156339+0000","last_clean":"2023-01-24T14:43:35.924477+0000","last_became_active":"2023-01-24T15:10:04.707850+0000","last_became_peered":"2023-01-24T15:10:04.707850+0000","last_unstale":"2023-01-24T15:11:00.156339+0000","last_undegraded":"2023-01-24T15:11:00.156339+0000","last_fullsized":"2023-01-24T15:10:03.753405+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T14:43:35.924477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T14:43:35.924477+0000","last_clean_scrub_stamp":"2023-01-24T14:43:35.924477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0],"acting":[0],"avail_no_missing":["0"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":396,"reported_epoch":538,"state":"active+clean","last_fresh":"2023-01-24T15:11:06.944754+0000","last_change":"2023-01-24T14:42:43.412373+0000","last_active":"2023-01-24T15:11:06.944754+0000","last_peered":"2023-01-24T15:11:06.944754+0000","last_clean":"2023-01-24T15:11:06.944754+0000","last_became_active":"2023-01-24T14:42:43.409737+0000","last_became_peered":"2023-01-24T14:42:43.409737+0000","last_unstale":"2023-01-24T15:11:06.944754+0000","last_undegraded":"2023-01-24T15:11:06.944754+0000","last_fullsized":"2023-01-24T15:11:06.944754+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T14:42:42.330702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T14:42:42.330702+0000","last_clean_scrub_stamp":"2023-01-24T14:42:42.330702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":364,"reported_epoch":539,"state":"active+undersized","last_fresh":"2023-01-24T15:11:08.658988+0000","last_change":"2023-01-24T15:09:58.513607+0000","last_active":"2023-01-24T15:11:08.658988+0000","last_peered":"2023-01-24T15:11:08.658988+0000","last_clean":"2023-01-24T15:09:54.053508+0000","last_became_active":"2023-01-24T15:09:58.513607+0000","last_became_peered":"2023-01-24T15:09:58.513607+0000","last_unstale":"2023-01-24T15:11:08.658988+0000","last_undegraded":"2023-01-24T15:11:08.658988+0000","last_fullsized":"2023-01-24T15:09:58.390668+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T14:43:35.924477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T14:43:35.924477+0000","last_clean_scrub_stamp":"2023-01-24T14:43:35.924477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":395,"reported_epoch":538,"state":"active+clean","last_fresh":"2023-01-24T15:11:04.808692+0000","last_change":"2023-01-24T14:42:43.472889+0000","last_active":"2023-01-24T15:11:04.808692+0000","last_peered":"2023-01-24T15:11:04.808692+0000","last_clean":"2023-01-24T15:11:04.808692+0000","last_became_active":"2023-01-24T14:42:43.469425+0000","last_became_peered":"2023-01-24T14:42:43.469425+0000","last_unstale":"2023-01-24T15:11:04.808692+0000","last_undegraded":"2023-01-24T15:11:04.808692+0000","last_fullsized":"2023-01-24T15:11:04.808692+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T14:42:42.330702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T14:42:42.330702+0000","last_clean_scrub_stamp":"2023-01-24T14:42:42.330702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":463,"reported_epoch":539,"state":"active+clean","last_fresh":"2023-01-24T15:11:07.800091+0000","last_change":"2023-01-24T14:42:39.395501+0000","last_active":"2023-01-24T15:11:07.800091+0000","last_peered":"2023-01-24T15:11:07.800091+0000","last_clean":"2023-01-24T15:11:07.800091+0000","last_became_active":"2023-01-24T14:42:39.392286+0000","last_became_peered":"2023-01-24T14:42:39.392286+0000","last_unstale":"2023-01-24T15:11:07.800091+0000","last_undegraded":"2023-01-24T15:11:07.800091+0000","last_fullsized":"2023-01-24T15:11:07.800091+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T14:42:38.317142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T14:42:38.317142+0000","last_clean_scrub_stamp":"2023-01-24T14:42:38.317142+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":39,"num_read_kb":29,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":392,"reported_epoch":538,"state":"active+clean","last_fresh":"2023-01-24T15:11:05.384241+0000","last_change":"2023-01-24T14:43:37.157359+0000","last_active":"2023-01-24T15:11:05.384241+0000","last_peered":"2023-01-24T15:11:05.384241+0000","last_clean":"2023-01-24T15:11:05.384241+0000","last_became_active":"2023-01-24T14:43:37.148439+0000","last_became_peered":"2023-01-24T14:43:37.148439+0000","last_unstale":"2023-01-24T15:11:05.384241+0000","last_undegraded":"2023-01-24T15:11:05.384241+0000","last_fullsized":"2023-01-24T15:11:05.384241+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T14:43:35.924477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T14:43:35.924477+0000","last_clean_scrub_stamp":"2023-01-24T14:43:35.924477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":400,"reported_epoch":536,"state":"active+clean","last_fresh":"2023-01-24T15:11:00.748521+0000","last_change":"2023-01-24T14:42:43.426068+0000","last_active":"2023-01-24T15:11:00.748521+0000","last_peered":"2023-01-24T15:11:00.748521+0000","last_clean":"2023-01-24T15:11:00.748521+0000","last_became_active":"2023-01-24T14:42:43.422382+0000","last_became_peered":"2023-01-24T14:42:43.422382+0000","last_unstale":"2023-01-24T15:11:00.748521+0000","last_undegraded":"2023-01-24T15:11:00.748521+0000","last_fullsized":"2023-01-24T15:11:00.748521+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T14:42:42.330702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T14:42:42.330702+0000","last_clean_scrub_stamp":"2023-01-24T14:42:42.330702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":360,"reported_epoch":539,"state":"active+clean","last_fresh":"2023-01-24T15:11:08.617907+0000","last_change":"2023-01-24T14:43:37.191358+0000","last_active":"2023-01-24T15:11:08.617907+0000","last_peered":"2023-01-24T15:11:08.617907+0000","last_clean":"2023-01-24T15:11:08.617907+0000","last_became_active":"2023-01-24T14:43:37.187180+0000","last_became_peered":"2023-01-24T14:43:37.187180+0000","last_unstale":"2023-01-24T15:11:08.617907+0000","last_undegraded":"2023-01-24T15:11:08.617907+0000","last_fullsized":"2023-01-24T15:11:08.617907+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T14:43:35.924477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T14:43:35.924477+0000","last_clean_scrub_stamp":"2023-01-24T14:43:35.924477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":367,"reported_epoch":539,"state":"active+undersized","last_fresh":"2023-01-24T15:11:08.229549+0000","last_change":"2023-01-24T15:09:58.519535+0000","last_active":"2023-01-24T15:11:08.229549+0000","last_peered":"2023-01-24T15:11:08.229549+0000","last_clean":"2023-01-24T15:09:53.661175+0000","last_became_active":"2023-01-24T15:09:58.519535+0000","last_became_peered":"2023-01-24T15:09:58.519535+0000","last_unstale":"2023-01-24T15:11:08.229549+0000","last_undegraded":"2023-01-24T15:11:08.229549+0000","last_fullsized":"2023-01-24T15:09:58.385532+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T14:42:42.330702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T14:42:42.330702+0000","last_clean_scrub_stamp":"2023-01-24T14:42:42.330702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":35,"reported_epoch":539,"state":"active+undersized","last_fresh":"2023-01-24T15:11:08.285346+0000","last_change":"2023-01-24T15:09:58.521178+0000","last_active":"2023-01-24T15:11:08.285346+0000","last_peered":"2023-01-24T15:11:08.285346+0000","last_clean":"2023-01-24T14:43:35.924477+0000","last_became_active":"2023-01-24T15:09:58.521178+0000","last_became_peered":"2023-01-24T15:09:58.521178+0000","last_unstale":"2023-01-24T15:11:08.285346+0000","last_undegraded":"2023-01-24T15:11:08.285346+0000","last_fullsized":"2023-01-24T15:09:58.343642+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T14:43:35.924477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T14:43:35.924477+0000","last_clean_scrub_stamp":"2023-01-24T14:43:35.924477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":360,"reported_epoch":539,"state":"active+clean","last_fresh":"2023-01-24T15:11:07.252862+0000","last_change":"2023-01-24T14:43:37.199321+0000","last_active":"2023-01-24T15:11:07.252862+0000","last_peered":"2023-01-24T15:11:07.252862+0000","last_clean":"2023-01-24T15:11:07.252862+0000","last_became_active":"2023-01-24T14:43:37.195467+0000","last_became_peered":"2023-01-24T14:43:37.195467+0000","last_unstale":"2023-01-24T15:11:07.252862+0000","last_undegraded":"2023-01-24T15:11:07.252862+0000","last_fullsized":"2023-01-24T15:11:07.252862+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T14:43:35.924477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T14:43:35.924477+0000","last_clean_scrub_stamp":"2023-01-24T14:43:35.924477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":26,"reported_epoch":536,"state":"active+undersized","last_fresh":"2023-01-24T15:11:01.815311+0000","last_change":"2023-01-24T15:10:04.697357+0000","last_active":"2023-01-24T15:11:01.815311+0000","last_peered":"2023-01-24T15:11:01.815311+0000","last_clean":"2023-01-24T14:42:42.330702+0000","last_became_active":"2023-01-24T15:10:04.697357+0000","last_became_peered":"2023-01-24T15:10:04.697357+0000","last_unstale":"2023-01-24T15:11:01.815311+0000","last_undegraded":"2023-01-24T15:11:01.815311+0000","last_fullsized":"2023-01-24T15:10:04.194347+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T14:42:42.330702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T14:42:42.330702+0000","last_clean_scrub_stamp":"2023-01-24T14:42:42.330702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0],"acting":[0],"avail_no_missing":["0"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":396,"reported_epoch":538,"state":"active+clean","last_fresh":"2023-01-24T15:11:08.458769+0000","last_change":"2023-01-24T14:42:43.463202+0000","last_active":"2023-01-24T15:11:08.458769+0000","last_peered":"2023-01-24T15:11:08.458769+0000","last_clean":"2023-01-24T15:11:08.458769+0000","last_became_active":"2023-01-24T14:42:43.459539+0000","last_became_peered":"2023-01-24T14:42:43.459539+0000","last_unstale":"2023-01-24T15:11:08.458769+0000","last_undegraded":"2023-01-24T15:11:08.458769+0000","last_fullsized":"2023-01-24T15:11:08.458769+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T14:42:42.330702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T14:42:42.330702+0000","last_clean_scrub_stamp":"2023-01-24T14:42:42.330702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":35,"reported_epoch":539,"state":"active+undersized","last_fresh":"2023-01-24T15:11:08.649521+0000","last_change":"2023-01-24T15:09:58.516792+0000","last_active":"2023-01-24T15:11:08.649521+0000","last_peered":"2023-01-24T15:11:08.649521+0000","last_clean":"2023-01-24T14:43:35.924477+0000","last_became_active":"2023-01-24T15:09:58.516792+0000","last_became_peered":"2023-01-24T15:09:58.516792+0000","last_unstale":"2023-01-24T15:11:08.649521+0000","last_undegraded":"2023-01-24T15:11:08.649521+0000","last_fullsized":"2023-01-24T15:09:57.969488+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T14:43:35.924477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T14:43:35.924477+0000","last_clean_scrub_stamp":"2023-01-24T14:43:35.924477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":27,"acting":27,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":39,"num_read_kb":29,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":12,"acting":12,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654707323,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":26052,"kb_used_data":2812,"kb_used_omap":0,"kb_used_meta":23232,"kb_avail":104831548,"statfs":{"total":107374182400,"available":107347505152,"internally_reserved":0,"allocated":2879488,"data_stored":2378605,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":23789568},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":9,"seq":38654707225,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":26912,"kb_used_data":2008,"kb_used_omap":0,"kb_used_meta":24896,"kb_avail":104830688,"statfs":{"total":107374182400,"available":107346624512,"internally_reserved":0,"allocated":2056192,"data_stored":1599296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":25493504},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":8,"seq":34359740011,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":22004,"kb_used_data":2796,"kb_used_omap":0,"kb_used_meta":19200,"kb_avail":104835596,"statfs":{"total":107374182400,"available":107351650304,"internally_reserved":0,"allocated":2863104,"data_stored":2364751,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":19660800},"hb_peers":[1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":8,"seq":34359740019,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":36632,"kb_used_data":2192,"kb_used_omap":0,"kb_used_meta":34432,"kb_avail":104820968,"statfs":{"total":107374182400,"available":107336671232,"internally_reserved":0,"allocated":2244608,"data_stored":1781283,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":35258368},"hb_peers":[0,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2023-01-24T15:11:11.728 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-01-24T15:11:12.067 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-01-24T15:11:12.069 INFO:teuthology.orchestra.run.smithi002.stderr:dumped all 2023-01-24T15:11:12.085 INFO:teuthology.orchestra.run.smithi002.stdout:{"pg_ready":true,"pg_map":{"version":1389,"stamp":"2023-01-24T15:11:11.391081+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":39,"num_read_kb":29,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":41,"acting":41,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":419430400,"kb_used":111600,"kb_used_data":9808,"kb_used_omap":0,"kb_used_meta":101760,"kb_avail":419318800,"statfs":{"total":429496729600,"available":429382451200,"internally_reserved":0,"allocated":10043392,"data_stored":8123935,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":104202240},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-1,"num_write_kb":-1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.157534"},"pg_stats":[{"pgid":"3.f","version":"0'0","reported_seq":397,"reported_epoch":536,"state":"active+clean","last_fresh":"2023-01-24T15:11:01.451955+0000","last_change":"2023-01-24T14:43:37.151318+0000","last_active":"2023-01-24T15:11:01.451955+0000","last_peered":"2023-01-24T15:11:01.451955+0000","last_clean":"2023-01-24T15:11:01.451955+0000","last_became_active":"2023-01-24T14:43:37.134254+0000","last_became_peered":"2023-01-24T14:43:37.134254+0000","last_unstale":"2023-01-24T15:11:01.451955+0000","last_undegraded":"2023-01-24T15:11:01.451955+0000","last_fullsized":"2023-01-24T15:11:01.451955+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T14:43:35.924477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T14:43:35.924477+0000","last_clean_scrub_stamp":"2023-01-24T14:43:35.924477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":28,"reported_epoch":538,"state":"active+undersized","last_fresh":"2023-01-24T15:11:04.568949+0000","last_change":"2023-01-24T15:10:01.868687+0000","last_active":"2023-01-24T15:11:04.568949+0000","last_peered":"2023-01-24T15:11:04.568949+0000","last_clean":"2023-01-24T14:43:35.924477+0000","last_became_active":"2023-01-24T15:10:01.868687+0000","last_became_peered":"2023-01-24T15:10:01.868687+0000","last_unstale":"2023-01-24T15:11:04.568949+0000","last_undegraded":"2023-01-24T15:11:04.568949+0000","last_fullsized":"2023-01-24T15:10:00.258846+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T14:43:35.924477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T14:43:35.924477+0000","last_clean_scrub_stamp":"2023-01-24T14:43:35.924477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":393,"reported_epoch":538,"state":"active+clean","last_fresh":"2023-01-24T15:11:08.147866+0000","last_change":"2023-01-24T14:43:37.258438+0000","last_active":"2023-01-24T15:11:08.147866+0000","last_peered":"2023-01-24T15:11:08.147866+0000","last_clean":"2023-01-24T15:11:08.147866+0000","last_became_active":"2023-01-24T14:43:37.248775+0000","last_became_peered":"2023-01-24T14:43:37.248775+0000","last_unstale":"2023-01-24T15:11:08.147866+0000","last_undegraded":"2023-01-24T15:11:08.147866+0000","last_fullsized":"2023-01-24T15:11:08.147866+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T14:43:35.924477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T14:43:35.924477+0000","last_clean_scrub_stamp":"2023-01-24T14:43:35.924477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":397,"reported_epoch":536,"state":"active+clean","last_fresh":"2023-01-24T15:11:00.367196+0000","last_change":"2023-01-24T14:43:37.149037+0000","last_active":"2023-01-24T15:11:00.367196+0000","last_peered":"2023-01-24T15:11:00.367196+0000","last_clean":"2023-01-24T15:11:00.367196+0000","last_became_active":"2023-01-24T14:43:37.127163+0000","last_became_peered":"2023-01-24T14:43:37.127163+0000","last_unstale":"2023-01-24T15:11:00.367196+0000","last_undegraded":"2023-01-24T15:11:00.367196+0000","last_fullsized":"2023-01-24T15:11:00.367196+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T14:43:35.924477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T14:43:35.924477+0000","last_clean_scrub_stamp":"2023-01-24T14:43:35.924477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":360,"reported_epoch":539,"state":"active+clean","last_fresh":"2023-01-24T15:11:08.105890+0000","last_change":"2023-01-24T14:43:37.181952+0000","last_active":"2023-01-24T15:11:08.105890+0000","last_peered":"2023-01-24T15:11:08.105890+0000","last_clean":"2023-01-24T15:11:08.105890+0000","last_became_active":"2023-01-24T14:43:37.166010+0000","last_became_peered":"2023-01-24T14:43:37.166010+0000","last_unstale":"2023-01-24T15:11:08.105890+0000","last_undegraded":"2023-01-24T15:11:08.105890+0000","last_fullsized":"2023-01-24T15:11:08.105890+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T14:43:35.924477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T14:43:35.924477+0000","last_clean_scrub_stamp":"2023-01-24T14:43:35.924477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":397,"reported_epoch":536,"state":"active+clean","last_fresh":"2023-01-24T15:11:01.461527+0000","last_change":"2023-01-24T14:43:37.208750+0000","last_active":"2023-01-24T15:11:01.461527+0000","last_peered":"2023-01-24T15:11:01.461527+0000","last_clean":"2023-01-24T15:11:01.461527+0000","last_became_active":"2023-01-24T14:43:37.193886+0000","last_became_peered":"2023-01-24T14:43:37.193886+0000","last_unstale":"2023-01-24T15:11:01.461527+0000","last_undegraded":"2023-01-24T15:11:01.461527+0000","last_fullsized":"2023-01-24T15:11:01.461527+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T14:43:35.924477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T14:43:35.924477+0000","last_clean_scrub_stamp":"2023-01-24T14:43:35.924477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":397,"reported_epoch":536,"state":"active+clean","last_fresh":"2023-01-24T15:10:59.857138+0000","last_change":"2023-01-24T14:43:37.245196+0000","last_active":"2023-01-24T15:10:59.857138+0000","last_peered":"2023-01-24T15:10:59.857138+0000","last_clean":"2023-01-24T15:10:59.857138+0000","last_became_active":"2023-01-24T14:43:37.233660+0000","last_became_peered":"2023-01-24T14:43:37.233660+0000","last_unstale":"2023-01-24T15:10:59.857138+0000","last_undegraded":"2023-01-24T15:10:59.857138+0000","last_fullsized":"2023-01-24T15:10:59.857138+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T14:43:35.924477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T14:43:35.924477+0000","last_clean_scrub_stamp":"2023-01-24T14:43:35.924477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":393,"reported_epoch":538,"state":"active+clean","last_fresh":"2023-01-24T15:11:08.774830+0000","last_change":"2023-01-24T14:43:37.273633+0000","last_active":"2023-01-24T15:11:08.774830+0000","last_peered":"2023-01-24T15:11:08.774830+0000","last_clean":"2023-01-24T15:11:08.774830+0000","last_became_active":"2023-01-24T14:43:37.267461+0000","last_became_peered":"2023-01-24T14:43:37.267461+0000","last_unstale":"2023-01-24T15:11:08.774830+0000","last_undegraded":"2023-01-24T15:11:08.774830+0000","last_fullsized":"2023-01-24T15:11:08.774830+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T14:43:35.924477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T14:43:35.924477+0000","last_clean_scrub_stamp":"2023-01-24T14:43:35.924477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":35,"reported_epoch":539,"state":"active+undersized","last_fresh":"2023-01-24T15:11:08.631228+0000","last_change":"2023-01-24T15:09:58.515020+0000","last_active":"2023-01-24T15:11:08.631228+0000","last_peered":"2023-01-24T15:11:08.631228+0000","last_clean":"2023-01-24T14:42:42.330702+0000","last_became_active":"2023-01-24T15:09:58.515020+0000","last_became_peered":"2023-01-24T15:09:58.515020+0000","last_unstale":"2023-01-24T15:11:08.631228+0000","last_undegraded":"2023-01-24T15:11:08.631228+0000","last_fullsized":"2023-01-24T15:09:58.124818+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T14:42:42.330702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T14:42:42.330702+0000","last_clean_scrub_stamp":"2023-01-24T14:42:42.330702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":392,"reported_epoch":538,"state":"active+clean","last_fresh":"2023-01-24T15:11:04.771745+0000","last_change":"2023-01-24T14:43:37.209033+0000","last_active":"2023-01-24T15:11:04.771745+0000","last_peered":"2023-01-24T15:11:04.771745+0000","last_clean":"2023-01-24T15:11:04.771745+0000","last_became_active":"2023-01-24T14:43:37.197571+0000","last_became_peered":"2023-01-24T14:43:37.197571+0000","last_unstale":"2023-01-24T15:11:04.771745+0000","last_undegraded":"2023-01-24T15:11:04.771745+0000","last_fullsized":"2023-01-24T15:11:04.771745+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T14:43:35.924477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T14:43:35.924477+0000","last_clean_scrub_stamp":"2023-01-24T14:43:35.924477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":404,"reported_epoch":536,"state":"active+undersized","last_fresh":"2023-01-24T15:11:01.693427+0000","last_change":"2023-01-24T15:10:04.691115+0000","last_active":"2023-01-24T15:11:01.693427+0000","last_peered":"2023-01-24T15:11:01.693427+0000","last_clean":"2023-01-24T15:09:53.938902+0000","last_became_active":"2023-01-24T15:10:04.691115+0000","last_became_peered":"2023-01-24T15:10:04.691115+0000","last_unstale":"2023-01-24T15:11:01.693427+0000","last_undegraded":"2023-01-24T15:11:01.693427+0000","last_fullsized":"2023-01-24T15:10:03.477405+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T14:42:42.330702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T14:42:42.330702+0000","last_clean_scrub_stamp":"2023-01-24T14:42:42.330702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0],"acting":[0],"avail_no_missing":["0"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":26,"reported_epoch":536,"state":"active+undersized","last_fresh":"2023-01-24T15:11:00.156339+0000","last_change":"2023-01-24T15:10:04.707850+0000","last_active":"2023-01-24T15:11:00.156339+0000","last_peered":"2023-01-24T15:11:00.156339+0000","last_clean":"2023-01-24T14:43:35.924477+0000","last_became_active":"2023-01-24T15:10:04.707850+0000","last_became_peered":"2023-01-24T15:10:04.707850+0000","last_unstale":"2023-01-24T15:11:00.156339+0000","last_undegraded":"2023-01-24T15:11:00.156339+0000","last_fullsized":"2023-01-24T15:10:03.753405+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T14:43:35.924477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T14:43:35.924477+0000","last_clean_scrub_stamp":"2023-01-24T14:43:35.924477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0],"acting":[0],"avail_no_missing":["0"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":396,"reported_epoch":538,"state":"active+clean","last_fresh":"2023-01-24T15:11:06.944754+0000","last_change":"2023-01-24T14:42:43.412373+0000","last_active":"2023-01-24T15:11:06.944754+0000","last_peered":"2023-01-24T15:11:06.944754+0000","last_clean":"2023-01-24T15:11:06.944754+0000","last_became_active":"2023-01-24T14:42:43.409737+0000","last_became_peered":"2023-01-24T14:42:43.409737+0000","last_unstale":"2023-01-24T15:11:06.944754+0000","last_undegraded":"2023-01-24T15:11:06.944754+0000","last_fullsized":"2023-01-24T15:11:06.944754+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T14:42:42.330702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T14:42:42.330702+0000","last_clean_scrub_stamp":"2023-01-24T14:42:42.330702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":364,"reported_epoch":539,"state":"active+undersized","last_fresh":"2023-01-24T15:11:08.658988+0000","last_change":"2023-01-24T15:09:58.513607+0000","last_active":"2023-01-24T15:11:08.658988+0000","last_peered":"2023-01-24T15:11:08.658988+0000","last_clean":"2023-01-24T15:09:54.053508+0000","last_became_active":"2023-01-24T15:09:58.513607+0000","last_became_peered":"2023-01-24T15:09:58.513607+0000","last_unstale":"2023-01-24T15:11:08.658988+0000","last_undegraded":"2023-01-24T15:11:08.658988+0000","last_fullsized":"2023-01-24T15:09:58.390668+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T14:43:35.924477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T14:43:35.924477+0000","last_clean_scrub_stamp":"2023-01-24T14:43:35.924477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":395,"reported_epoch":538,"state":"active+clean","last_fresh":"2023-01-24T15:11:04.808692+0000","last_change":"2023-01-24T14:42:43.472889+0000","last_active":"2023-01-24T15:11:04.808692+0000","last_peered":"2023-01-24T15:11:04.808692+0000","last_clean":"2023-01-24T15:11:04.808692+0000","last_became_active":"2023-01-24T14:42:43.469425+0000","last_became_peered":"2023-01-24T14:42:43.469425+0000","last_unstale":"2023-01-24T15:11:04.808692+0000","last_undegraded":"2023-01-24T15:11:04.808692+0000","last_fullsized":"2023-01-24T15:11:04.808692+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T14:42:42.330702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T14:42:42.330702+0000","last_clean_scrub_stamp":"2023-01-24T14:42:42.330702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":463,"reported_epoch":539,"state":"active+clean","last_fresh":"2023-01-24T15:11:07.800091+0000","last_change":"2023-01-24T14:42:39.395501+0000","last_active":"2023-01-24T15:11:07.800091+0000","last_peered":"2023-01-24T15:11:07.800091+0000","last_clean":"2023-01-24T15:11:07.800091+0000","last_became_active":"2023-01-24T14:42:39.392286+0000","last_became_peered":"2023-01-24T14:42:39.392286+0000","last_unstale":"2023-01-24T15:11:07.800091+0000","last_undegraded":"2023-01-24T15:11:07.800091+0000","last_fullsized":"2023-01-24T15:11:07.800091+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T14:42:38.317142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T14:42:38.317142+0000","last_clean_scrub_stamp":"2023-01-24T14:42:38.317142+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":39,"num_read_kb":29,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":392,"reported_epoch":538,"state":"active+clean","last_fresh":"2023-01-24T15:11:05.384241+0000","last_change":"2023-01-24T14:43:37.157359+0000","last_active":"2023-01-24T15:11:05.384241+0000","last_peered":"2023-01-24T15:11:05.384241+0000","last_clean":"2023-01-24T15:11:05.384241+0000","last_became_active":"2023-01-24T14:43:37.148439+0000","last_became_peered":"2023-01-24T14:43:37.148439+0000","last_unstale":"2023-01-24T15:11:05.384241+0000","last_undegraded":"2023-01-24T15:11:05.384241+0000","last_fullsized":"2023-01-24T15:11:05.384241+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T14:43:35.924477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T14:43:35.924477+0000","last_clean_scrub_stamp":"2023-01-24T14:43:35.924477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":400,"reported_epoch":536,"state":"active+clean","last_fresh":"2023-01-24T15:11:00.748521+0000","last_change":"2023-01-24T14:42:43.426068+0000","last_active":"2023-01-24T15:11:00.748521+0000","last_peered":"2023-01-24T15:11:00.748521+0000","last_clean":"2023-01-24T15:11:00.748521+0000","last_became_active":"2023-01-24T14:42:43.422382+0000","last_became_peered":"2023-01-24T14:42:43.422382+0000","last_unstale":"2023-01-24T15:11:00.748521+0000","last_undegraded":"2023-01-24T15:11:00.748521+0000","last_fullsized":"2023-01-24T15:11:00.748521+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T14:42:42.330702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T14:42:42.330702+0000","last_clean_scrub_stamp":"2023-01-24T14:42:42.330702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":360,"reported_epoch":539,"state":"active+clean","last_fresh":"2023-01-24T15:11:08.617907+0000","last_change":"2023-01-24T14:43:37.191358+0000","last_active":"2023-01-24T15:11:08.617907+0000","last_peered":"2023-01-24T15:11:08.617907+0000","last_clean":"2023-01-24T15:11:08.617907+0000","last_became_active":"2023-01-24T14:43:37.187180+0000","last_became_peered":"2023-01-24T14:43:37.187180+0000","last_unstale":"2023-01-24T15:11:08.617907+0000","last_undegraded":"2023-01-24T15:11:08.617907+0000","last_fullsized":"2023-01-24T15:11:08.617907+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T14:43:35.924477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T14:43:35.924477+0000","last_clean_scrub_stamp":"2023-01-24T14:43:35.924477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":367,"reported_epoch":539,"state":"active+undersized","last_fresh":"2023-01-24T15:11:08.229549+0000","last_change":"2023-01-24T15:09:58.519535+0000","last_active":"2023-01-24T15:11:08.229549+0000","last_peered":"2023-01-24T15:11:08.229549+0000","last_clean":"2023-01-24T15:09:53.661175+0000","last_became_active":"2023-01-24T15:09:58.519535+0000","last_became_peered":"2023-01-24T15:09:58.519535+0000","last_unstale":"2023-01-24T15:11:08.229549+0000","last_undegraded":"2023-01-24T15:11:08.229549+0000","last_fullsized":"2023-01-24T15:09:58.385532+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T14:42:42.330702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T14:42:42.330702+0000","last_clean_scrub_stamp":"2023-01-24T14:42:42.330702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":35,"reported_epoch":539,"state":"active+undersized","last_fresh":"2023-01-24T15:11:08.285346+0000","last_change":"2023-01-24T15:09:58.521178+0000","last_active":"2023-01-24T15:11:08.285346+0000","last_peered":"2023-01-24T15:11:08.285346+0000","last_clean":"2023-01-24T14:43:35.924477+0000","last_became_active":"2023-01-24T15:09:58.521178+0000","last_became_peered":"2023-01-24T15:09:58.521178+0000","last_unstale":"2023-01-24T15:11:08.285346+0000","last_undegraded":"2023-01-24T15:11:08.285346+0000","last_fullsized":"2023-01-24T15:09:58.343642+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T14:43:35.924477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T14:43:35.924477+0000","last_clean_scrub_stamp":"2023-01-24T14:43:35.924477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":360,"reported_epoch":539,"state":"active+clean","last_fresh":"2023-01-24T15:11:07.252862+0000","last_change":"2023-01-24T14:43:37.199321+0000","last_active":"2023-01-24T15:11:07.252862+0000","last_peered":"2023-01-24T15:11:07.252862+0000","last_clean":"2023-01-24T15:11:07.252862+0000","last_became_active":"2023-01-24T14:43:37.195467+0000","last_became_peered":"2023-01-24T14:43:37.195467+0000","last_unstale":"2023-01-24T15:11:07.252862+0000","last_undegraded":"2023-01-24T15:11:07.252862+0000","last_fullsized":"2023-01-24T15:11:07.252862+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T14:43:35.924477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T14:43:35.924477+0000","last_clean_scrub_stamp":"2023-01-24T14:43:35.924477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":26,"reported_epoch":536,"state":"active+undersized","last_fresh":"2023-01-24T15:11:01.815311+0000","last_change":"2023-01-24T15:10:04.697357+0000","last_active":"2023-01-24T15:11:01.815311+0000","last_peered":"2023-01-24T15:11:01.815311+0000","last_clean":"2023-01-24T14:42:42.330702+0000","last_became_active":"2023-01-24T15:10:04.697357+0000","last_became_peered":"2023-01-24T15:10:04.697357+0000","last_unstale":"2023-01-24T15:11:01.815311+0000","last_undegraded":"2023-01-24T15:11:01.815311+0000","last_fullsized":"2023-01-24T15:10:04.194347+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T14:42:42.330702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T14:42:42.330702+0000","last_clean_scrub_stamp":"2023-01-24T14:42:42.330702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0],"acting":[0],"avail_no_missing":["0"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":396,"reported_epoch":538,"state":"active+clean","last_fresh":"2023-01-24T15:11:08.458769+0000","last_change":"2023-01-24T14:42:43.463202+0000","last_active":"2023-01-24T15:11:08.458769+0000","last_peered":"2023-01-24T15:11:08.458769+0000","last_clean":"2023-01-24T15:11:08.458769+0000","last_became_active":"2023-01-24T14:42:43.459539+0000","last_became_peered":"2023-01-24T14:42:43.459539+0000","last_unstale":"2023-01-24T15:11:08.458769+0000","last_undegraded":"2023-01-24T15:11:08.458769+0000","last_fullsized":"2023-01-24T15:11:08.458769+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T14:42:42.330702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T14:42:42.330702+0000","last_clean_scrub_stamp":"2023-01-24T14:42:42.330702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":35,"reported_epoch":539,"state":"active+undersized","last_fresh":"2023-01-24T15:11:08.649521+0000","last_change":"2023-01-24T15:09:58.516792+0000","last_active":"2023-01-24T15:11:08.649521+0000","last_peered":"2023-01-24T15:11:08.649521+0000","last_clean":"2023-01-24T14:43:35.924477+0000","last_became_active":"2023-01-24T15:09:58.516792+0000","last_became_peered":"2023-01-24T15:09:58.516792+0000","last_unstale":"2023-01-24T15:11:08.649521+0000","last_undegraded":"2023-01-24T15:11:08.649521+0000","last_fullsized":"2023-01-24T15:09:57.969488+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T14:43:35.924477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T14:43:35.924477+0000","last_clean_scrub_stamp":"2023-01-24T14:43:35.924477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":27,"acting":27,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":39,"num_read_kb":29,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":12,"acting":12,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654707323,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":26052,"kb_used_data":2812,"kb_used_omap":0,"kb_used_meta":23232,"kb_avail":104831548,"statfs":{"total":107374182400,"available":107347505152,"internally_reserved":0,"allocated":2879488,"data_stored":2378605,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":23789568},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":9,"seq":38654707225,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":26912,"kb_used_data":2008,"kb_used_omap":0,"kb_used_meta":24896,"kb_avail":104830688,"statfs":{"total":107374182400,"available":107346624512,"internally_reserved":0,"allocated":2056192,"data_stored":1599296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":25493504},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":8,"seq":34359740011,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":22004,"kb_used_data":2796,"kb_used_omap":0,"kb_used_meta":19200,"kb_avail":104835596,"statfs":{"total":107374182400,"available":107351650304,"internally_reserved":0,"allocated":2863104,"data_stored":2364751,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":19660800},"hb_peers":[1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":8,"seq":34359740019,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":36632,"kb_used_data":2192,"kb_used_omap":0,"kb_used_meta":34432,"kb_avail":104820968,"statfs":{"total":107374182400,"available":107336671232,"internally_reserved":0,"allocated":2244608,"data_stored":1781283,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":35258368},"hb_peers":[0,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2023-01-24T15:11:12.087 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-01-24T15:11:12.430 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-01-24T15:11:12.433 INFO:teuthology.orchestra.run.smithi002.stderr:dumped all 2023-01-24T15:11:12.445 INFO:teuthology.orchestra.run.smithi002.stdout:{"pg_ready":true,"pg_map":{"version":1391,"stamp":"2023-01-24T15:11:12.234883+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":39,"num_read_kb":29,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":41,"acting":41,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":419430400,"kb_used":111608,"kb_used_data":9816,"kb_used_omap":0,"kb_used_meta":101760,"kb_avail":419318792,"statfs":{"total":429496729600,"available":429382443008,"internally_reserved":0,"allocated":10051584,"data_stored":8130863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":104202240},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-1,"num_write_kb":-1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.904938"},"pg_stats":[{"pgid":"3.f","version":"0'0","reported_seq":399,"reported_epoch":539,"state":"active+clean","last_fresh":"2023-01-24T15:11:06.992944+0000","last_change":"2023-01-24T14:43:37.151318+0000","last_active":"2023-01-24T15:11:06.992944+0000","last_peered":"2023-01-24T15:11:06.992944+0000","last_clean":"2023-01-24T15:11:06.992944+0000","last_became_active":"2023-01-24T14:43:37.134254+0000","last_became_peered":"2023-01-24T14:43:37.134254+0000","last_unstale":"2023-01-24T15:11:06.992944+0000","last_undegraded":"2023-01-24T15:11:06.992944+0000","last_fullsized":"2023-01-24T15:11:06.992944+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T14:43:35.924477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T14:43:35.924477+0000","last_clean_scrub_stamp":"2023-01-24T14:43:35.924477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":28,"reported_epoch":538,"state":"active+undersized","last_fresh":"2023-01-24T15:11:04.568949+0000","last_change":"2023-01-24T15:10:01.868687+0000","last_active":"2023-01-24T15:11:04.568949+0000","last_peered":"2023-01-24T15:11:04.568949+0000","last_clean":"2023-01-24T14:43:35.924477+0000","last_became_active":"2023-01-24T15:10:01.868687+0000","last_became_peered":"2023-01-24T15:10:01.868687+0000","last_unstale":"2023-01-24T15:11:04.568949+0000","last_undegraded":"2023-01-24T15:11:04.568949+0000","last_fullsized":"2023-01-24T15:10:00.258846+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T14:43:35.924477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T14:43:35.924477+0000","last_clean_scrub_stamp":"2023-01-24T14:43:35.924477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":393,"reported_epoch":538,"state":"active+clean","last_fresh":"2023-01-24T15:11:08.147866+0000","last_change":"2023-01-24T14:43:37.258438+0000","last_active":"2023-01-24T15:11:08.147866+0000","last_peered":"2023-01-24T15:11:08.147866+0000","last_clean":"2023-01-24T15:11:08.147866+0000","last_became_active":"2023-01-24T14:43:37.248775+0000","last_became_peered":"2023-01-24T14:43:37.248775+0000","last_unstale":"2023-01-24T15:11:08.147866+0000","last_undegraded":"2023-01-24T15:11:08.147866+0000","last_fullsized":"2023-01-24T15:11:08.147866+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T14:43:35.924477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T14:43:35.924477+0000","last_clean_scrub_stamp":"2023-01-24T14:43:35.924477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":399,"reported_epoch":539,"state":"active+clean","last_fresh":"2023-01-24T15:11:06.175340+0000","last_change":"2023-01-24T14:43:37.149037+0000","last_active":"2023-01-24T15:11:06.175340+0000","last_peered":"2023-01-24T15:11:06.175340+0000","last_clean":"2023-01-24T15:11:06.175340+0000","last_became_active":"2023-01-24T14:43:37.127163+0000","last_became_peered":"2023-01-24T14:43:37.127163+0000","last_unstale":"2023-01-24T15:11:06.175340+0000","last_undegraded":"2023-01-24T15:11:06.175340+0000","last_fullsized":"2023-01-24T15:11:06.175340+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T14:43:35.924477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T14:43:35.924477+0000","last_clean_scrub_stamp":"2023-01-24T14:43:35.924477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":360,"reported_epoch":539,"state":"active+clean","last_fresh":"2023-01-24T15:11:08.105890+0000","last_change":"2023-01-24T14:43:37.181952+0000","last_active":"2023-01-24T15:11:08.105890+0000","last_peered":"2023-01-24T15:11:08.105890+0000","last_clean":"2023-01-24T15:11:08.105890+0000","last_became_active":"2023-01-24T14:43:37.166010+0000","last_became_peered":"2023-01-24T14:43:37.166010+0000","last_unstale":"2023-01-24T15:11:08.105890+0000","last_undegraded":"2023-01-24T15:11:08.105890+0000","last_fullsized":"2023-01-24T15:11:08.105890+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T14:43:35.924477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T14:43:35.924477+0000","last_clean_scrub_stamp":"2023-01-24T14:43:35.924477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":399,"reported_epoch":539,"state":"active+clean","last_fresh":"2023-01-24T15:11:06.211204+0000","last_change":"2023-01-24T14:43:37.208750+0000","last_active":"2023-01-24T15:11:06.211204+0000","last_peered":"2023-01-24T15:11:06.211204+0000","last_clean":"2023-01-24T15:11:06.211204+0000","last_became_active":"2023-01-24T14:43:37.193886+0000","last_became_peered":"2023-01-24T14:43:37.193886+0000","last_unstale":"2023-01-24T15:11:06.211204+0000","last_undegraded":"2023-01-24T15:11:06.211204+0000","last_fullsized":"2023-01-24T15:11:06.211204+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T14:43:35.924477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T14:43:35.924477+0000","last_clean_scrub_stamp":"2023-01-24T14:43:35.924477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":399,"reported_epoch":539,"state":"active+clean","last_fresh":"2023-01-24T15:11:06.184261+0000","last_change":"2023-01-24T14:43:37.245196+0000","last_active":"2023-01-24T15:11:06.184261+0000","last_peered":"2023-01-24T15:11:06.184261+0000","last_clean":"2023-01-24T15:11:06.184261+0000","last_became_active":"2023-01-24T14:43:37.233660+0000","last_became_peered":"2023-01-24T14:43:37.233660+0000","last_unstale":"2023-01-24T15:11:06.184261+0000","last_undegraded":"2023-01-24T15:11:06.184261+0000","last_fullsized":"2023-01-24T15:11:06.184261+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T14:43:35.924477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T14:43:35.924477+0000","last_clean_scrub_stamp":"2023-01-24T14:43:35.924477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":393,"reported_epoch":538,"state":"active+clean","last_fresh":"2023-01-24T15:11:08.774830+0000","last_change":"2023-01-24T14:43:37.273633+0000","last_active":"2023-01-24T15:11:08.774830+0000","last_peered":"2023-01-24T15:11:08.774830+0000","last_clean":"2023-01-24T15:11:08.774830+0000","last_became_active":"2023-01-24T14:43:37.267461+0000","last_became_peered":"2023-01-24T14:43:37.267461+0000","last_unstale":"2023-01-24T15:11:08.774830+0000","last_undegraded":"2023-01-24T15:11:08.774830+0000","last_fullsized":"2023-01-24T15:11:08.774830+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T14:43:35.924477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T14:43:35.924477+0000","last_clean_scrub_stamp":"2023-01-24T14:43:35.924477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":35,"reported_epoch":539,"state":"active+undersized","last_fresh":"2023-01-24T15:11:08.631228+0000","last_change":"2023-01-24T15:09:58.515020+0000","last_active":"2023-01-24T15:11:08.631228+0000","last_peered":"2023-01-24T15:11:08.631228+0000","last_clean":"2023-01-24T14:42:42.330702+0000","last_became_active":"2023-01-24T15:09:58.515020+0000","last_became_peered":"2023-01-24T15:09:58.515020+0000","last_unstale":"2023-01-24T15:11:08.631228+0000","last_undegraded":"2023-01-24T15:11:08.631228+0000","last_fullsized":"2023-01-24T15:09:58.124818+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T14:42:42.330702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T14:42:42.330702+0000","last_clean_scrub_stamp":"2023-01-24T14:42:42.330702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":392,"reported_epoch":538,"state":"active+clean","last_fresh":"2023-01-24T15:11:04.771745+0000","last_change":"2023-01-24T14:43:37.209033+0000","last_active":"2023-01-24T15:11:04.771745+0000","last_peered":"2023-01-24T15:11:04.771745+0000","last_clean":"2023-01-24T15:11:04.771745+0000","last_became_active":"2023-01-24T14:43:37.197571+0000","last_became_peered":"2023-01-24T14:43:37.197571+0000","last_unstale":"2023-01-24T15:11:04.771745+0000","last_undegraded":"2023-01-24T15:11:04.771745+0000","last_fullsized":"2023-01-24T15:11:04.771745+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T14:43:35.924477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T14:43:35.924477+0000","last_clean_scrub_stamp":"2023-01-24T14:43:35.924477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":406,"reported_epoch":539,"state":"active+undersized","last_fresh":"2023-01-24T15:11:06.044420+0000","last_change":"2023-01-24T15:10:04.691115+0000","last_active":"2023-01-24T15:11:06.044420+0000","last_peered":"2023-01-24T15:11:06.044420+0000","last_clean":"2023-01-24T15:09:53.938902+0000","last_became_active":"2023-01-24T15:10:04.691115+0000","last_became_peered":"2023-01-24T15:10:04.691115+0000","last_unstale":"2023-01-24T15:11:06.044420+0000","last_undegraded":"2023-01-24T15:11:06.044420+0000","last_fullsized":"2023-01-24T15:10:03.477405+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T14:42:42.330702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T14:42:42.330702+0000","last_clean_scrub_stamp":"2023-01-24T14:42:42.330702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0],"acting":[0],"avail_no_missing":["0"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":28,"reported_epoch":539,"state":"active+undersized","last_fresh":"2023-01-24T15:11:06.745162+0000","last_change":"2023-01-24T15:10:04.707850+0000","last_active":"2023-01-24T15:11:06.745162+0000","last_peered":"2023-01-24T15:11:06.745162+0000","last_clean":"2023-01-24T14:43:35.924477+0000","last_became_active":"2023-01-24T15:10:04.707850+0000","last_became_peered":"2023-01-24T15:10:04.707850+0000","last_unstale":"2023-01-24T15:11:06.745162+0000","last_undegraded":"2023-01-24T15:11:06.745162+0000","last_fullsized":"2023-01-24T15:10:03.753405+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T14:43:35.924477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T14:43:35.924477+0000","last_clean_scrub_stamp":"2023-01-24T14:43:35.924477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0],"acting":[0],"avail_no_missing":["0"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":396,"reported_epoch":538,"state":"active+clean","last_fresh":"2023-01-24T15:11:06.944754+0000","last_change":"2023-01-24T14:42:43.412373+0000","last_active":"2023-01-24T15:11:06.944754+0000","last_peered":"2023-01-24T15:11:06.944754+0000","last_clean":"2023-01-24T15:11:06.944754+0000","last_became_active":"2023-01-24T14:42:43.409737+0000","last_became_peered":"2023-01-24T14:42:43.409737+0000","last_unstale":"2023-01-24T15:11:06.944754+0000","last_undegraded":"2023-01-24T15:11:06.944754+0000","last_fullsized":"2023-01-24T15:11:06.944754+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T14:42:42.330702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T14:42:42.330702+0000","last_clean_scrub_stamp":"2023-01-24T14:42:42.330702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":364,"reported_epoch":539,"state":"active+undersized","last_fresh":"2023-01-24T15:11:08.658988+0000","last_change":"2023-01-24T15:09:58.513607+0000","last_active":"2023-01-24T15:11:08.658988+0000","last_peered":"2023-01-24T15:11:08.658988+0000","last_clean":"2023-01-24T15:09:54.053508+0000","last_became_active":"2023-01-24T15:09:58.513607+0000","last_became_peered":"2023-01-24T15:09:58.513607+0000","last_unstale":"2023-01-24T15:11:08.658988+0000","last_undegraded":"2023-01-24T15:11:08.658988+0000","last_fullsized":"2023-01-24T15:09:58.390668+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T14:43:35.924477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T14:43:35.924477+0000","last_clean_scrub_stamp":"2023-01-24T14:43:35.924477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":395,"reported_epoch":538,"state":"active+clean","last_fresh":"2023-01-24T15:11:04.808692+0000","last_change":"2023-01-24T14:42:43.472889+0000","last_active":"2023-01-24T15:11:04.808692+0000","last_peered":"2023-01-24T15:11:04.808692+0000","last_clean":"2023-01-24T15:11:04.808692+0000","last_became_active":"2023-01-24T14:42:43.469425+0000","last_became_peered":"2023-01-24T14:42:43.469425+0000","last_unstale":"2023-01-24T15:11:04.808692+0000","last_undegraded":"2023-01-24T15:11:04.808692+0000","last_fullsized":"2023-01-24T15:11:04.808692+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T14:42:42.330702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T14:42:42.330702+0000","last_clean_scrub_stamp":"2023-01-24T14:42:42.330702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq