2023-01-24T14:18:46.179 INFO:root:teuthology version: 0.0.1.dev22+g8e1bba1 2023-01-24T14:18:46.180 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_rzarzynski@teuthology --name rzarzynski-2023-01-24_14:08:39-crimson-rados-main-distro-default-smithi --archive /home/teuthworker/archive/rzarzynski-2023-01-24_14:08:39-crimson-rados-main-distro-default-smithi/7135317 --description crimson-rados/rbd/{centos_latest clusters/fixed-1 crimson_qa_overrides deploy/ceph tasks/rbd_lock_and_fence} --verbose -- /home/teuthworker/archive/rzarzynski-2023-01-24_14:08:39-crimson-rados-main-distro-default-smithi/7135317/orig.config.yaml 2023-01-24T14:18:46.205 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2023-01-24T14:18:46.261 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/rzarzynski-2023-01-24_14:08:39-crimson-rados-main-distro-default-smithi/7135317 branch: main description: crimson-rados/rbd/{centos_latest clusters/fixed-1 crimson_qa_overrides deploy/ceph tasks/rbd_lock_and_fence} email: null first_in_suite: false job_id: '7135317' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: rzarzynski-2023-01-24_14:08:39-crimson-rados-main-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 8.stream overrides: admin_socket: branch: main ceph: conf: global: ms client mode: crc ms cluster mode: crc ms mon client mode: crc ms mon cluster mode: crc ms mon service mode: crc ms service mode: crc mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 osd shutdown pgref assert: true flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 1a671c198559f54389cd5afa22085a0c7b29a1ef ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log global: osd crush chooseleaf type: 0 osd pool default pg num: 128 osd pool default pgp num: 128 osd pool default size: 2 mon: osd default pool size: 2 install: ceph: flavor: crimson sha1: 1a671c198559f54389cd5afa22085a0c7b29a1ef selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: main env: CRIMSON_COMPAT: '1' sha1: 510284b66513490445619d1430aa869868c71a09 owner: scheduled_rzarzynski@teuthology priority: 60 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 sha1: 1a671c198559f54389cd5afa22085a0c7b29a1ef sleep_before_teardown: 0 suite: crimson-rados suite_branch: main suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_510284b66513490445619d1430aa869868c71a09/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 510284b66513490445619d1430aa869868c71a09 targets: smithi179.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDhcVdRI0vM1zDtZgKT9dJkSvZ2ScPJA++AreyF5LS+WNJw95QHg7bH3ztqfaElQcRF/7W0nP9mvKzB6JNgCtOodGC0AfXVksSzrrpGlzizloEFQ7ZHLZOv7WlhwmAq3+2hoeSGsEW9nDTWZgkNHRk6ytj3Ic7w0rmBloX41oZeCiSBJDFIHRMOvTf61J1lWSXucJYKTqiFM9/XzLr1zPZov3bTj29nHP8E+AqSXI3lV76oShfa5gPgkpvM0VJhbNQcnUMxe5r+C5OjvWGwbup8UNUMsJuD0QjDl7jP3giakJZ/aK2D3V4phnGPAn80VkK7CadnRiTNyuM/+6pcV37C5W3+2xc7Kdy+2Ykpp+ufQ44ORL6Ibpll6xQGlGG/Fpn0gvINGM/ArivhRN/5jt5Mj4TUVbjii3Q+beU2QzugtsP3kJ/Vo/ExPnM9mt3w6aProdoJ4s6EowljAU+dV7pjMIgKtrXqr/xAEmAN6PyWBeaS2mIhXi03WTr6JpDHDk8= tasks: - install: null - ceph: conf: mon: mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 paxos service trim min: 10 osd: debug monc: 20 flavor: crimson - workunit: clients: client.0: - rbd/test_lock_fence.sh teuthology_branch: main teuthology_sha1: 8e1bba12e584b7ae912e92e94afe9c075ab51fa3 timestamp: 2023-01-24_14:08:39 tube: smithi user: rzarzynski verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.17735 2023-01-24T14:18:46.262 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_510284b66513490445619d1430aa869868c71a09/qa; will attempt to use it 2023-01-24T14:18:46.262 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_510284b66513490445619d1430aa869868c71a09/qa/tasks 2023-01-24T14:18:46.263 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-01-24T14:18:46.264 INFO:teuthology.task.internal:Checking packages... 2023-01-24T14:18:46.291 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'crimson' and ceph hash '1a671c198559f54389cd5afa22085a0c7b29a1ef' 2023-01-24T14:18:46.291 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-01-24T14:18:46.292 INFO:teuthology.packaging:ref: None 2023-01-24T14:18:46.292 INFO:teuthology.packaging:tag: None 2023-01-24T14:18:46.292 INFO:teuthology.packaging:branch: main 2023-01-24T14:18:46.292 INFO:teuthology.packaging:sha1: 1a671c198559f54389cd5afa22085a0c7b29a1ef 2023-01-24T14:18:46.293 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F8%2Fx86_64&ref=main 2023-01-24T14:18:46.522 INFO:teuthology.task.internal:Found packages for ceph version 18.0.0-1954.g1a671c19 2023-01-24T14:18:46.523 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-01-24T14:18:46.529 INFO:teuthology.task.internal:no buildpackages task found 2023-01-24T14:18:46.530 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-01-24T14:18:46.542 INFO:teuthology.task.internal:Saving configuration 2023-01-24T14:18:46.553 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-01-24T14:18:46.559 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-01-24T14:18:46.581 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi179.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rzarzynski-2023-01-24_14:08:39-crimson-rados-main-distro-default-smithi/7135317', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-01-24 14:13:47.250613', 'locked_by': 'scheduled_rzarzynski@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPUM2upMNuetcXh1temCdppKPlqbHMvd5NjS016RGwfQQ3dcEzHlWvf/sDemtaAZy2ZnMMl8t5kIv/tS4Y7EmJk='} 2023-01-24T14:18:46.582 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-01-24T14:18:46.587 INFO:teuthology.task.internal:roles: ubuntu@smithi179.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2023-01-24T14:18:46.588 INFO:teuthology.run_tasks:Running task console_log... 2023-01-24T14:18:46.625 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f68aa3775e0>, signals=[15]) 2023-01-24T14:18:46.626 INFO:teuthology.run_tasks:Running task internal.connect... 2023-01-24T14:18:46.633 INFO:teuthology.task.internal:Opening connections... 2023-01-24T14:18:46.633 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi179.front.sepia.ceph.com 2023-01-24T14:18:46.635 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi179.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-01-24T14:18:46.702 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-01-24T14:18:46.709 DEBUG:teuthology.orchestra.run.smithi179:> uname -m 2023-01-24T14:18:46.773 INFO:teuthology.orchestra.run.smithi179.stdout:x86_64 2023-01-24T14:18:46.773 DEBUG:teuthology.orchestra.run.smithi179:> cat /etc/os-release 2023-01-24T14:18:46.832 INFO:teuthology.orchestra.run.smithi179.stdout:NAME="CentOS Stream" 2023-01-24T14:18:46.832 INFO:teuthology.orchestra.run.smithi179.stdout:VERSION="8" 2023-01-24T14:18:46.832 INFO:teuthology.orchestra.run.smithi179.stdout:ID="centos" 2023-01-24T14:18:46.833 INFO:teuthology.orchestra.run.smithi179.stdout:ID_LIKE="rhel fedora" 2023-01-24T14:18:46.833 INFO:teuthology.orchestra.run.smithi179.stdout:VERSION_ID="8" 2023-01-24T14:18:46.833 INFO:teuthology.orchestra.run.smithi179.stdout:PLATFORM_ID="platform:el8" 2023-01-24T14:18:46.834 INFO:teuthology.orchestra.run.smithi179.stdout:PRETTY_NAME="CentOS Stream 8" 2023-01-24T14:18:46.834 INFO:teuthology.orchestra.run.smithi179.stdout:ANSI_COLOR="0;31" 2023-01-24T14:18:46.834 INFO:teuthology.orchestra.run.smithi179.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2023-01-24T14:18:46.834 INFO:teuthology.orchestra.run.smithi179.stdout:HOME_URL="https://centos.org/" 2023-01-24T14:18:46.834 INFO:teuthology.orchestra.run.smithi179.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-01-24T14:18:46.835 INFO:teuthology.orchestra.run.smithi179.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2023-01-24T14:18:46.835 INFO:teuthology.orchestra.run.smithi179.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-01-24T14:18:46.836 INFO:teuthology.lock.ops:Updating smithi179.front.sepia.ceph.com on lock server 2023-01-24T14:18:46.851 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-01-24T14:18:46.859 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-01-24T14:18:46.865 INFO:teuthology.task.internal:Checking for old test directory... 2023-01-24T14:18:46.866 DEBUG:teuthology.orchestra.run.smithi179:> test '!' -e /home/ubuntu/cephtest 2023-01-24T14:18:46.890 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-01-24T14:18:46.896 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-01-24T14:18:46.897 DEBUG:teuthology.orchestra.run.smithi179:> test -z $(ls -A /var/lib/ceph) 2023-01-24T14:18:46.950 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-01-24T14:18:46.979 INFO:teuthology.run_tasks:Running task kernel... 2023-01-24T14:18:46.993 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-01-24T14:18:46.993 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-01-24T14:18:46.994 DEBUG:teuthology.orchestra.run.smithi179:> test -f /run/.containerenv -o -f /.dockerenv 2023-01-24T14:18:47.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-01-24T14:18:47.012 DEBUG:teuthology.orchestra.run.smithi179:> uname -r 2023-01-24T14:18:47.071 INFO:teuthology.orchestra.run.smithi179.stdout:4.18.0-408.el8.x86_64 2023-01-24T14:18:47.071 INFO:teuthology.task.kernel:Running kernel on smithi179: 4.18.0-408.el8.x86_64 2023-01-24T14:18:47.071 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum install -y kernel 2023-01-24T14:18:50.785 INFO:teuthology.orchestra.run.smithi179.stdout:CentOS-8 - AppStream 34 MB/s | 27 MB 00:00 2023-01-24T14:18:56.418 INFO:teuthology.orchestra.run.smithi179.stdout:CentOS-8 - BaseOS 42 MB/s | 27 MB 00:00 2023-01-24T14:19:00.733 INFO:teuthology.orchestra.run.smithi179.stdout:CentOS-8 - Extras 157 kB/s | 18 kB 00:00 2023-01-24T14:19:01.160 INFO:teuthology.orchestra.run.smithi179.stdout:CentOS-8 - PowerTools 19 MB/s | 5.6 MB 00:00 2023-01-24T14:19:02.385 INFO:teuthology.orchestra.run.smithi179.stdout:Copr repo for python3-asyncssh owned by ceph 25 kB/s | 3.5 kB 00:00 2023-01-24T14:19:03.485 INFO:teuthology.orchestra.run.smithi179.stdout:Extra Packages for Enterprise Linux 14 MB/s | 13 MB 00:00 2023-01-24T14:19:06.811 INFO:teuthology.orchestra.run.smithi179.stdout:lab-extras 248 kB/s | 24 kB 00:00 2023-01-24T14:19:09.917 INFO:teuthology.orchestra.run.smithi179.stdout:Package kernel-4.18.0-408.el8.x86_64 is already installed. 2023-01-24T14:19:09.917 INFO:teuthology.orchestra.run.smithi179.stdout:Package kernel-4.18.0-305.3.1.el8.x86_64 is already installed. 2023-01-24T14:19:10.082 INFO:teuthology.orchestra.run.smithi179.stdout:Dependencies resolved. 2023-01-24T14:19:10.083 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2023-01-24T14:19:10.084 INFO:teuthology.orchestra.run.smithi179.stdout: Package Arch Version Repository Size 2023-01-24T14:19:10.084 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2023-01-24T14:19:10.084 INFO:teuthology.orchestra.run.smithi179.stdout:Installing: 2023-01-24T14:19:10.084 INFO:teuthology.orchestra.run.smithi179.stdout: kernel x86_64 4.18.0-448.el8 CentOS-BaseOS 9.2 M 2023-01-24T14:19:10.084 INFO:teuthology.orchestra.run.smithi179.stdout: kernel-core x86_64 4.18.0-448.el8 CentOS-BaseOS 41 M 2023-01-24T14:19:10.085 INFO:teuthology.orchestra.run.smithi179.stdout: kernel-modules x86_64 4.18.0-448.el8 CentOS-BaseOS 34 M 2023-01-24T14:19:10.085 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-01-24T14:19:10.085 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction Summary 2023-01-24T14:19:10.085 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2023-01-24T14:19:10.085 INFO:teuthology.orchestra.run.smithi179.stdout:Install 3 Packages 2023-01-24T14:19:10.086 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-01-24T14:19:10.086 INFO:teuthology.orchestra.run.smithi179.stdout:Total download size: 85 M 2023-01-24T14:19:10.086 INFO:teuthology.orchestra.run.smithi179.stdout:Installed size: 94 M 2023-01-24T14:19:10.086 INFO:teuthology.orchestra.run.smithi179.stdout:Downloading Packages: 2023-01-24T14:19:11.012 INFO:teuthology.orchestra.run.smithi179.stdout:(1/3): kernel-modules-4.18.0-448.el8.x86_64.rpm 37 MB/s | 34 MB 00:00 2023-01-24T14:19:11.129 INFO:teuthology.orchestra.run.smithi179.stdout:(2/3): kernel-4.18.0-448.el8.x86_64.rpm 8.8 MB/s | 9.2 MB 00:01 2023-01-24T14:19:11.721 INFO:teuthology.orchestra.run.smithi179.stdout:(3/3): kernel-core-4.18.0-448.el8.x86_64.rpm 25 MB/s | 41 MB 00:01 2023-01-24T14:19:11.724 INFO:teuthology.orchestra.run.smithi179.stdout:-------------------------------------------------------------------------------- 2023-01-24T14:19:11.724 INFO:teuthology.orchestra.run.smithi179.stdout:Total 52 MB/s | 85 MB 00:01 2023-01-24T14:19:12.356 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction check 2023-01-24T14:19:12.743 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction check succeeded. 2023-01-24T14:19:12.743 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction test 2023-01-24T14:19:14.803 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction test succeeded. 2023-01-24T14:19:14.846 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction 2023-01-24T14:19:17.561 INFO:teuthology.orchestra.run.smithi179.stdout: Preparing : 1/1 2023-01-24T14:19:18.110 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : kernel-core-4.18.0-448.el8.x86_64 1/3 2023-01-24T14:19:22.057 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: kernel-core-4.18.0-448.el8.x86_64 1/3 2023-01-24T14:19:22.493 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : kernel-modules-4.18.0-448.el8.x86_64 2/3 2023-01-24T14:19:25.809 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: kernel-modules-4.18.0-448.el8.x86_64 2/3 2023-01-24T14:19:26.052 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : kernel-4.18.0-448.el8.x86_64 3/3 2023-01-24T14:19:47.779 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: kernel-core-4.18.0-448.el8.x86_64 3/3 2023-01-24T14:19:47.805 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: kernel-modules-4.18.0-448.el8.x86_64 3/3 2023-01-24T14:19:48.225 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: kernel-4.18.0-448.el8.x86_64 3/3 2023-01-24T14:19:48.225 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : kernel-4.18.0-448.el8.x86_64 1/3 2023-01-24T14:19:48.225 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : kernel-core-4.18.0-448.el8.x86_64 2/3 2023-01-24T14:19:48.801 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : kernel-modules-4.18.0-448.el8.x86_64 3/3 2023-01-24T14:19:48.801 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-01-24T14:19:48.801 INFO:teuthology.orchestra.run.smithi179.stdout:Installed: 2023-01-24T14:19:48.802 INFO:teuthology.orchestra.run.smithi179.stdout: kernel-4.18.0-448.el8.x86_64 kernel-core-4.18.0-448.el8.x86_64 2023-01-24T14:19:48.802 INFO:teuthology.orchestra.run.smithi179.stdout: kernel-modules-4.18.0-448.el8.x86_64 2023-01-24T14:19:48.802 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-01-24T14:19:48.802 INFO:teuthology.orchestra.run.smithi179.stdout:Complete! 2023-01-24T14:19:48.908 DEBUG:teuthology.orchestra.run.smithi179:> rpm -q kernel --last | head -n 1 2023-01-24T14:19:49.103 INFO:teuthology.orchestra.run.smithi179.stdout:kernel-4.18.0-448.el8.x86_64 Tue 24 Jan 2023 02:19:25 PM UTC 2023-01-24T14:19:49.105 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-448.el8.x86_64 2023-01-24T14:19:49.105 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-408.el8.x86_64 Expected: 4.18.0-448.el8.x86_64 2023-01-24T14:19:49.105 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-01-24T14:19:49.105 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-01-24T14:19:49.106 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2023-01-24T14:19:49.106 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi179.front.sepia.ceph.com, path=None, version=distro) 2023-01-24T14:19:49.106 DEBUG:teuthology.orchestra.run.smithi179:> rpm -q kernel --last | head -n 1 2023-01-24T14:19:49.293 INFO:teuthology.orchestra.run.smithi179.stdout:kernel-4.18.0-448.el8.x86_64 Tue 24 Jan 2023 02:19:25 PM UTC 2023-01-24T14:19:49.294 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-448.el8.x86_64 2023-01-24T14:19:49.295 DEBUG:teuthology.orchestra.run.smithi179:> sudo rpm -qi grub2-tools 2023-01-24T14:19:49.328 INFO:teuthology.orchestra.run.smithi179.stdout:Name : grub2-tools 2023-01-24T14:19:49.329 INFO:teuthology.orchestra.run.smithi179.stdout:Epoch : 1 2023-01-24T14:19:49.329 INFO:teuthology.orchestra.run.smithi179.stdout:Version : 2.02 2023-01-24T14:19:49.329 INFO:teuthology.orchestra.run.smithi179.stdout:Release : 129.el8 2023-01-24T14:19:49.329 INFO:teuthology.orchestra.run.smithi179.stdout:Architecture: x86_64 2023-01-24T14:19:49.329 INFO:teuthology.orchestra.run.smithi179.stdout:Install Date: Sun 28 Aug 2022 01:03:50 PM UTC 2023-01-24T14:19:49.330 INFO:teuthology.orchestra.run.smithi179.stdout:Group : System Environment/Base 2023-01-24T14:19:49.330 INFO:teuthology.orchestra.run.smithi179.stdout:Size : 9265586 2023-01-24T14:19:49.330 INFO:teuthology.orchestra.run.smithi179.stdout:License : GPLv3+ 2023-01-24T14:19:49.330 INFO:teuthology.orchestra.run.smithi179.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2023-01-24T14:19:49.330 INFO:teuthology.orchestra.run.smithi179.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2023-01-24T14:19:49.331 INFO:teuthology.orchestra.run.smithi179.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2023-01-24T14:19:49.331 INFO:teuthology.orchestra.run.smithi179.stdout:Build Host : kbuilder.bsys.centos.org 2023-01-24T14:19:49.331 INFO:teuthology.orchestra.run.smithi179.stdout:Relocations : (not relocatable) 2023-01-24T14:19:49.331 INFO:teuthology.orchestra.run.smithi179.stdout:Packager : CentOS BuildSystem 2023-01-24T14:19:49.331 INFO:teuthology.orchestra.run.smithi179.stdout:Vendor : CentOS 2023-01-24T14:19:49.332 INFO:teuthology.orchestra.run.smithi179.stdout:URL : http://www.gnu.org/software/grub/ 2023-01-24T14:19:49.332 INFO:teuthology.orchestra.run.smithi179.stdout:Summary : Support tools for GRUB. 2023-01-24T14:19:49.332 INFO:teuthology.orchestra.run.smithi179.stdout:Description : 2023-01-24T14:19:49.332 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-01-24T14:19:49.332 INFO:teuthology.orchestra.run.smithi179.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-01-24T14:19:49.333 INFO:teuthology.orchestra.run.smithi179.stdout:customizable bootloader with modular architecture. It supports a rich 2023-01-24T14:19:49.333 INFO:teuthology.orchestra.run.smithi179.stdout:variety of kernel formats, file systems, computer architectures and 2023-01-24T14:19:49.333 INFO:teuthology.orchestra.run.smithi179.stdout:hardware devices. 2023-01-24T14:19:49.333 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-01-24T14:19:49.333 INFO:teuthology.orchestra.run.smithi179.stdout:This subpackage provides tools for support of all platforms. 2023-01-24T14:19:49.334 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-01-24T14:19:49.335 INFO:teuthology.task.kernel:Updating grub on smithi179 to boot 4.18.0-448.el8.x86_64 2023-01-24T14:19:49.335 DEBUG:teuthology.orchestra.run.smithi179:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-01-24T14:19:51.123 INFO:teuthology.orchestra.run.smithi179.stderr:Generating grub configuration file ... 2023-01-24T14:19:53.267 INFO:teuthology.orchestra.run.smithi179.stderr:done 2023-01-24T14:19:53.269 DEBUG:teuthology.orchestra.run.smithi179:> mktemp 2023-01-24T14:19:53.306 INFO:teuthology.orchestra.run.smithi179.stdout:/tmp/tmp.p44jDlSGZH 2023-01-24T14:19:53.306 DEBUG:teuthology.orchestra.run.smithi179:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.p44jDlSGZH 2023-01-24T14:19:53.384 DEBUG:teuthology.orchestra.run.smithi179:> sudo chmod 0666 /tmp/tmp.p44jDlSGZH 2023-01-24T14:19:53.538 DEBUG:teuthology.orchestra.remote:smithi179:/tmp/tmp.p44jDlSGZH is 6KB 2023-01-24T14:19:53.589 DEBUG:teuthology.orchestra.run.smithi179:> rm -fr /tmp/tmp.p44jDlSGZH 2023-01-24T14:19:53.606 DEBUG:teuthology.orchestra.run.smithi179:> sudo /bin/ls /boot/loader/entries || true 2023-01-24T14:19:53.675 INFO:teuthology.orchestra.run.smithi179.stdout:2ff282740cf84d4fa02ff60ed95fbbff-0-rescue.conf 2023-01-24T14:19:53.675 INFO:teuthology.orchestra.run.smithi179.stdout:2ff282740cf84d4fa02ff60ed95fbbff-4.18.0-305.3.1.el8.x86_64.conf 2023-01-24T14:19:53.675 INFO:teuthology.orchestra.run.smithi179.stdout:2ff282740cf84d4fa02ff60ed95fbbff-4.18.0-408.el8.x86_64.conf 2023-01-24T14:19:53.675 INFO:teuthology.orchestra.run.smithi179.stdout:2ff282740cf84d4fa02ff60ed95fbbff-4.18.0-448.el8.x86_64.conf 2023-01-24T14:19:53.676 DEBUG:teuthology.orchestra.run.smithi179:> sudo grub2-set-default 2ff282740cf84d4fa02ff60ed95fbbff-4.18.0-448.el8.x86_64 2023-01-24T14:19:53.853 DEBUG:teuthology.orchestra.run.smithi179:> sudo shutdown -r now 2023-01-24T14:20:23.884 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2023-01-24T14:20:23.885 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi179.front.sepia.ceph.com' 2023-01-24T14:20:23.886 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi179.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-01-24T14:20:42.216 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.179 2023-01-24T14:21:12.247 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi179.front.sepia.ceph.com' 2023-01-24T14:21:12.248 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi179.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-01-24T14:21:30.600 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.179 2023-01-24T14:22:00.629 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi179.front.sepia.ceph.com' 2023-01-24T14:22:00.630 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi179.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-01-24T14:22:00.938 DEBUG:teuthology.orchestra.run.smithi179:> true 2023-01-24T14:22:01.229 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi179.front.sepia.ceph.com' 2023-01-24T14:22:01.229 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "4.18.0-448.el8.x86_64"... 2023-01-24T14:22:01.229 DEBUG:teuthology.orchestra.run.smithi179:> uname -r 2023-01-24T14:22:01.248 INFO:teuthology.orchestra.run.smithi179.stdout:4.18.0-448.el8.x86_64 2023-01-24T14:22:01.249 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-448.el8.x86_64 vs 4.18.0-448.el8.x86_64 2023-01-24T14:22:01.249 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-01-24T14:22:01.249 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-01-24T14:22:02.251 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2023-01-24T14:22:02.251 DEBUG:teuthology.orchestra.run.smithi179:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-01-24T14:22:02.356 INFO:teuthology.orchestra.run.smithi179.stdout:ttyS1 2023-01-24T14:22:02.372 DEBUG:teuthology.parallel:result is None 2023-01-24T14:22:02.372 INFO:teuthology.run_tasks:Running task internal.base... 2023-01-24T14:22:02.379 INFO:teuthology.task.internal:Creating test directory... 2023-01-24T14:22:02.379 DEBUG:teuthology.orchestra.run.smithi179:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-01-24T14:22:02.401 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-01-24T14:22:02.409 INFO:teuthology.run_tasks:Running task internal.archive... 2023-01-24T14:22:02.416 INFO:teuthology.task.internal:Creating archive directory... 2023-01-24T14:22:02.416 DEBUG:teuthology.orchestra.run.smithi179:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-01-24T14:22:02.475 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-01-24T14:22:02.482 INFO:teuthology.task.internal:Enabling coredump saving... 2023-01-24T14:22:02.483 DEBUG:teuthology.orchestra.run.smithi179:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-01-24T14:22:02.548 INFO:teuthology.orchestra.run.smithi179.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-01-24T14:22:02.559 INFO:teuthology.orchestra.run.smithi179.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-01-24T14:22:02.561 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-01-24T14:22:02.568 INFO:teuthology.task.internal:Configuring sudo... 2023-01-24T14:22:02.569 DEBUG:teuthology.orchestra.run.smithi179:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-01-24T14:22:02.631 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-01-24T14:22:02.640 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-01-24T14:22:02.640 DEBUG:teuthology.orchestra.run.smithi179:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-01-24T14:22:02.690 DEBUG:teuthology.orchestra.run.smithi179:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-01-24T14:22:02.781 DEBUG:teuthology.orchestra.run.smithi179:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-01-24T14:22:02.827 DEBUG:teuthology.orchestra.run.smithi179:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-01-24T14:22:02.910 DEBUG:teuthology.orchestra.run.smithi179:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-01-24T14:22:02.954 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2023-01-24T14:22:02.955 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-01-24T14:22:03.022 DEBUG:teuthology.orchestra.run.smithi179:> sudo service rsyslog restart 2023-01-24T14:22:03.119 INFO:teuthology.orchestra.run.smithi179.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-01-24T14:22:03.461 INFO:teuthology.run_tasks:Running task internal.timer... 2023-01-24T14:22:03.469 INFO:teuthology.task.internal:Starting timer... 2023-01-24T14:22:03.469 INFO:teuthology.run_tasks:Running task pcp... 2023-01-24T14:22:03.485 INFO:teuthology.run_tasks:Running task selinux... 2023-01-24T14:22:03.499 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2023-01-24T14:22:03.499 DEBUG:teuthology.orchestra.run.smithi179:> sudo service auditd rotate 2023-01-24T14:22:03.575 INFO:teuthology.orchestra.run.smithi179.stdout:Rotating logs: 2023-01-24T14:22:03.577 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-01-24T14:22:03.577 DEBUG:teuthology.orchestra.run.smithi179:> /usr/sbin/getenforce 2023-01-24T14:22:03.608 INFO:teuthology.orchestra.run.smithi179.stdout:Permissive 2023-01-24T14:22:03.609 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi179.front.sepia.ceph.com': 'permissive'} 2023-01-24T14:22:03.609 DEBUG:teuthology.orchestra.run.smithi179:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|comm="ksmtuned"\|comm="sssd"\|comm="sss_cache"\|context=system_u:system_r:NetworkManager_dispatcher_t:s0\)' 2023-01-24T14:22:03.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-01-24T14:22:03.703 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-01-24T14:22:03.703 DEBUG:teuthology.orchestra.run.smithi179:> sudo /usr/sbin/setenforce permissive 2023-01-24T14:22:03.775 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-01-24T14:22:03.789 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main was just updated or references a specific commit; assuming it is current 2023-01-24T14:22:03.789 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-01-24T14:22:03.811 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-01-24T14:22:03.813 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi179.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-01-24T14:28:26.303 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi179.front.sepia.ceph.com')] 2023-01-24T14:28:26.305 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi179.front.sepia.ceph.com' 2023-01-24T14:28:26.306 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi179.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-01-24T14:28:26.373 DEBUG:teuthology.orchestra.run.smithi179:> true 2023-01-24T14:28:26.438 INFO:teuthology.run_tasks:Running task clock... 2023-01-24T14:28:26.448 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-01-24T14:28:26.448 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-01-24T14:28:26.449 DEBUG:teuthology.orchestra.run.smithi179:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-01-24T14:28:26.510 INFO:teuthology.orchestra.run.smithi179.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-01-24T14:28:26.524 INFO:teuthology.orchestra.run.smithi179.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-01-24T14:28:26.593 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: ntpd: command not found 2023-01-24T14:28:26.604 INFO:teuthology.orchestra.run.smithi179.stdout:506 Cannot talk to daemon 2023-01-24T14:28:26.619 INFO:teuthology.orchestra.run.smithi179.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-01-24T14:28:26.632 INFO:teuthology.orchestra.run.smithi179.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-01-24T14:28:26.701 INFO:teuthology.orchestra.run.smithi179.stderr:bash: ntpq: command not found 2023-01-24T14:28:26.705 INFO:teuthology.orchestra.run.smithi179.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-01-24T14:28:26.705 INFO:teuthology.orchestra.run.smithi179.stdout:=============================================================================== 2023-01-24T14:28:26.705 INFO:teuthology.orchestra.run.smithi179.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-01-24T14:28:26.705 INFO:teuthology.orchestra.run.smithi179.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-01-24T14:28:26.706 INFO:teuthology.orchestra.run.smithi179.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-01-24T14:28:26.706 INFO:teuthology.orchestra.run.smithi179.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-01-24T14:28:26.707 INFO:teuthology.run_tasks:Running task install... 2023-01-24T14:28:26.716 DEBUG:teuthology.task.install:project ceph 2023-01-24T14:28:26.716 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'crimson', 'sha1': '1a671c198559f54389cd5afa22085a0c7b29a1ef'}} 2023-01-24T14:28:26.716 DEBUG:teuthology.task.install:config {'flavor': 'crimson', 'sha1': '1a671c198559f54389cd5afa22085a0c7b29a1ef'} 2023-01-24T14:28:26.717 INFO:teuthology.task.install:Using flavor: crimson 2023-01-24T14:28:26.723 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-01-24T14:28:26.723 INFO:teuthology.task.install:extra packages: [] 2023-01-24T14:28:26.723 DEBUG:teuthology.orchestra.run.smithi179:> sudo lsb_release -is 2023-01-24T14:28:26.784 INFO:teuthology.orchestra.run.smithi179.stdout:CentOSStream 2023-01-24T14:28:26.785 DEBUG:teuthology.misc:System to be installed: CentOSStream 2023-01-24T14:28:26.786 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'flavor': 'crimson', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '1a671c198559f54389cd5afa22085a0c7b29a1ef', 'tag': None, 'wait_for_package': False} 2023-01-24T14:28:26.786 DEBUG:teuthology.orchestra.run.smithi179:> sudo lsb_release -is 2023-01-24T14:28:26.861 INFO:teuthology.orchestra.run.smithi179.stdout:CentOSStream 2023-01-24T14:28:26.863 DEBUG:teuthology.misc:System to be installed: CentOSStream 2023-01-24T14:28:26.863 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F8%2Fx86_64&sha1=1a671c198559f54389cd5afa22085a0c7b29a1ef 2023-01-24T14:28:27.014 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/main/1a671c198559f54389cd5afa22085a0c7b29a1ef/centos/8/flavors/crimson/ 2023-01-24T14:28:27.014 INFO:teuthology.task.install.rpm:Package version is 18.0.0-1954.g1a671c19 2023-01-24T14:28:27.236 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/main/1a671c198559f54389cd5afa22085a0c7b29a1ef/centos/8/flavors/crimson/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/main/1a671c198559f54389cd5afa22085a0c7b29a1ef/centos/8/flavors/crimson/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/main/1a671c198559f54389cd5afa22085a0c7b29a1ef/centos/8/flavors/crimson/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-01-24T14:28:27.237 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2023-01-24T14:28:27.237 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-01-24T14:28:27.266 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2023-01-24T14:28:27.266 DEBUG:teuthology.orchestra.run.smithi179:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/1a671c198559f54389cd5afa22085a0c7b29a1ef/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-01-24T14:28:27.336 DEBUG:teuthology.orchestra.run.smithi179:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-01-24T14:28:27.414 DEBUG:teuthology.orchestra.run.smithi179:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-01-24T14:28:27.481 INFO:teuthology.orchestra.run.smithi179.stdout:check_obsoletes = 1 2023-01-24T14:28:27.482 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum clean all 2023-01-24T14:28:27.732 INFO:teuthology.orchestra.run.smithi179.stdout:61 files removed 2023-01-24T14:28:27.760 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum -y install ceph-radosgw 2023-01-24T14:28:28.798 INFO:teuthology.orchestra.run.smithi179.stdout:ceph packages for x86_64 132 kB/s | 83 kB 00:00 2023-01-24T14:28:29.423 INFO:teuthology.orchestra.run.smithi179.stdout:ceph noarch packages 16 kB/s | 9.3 kB 00:00 2023-01-24T14:28:30.093 INFO:teuthology.orchestra.run.smithi179.stdout:ceph source packages 2.5 kB/s | 1.6 kB 00:00 2023-01-24T14:28:30.777 INFO:teuthology.orchestra.run.smithi179.stdout:CentOS-8 - AppStream 42 MB/s | 27 MB 00:00 2023-01-24T14:28:36.402 INFO:teuthology.orchestra.run.smithi179.stdout:CentOS-8 - BaseOS 43 MB/s | 27 MB 00:00 2023-01-24T14:28:40.642 INFO:teuthology.orchestra.run.smithi179.stdout:CentOS-8 - Extras 142 kB/s | 18 kB 00:00 2023-01-24T14:28:41.036 INFO:teuthology.orchestra.run.smithi179.stdout:CentOS-8 - PowerTools 21 MB/s | 5.6 MB 00:00 2023-01-24T14:28:42.286 INFO:teuthology.orchestra.run.smithi179.stdout:Copr repo for python3-asyncssh owned by ceph 24 kB/s | 3.5 kB 00:00 2023-01-24T14:28:44.253 INFO:teuthology.orchestra.run.smithi179.stdout:Extra Packages for Enterprise Linux 7.3 MB/s | 13 MB 00:01 2023-01-24T14:28:47.578 INFO:teuthology.orchestra.run.smithi179.stdout:lab-extras 450 kB/s | 24 kB 00:00 2023-01-24T14:28:51.061 INFO:teuthology.orchestra.run.smithi179.stdout:Dependencies resolved. 2023-01-24T14:28:51.064 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2023-01-24T14:28:51.064 INFO:teuthology.orchestra.run.smithi179.stdout: Package Arch Version Repository Size 2023-01-24T14:28:51.064 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2023-01-24T14:28:51.065 INFO:teuthology.orchestra.run.smithi179.stdout:Installing: 2023-01-24T14:28:51.065 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-radosgw x86_64 2:18.0.0-1954.g1a671c19.el8 ceph 14 M 2023-01-24T14:28:51.065 INFO:teuthology.orchestra.run.smithi179.stdout:Upgrading: 2023-01-24T14:28:51.066 INFO:teuthology.orchestra.run.smithi179.stdout: librados2 x86_64 2:18.0.0-1954.g1a671c19.el8 ceph 3.3 M 2023-01-24T14:28:51.066 INFO:teuthology.orchestra.run.smithi179.stdout: librbd1 x86_64 2:18.0.0-1954.g1a671c19.el8 ceph 3.5 M 2023-01-24T14:28:51.066 INFO:teuthology.orchestra.run.smithi179.stdout:Installing dependencies: 2023-01-24T14:28:51.067 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-base x86_64 2:18.0.0-1954.g1a671c19.el8 ceph 6.3 M 2023-01-24T14:28:51.067 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-common x86_64 2:18.0.0-1954.g1a671c19.el8 ceph 27 M 2023-01-24T14:28:51.067 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-selinux x86_64 2:18.0.0-1954.g1a671c19.el8 ceph 24 k 2023-01-24T14:28:51.068 INFO:teuthology.orchestra.run.smithi179.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-01-24T14:28:51.068 INFO:teuthology.orchestra.run.smithi179.stdout: libcephfs2 x86_64 2:18.0.0-1954.g1a671c19.el8 ceph 741 k 2023-01-24T14:28:51.068 INFO:teuthology.orchestra.run.smithi179.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-01-24T14:28:51.069 INFO:teuthology.orchestra.run.smithi179.stdout: librabbitmq x86_64 0.9.0-3.el8 CentOS-BaseOS 47 k 2023-01-24T14:28:51.069 INFO:teuthology.orchestra.run.smithi179.stdout: libradosstriper1 x86_64 2:18.0.0-1954.g1a671c19.el8 ceph 461 k 2023-01-24T14:28:51.069 INFO:teuthology.orchestra.run.smithi179.stdout: librdkafka x86_64 0.11.4-3.el8 CentOS-AppStream 354 k 2023-01-24T14:28:51.069 INFO:teuthology.orchestra.run.smithi179.stdout: librgw2 x86_64 2:18.0.0-1954.g1a671c19.el8 ceph 7.2 M 2023-01-24T14:28:51.070 INFO:teuthology.orchestra.run.smithi179.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-01-24T14:28:51.070 INFO:teuthology.orchestra.run.smithi179.stdout: lttng-ust x86_64 2.8.1-11.el8 CentOS-AppStream 259 k 2023-01-24T14:28:51.070 INFO:teuthology.orchestra.run.smithi179.stdout: python3-ceph-argparse 2023-01-24T14:28:51.071 INFO:teuthology.orchestra.run.smithi179.stdout: x86_64 2:18.0.0-1954.g1a671c19.el8 ceph 45 k 2023-01-24T14:28:51.071 INFO:teuthology.orchestra.run.smithi179.stdout: python3-ceph-common x86_64 2:18.0.0-1954.g1a671c19.el8 ceph 115 k 2023-01-24T14:28:51.071 INFO:teuthology.orchestra.run.smithi179.stdout: python3-cephfs x86_64 2:18.0.0-1954.g1a671c19.el8 ceph 190 k 2023-01-24T14:28:51.071 INFO:teuthology.orchestra.run.smithi179.stdout: python3-prettytable noarch 0.7.2-14.el8 CentOS-AppStream 44 k 2023-01-24T14:28:51.072 INFO:teuthology.orchestra.run.smithi179.stdout: python3-rados x86_64 2:18.0.0-1954.g1a671c19.el8 ceph 371 k 2023-01-24T14:28:51.072 INFO:teuthology.orchestra.run.smithi179.stdout: python3-rbd x86_64 2:18.0.0-1954.g1a671c19.el8 ceph 364 k 2023-01-24T14:28:51.072 INFO:teuthology.orchestra.run.smithi179.stdout: python3-rgw x86_64 2:18.0.0-1954.g1a671c19.el8 ceph 106 k 2023-01-24T14:28:51.073 INFO:teuthology.orchestra.run.smithi179.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-01-24T14:28:51.073 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-01-24T14:28:51.073 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction Summary 2023-01-24T14:28:51.074 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2023-01-24T14:28:51.074 INFO:teuthology.orchestra.run.smithi179.stdout:Install 21 Packages 2023-01-24T14:28:51.074 INFO:teuthology.orchestra.run.smithi179.stdout:Upgrade 2 Packages 2023-01-24T14:28:51.074 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-01-24T14:28:51.075 INFO:teuthology.orchestra.run.smithi179.stdout:Total download size: 67 M 2023-01-24T14:28:51.076 INFO:teuthology.orchestra.run.smithi179.stdout:Downloading Packages: 2023-01-24T14:28:52.054 INFO:teuthology.orchestra.run.smithi179.stdout:(1/23): ceph-base-18.0.0-1954.g1a671c19.el8.x86 6.4 MB/s | 6.3 MB 00:00 2023-01-24T14:28:52.129 INFO:teuthology.orchestra.run.smithi179.stdout:(2/23): ceph-selinux-18.0.0-1954.g1a671c19.el8. 325 kB/s | 24 kB 00:00 2023-01-24T14:28:52.505 INFO:teuthology.orchestra.run.smithi179.stdout:(3/23): libcephfs2-18.0.0-1954.g1a671c19.el8.x8 1.9 MB/s | 741 kB 00:00 2023-01-24T14:28:52.672 INFO:teuthology.orchestra.run.smithi179.stdout:(4/23): ceph-radosgw-18.0.0-1954.g1a671c19.el8. 9.0 MB/s | 14 MB 00:01 2023-01-24T14:28:52.697 INFO:teuthology.orchestra.run.smithi179.stdout:(5/23): libradosstriper1-18.0.0-1954.g1a671c19. 2.3 MB/s | 461 kB 00:00 2023-01-24T14:28:52.772 INFO:teuthology.orchestra.run.smithi179.stdout:(6/23): python3-ceph-argparse-18.0.0-1954.g1a67 593 kB/s | 45 kB 00:00 2023-01-24T14:28:52.847 INFO:teuthology.orchestra.run.smithi179.stdout:(7/23): python3-ceph-common-18.0.0-1954.g1a671c 1.5 MB/s | 115 kB 00:00 2023-01-24T14:28:52.931 INFO:teuthology.orchestra.run.smithi179.stdout:(8/23): python3-cephfs-18.0.0-1954.g1a671c19.el 2.2 MB/s | 190 kB 00:00 2023-01-24T14:28:53.023 INFO:teuthology.orchestra.run.smithi179.stdout:(9/23): python3-rados-18.0.0-1954.g1a671c19.el8 4.0 MB/s | 371 kB 00:00 2023-01-24T14:28:53.115 INFO:teuthology.orchestra.run.smithi179.stdout:(10/23): python3-rbd-18.0.0-1954.g1a671c19.el8. 3.9 MB/s | 364 kB 00:00 2023-01-24T14:28:53.190 INFO:teuthology.orchestra.run.smithi179.stdout:(11/23): python3-rgw-18.0.0-1954.g1a671c19.el8. 1.4 MB/s | 106 kB 00:00 2023-01-24T14:28:53.249 INFO:teuthology.orchestra.run.smithi179.stdout:(12/23): librdkafka-0.11.4-3.el8.x86_64.rpm 5.9 MB/s | 354 kB 00:00 2023-01-24T14:28:53.333 INFO:teuthology.orchestra.run.smithi179.stdout:(13/23): librgw2-18.0.0-1954.g1a671c19.el8.x86_ 11 MB/s | 7.2 MB 00:00 2023-01-24T14:28:53.366 INFO:teuthology.orchestra.run.smithi179.stdout:(14/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 2.2 MB/s | 259 kB 00:00 2023-01-24T14:28:53.391 INFO:teuthology.orchestra.run.smithi179.stdout:(15/23): librabbitmq-0.9.0-3.el8.x86_64.rpm 2.0 MB/s | 47 kB 00:00 2023-01-24T14:28:53.416 INFO:teuthology.orchestra.run.smithi179.stdout:(16/23): python3-prettytable-0.7.2-14.el8.noarc 524 kB/s | 44 kB 00:00 2023-01-24T14:28:53.714 INFO:teuthology.orchestra.run.smithi179.stdout:(17/23): ceph-common-18.0.0-1954.g1a671c19.el8. 10 MB/s | 27 MB 00:02 2023-01-24T14:28:53.834 INFO:teuthology.orchestra.run.smithi179.stdout:(18/23): liboath-2.6.2-3.el8.x86_64.rpm 141 kB/s | 59 kB 00:00 2023-01-24T14:28:53.851 INFO:teuthology.orchestra.run.smithi179.stdout:(19/23): libunwind-1.3.1-3.el8.x86_64.rpm 551 kB/s | 75 kB 00:00 2023-01-24T14:28:53.918 INFO:teuthology.orchestra.run.smithi179.stdout:(20/23): gperftools-libs-2.7-9.el8.x86_64.rpm 582 kB/s | 306 kB 00:00 2023-01-24T14:28:54.077 INFO:teuthology.orchestra.run.smithi179.stdout:(21/23): thrift-0.13.0-2.el8.x86_64.rpm 7.2 MB/s | 1.7 MB 00:00 2023-01-24T14:28:54.194 INFO:teuthology.orchestra.run.smithi179.stdout:(22/23): librados2-18.0.0-1954.g1a671c19.el8.x8 9.7 MB/s | 3.3 MB 00:00 2023-01-24T14:28:54.244 INFO:teuthology.orchestra.run.smithi179.stdout:(23/23): librbd1-18.0.0-1954.g1a671c19.el8.x86_ 11 MB/s | 3.5 MB 00:00 2023-01-24T14:28:54.249 INFO:teuthology.orchestra.run.smithi179.stdout:-------------------------------------------------------------------------------- 2023-01-24T14:28:54.249 INFO:teuthology.orchestra.run.smithi179.stdout:Total 21 MB/s | 67 MB 00:03 2023-01-24T14:28:54.289 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction check 2023-01-24T14:28:54.334 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction check succeeded. 2023-01-24T14:28:54.334 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction test 2023-01-24T14:28:54.953 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction test succeeded. 2023-01-24T14:28:54.957 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction 2023-01-24T14:28:56.024 INFO:teuthology.orchestra.run.smithi179.stdout: Preparing : 1/1 2023-01-24T14:28:56.246 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : liboath-2.6.2-3.el8.x86_64 1/25 2023-01-24T14:28:56.424 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 2/25 2023-01-24T14:28:56.817 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 2/25 2023-01-24T14:28:57.024 INFO:teuthology.orchestra.run.smithi179.stdout: Upgrading : librados2-2:18.0.0-1954.g1a671c19.el8.x86_64 3/25 2023-01-24T14:28:57.109 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: librados2-2:18.0.0-1954.g1a671c19.el8.x86_64 3/25 2023-01-24T14:28:57.369 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-rados-2:18.0.0-1954.g1a671c19.el8.x86_64 4/25 2023-01-24T14:28:57.532 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : libcephfs2-2:18.0.0-1954.g1a671c19.el8.x86_64 5/25 2023-01-24T14:28:57.957 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: libcephfs2-2:18.0.0-1954.g1a671c19.el8.x86_64 5/25 2023-01-24T14:28:58.157 INFO:teuthology.orchestra.run.smithi179.stdout: Upgrading : librbd1-2:18.0.0-1954.g1a671c19.el8.x86_64 6/25 2023-01-24T14:28:58.374 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: librbd1-2:18.0.0-1954.g1a671c19.el8.x86_64 6/25 2023-01-24T14:28:58.569 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : thrift-0.13.0-2.el8.x86_64 7/25 2023-01-24T14:28:59.137 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 8/25 2023-01-24T14:28:59.274 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 9/25 2023-01-24T14:28:59.345 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 9/25 2023-01-24T14:28:59.459 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-ceph-argparse-2:18.0.0-1954.g1a671c19.el8. 10/25 2023-01-24T14:28:59.684 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-cephfs-2:18.0.0-1954.g1a671c19.el8.x86_64 11/25 2023-01-24T14:28:59.952 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-rbd-2:18.0.0-1954.g1a671c19.el8.x86_64 12/25 2023-01-24T14:29:00.098 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : libradosstriper1-2:18.0.0-1954.g1a671c19.el8.x86_6 13/25 2023-01-24T14:29:00.167 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: libradosstriper1-2:18.0.0-1954.g1a671c19.el8.x86_6 13/25 2023-01-24T14:29:00.372 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 14/25 2023-01-24T14:29:01.311 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 15/25 2023-01-24T14:29:01.490 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : librgw2-2:18.0.0-1954.g1a671c19.el8.x86_64 16/25 2023-01-24T14:29:01.547 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: librgw2-2:18.0.0-1954.g1a671c19.el8.x86_64 16/25 2023-01-24T14:29:01.739 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-rgw-2:18.0.0-1954.g1a671c19.el8.x86_64 17/25 2023-01-24T14:29:01.955 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 18/25 2023-01-24T14:29:02.149 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-ceph-common-2:18.0.0-1954.g1a671c19.el8.x8 19/25 2023-01-24T14:29:06.334 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: ceph-common-2:18.0.0-1954.g1a671c19.el8.x86_64 20/25 2023-01-24T14:29:06.556 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-common-2:18.0.0-1954.g1a671c19.el8.x86_64 20/25 2023-01-24T14:29:07.309 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: ceph-common-2:18.0.0-1954.g1a671c19.el8.x86_64 20/25 2023-01-24T14:29:07.646 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-base-2:18.0.0-1954.g1a671c19.el8.x86_64 21/25 2023-01-24T14:29:07.738 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: ceph-base-2:18.0.0-1954.g1a671c19.el8.x86_64 21/25 2023-01-24T14:29:07.889 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-selinux-2:18.0.0-1954.g1a671c19.el8.x86_64 22/25 2023-01-24T14:29:28.095 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: ceph-selinux-2:18.0.0-1954.g1a671c19.el8.x86_64 22/25 2023-01-24T14:29:28.096 INFO:teuthology.orchestra.run.smithi179.stdout:skipping the directory /sys 2023-01-24T14:29:28.096 INFO:teuthology.orchestra.run.smithi179.stdout:skipping the directory /proc 2023-01-24T14:29:28.096 INFO:teuthology.orchestra.run.smithi179.stdout:skipping the directory /dev 2023-01-24T14:29:28.096 INFO:teuthology.orchestra.run.smithi179.stdout:skipping the directory /run 2023-01-24T14:29:28.097 INFO:teuthology.orchestra.run.smithi179.stdout:skipping the directory /mnt 2023-01-24T14:29:28.097 INFO:teuthology.orchestra.run.smithi179.stdout:skipping the directory /var/tmp 2023-01-24T14:29:28.097 INFO:teuthology.orchestra.run.smithi179.stdout:skipping the directory /home 2023-01-24T14:29:28.097 INFO:teuthology.orchestra.run.smithi179.stdout:skipping the directory /tmp 2023-01-24T14:29:28.097 INFO:teuthology.orchestra.run.smithi179.stdout:skipping the directory /dev 2023-01-24T14:29:28.098 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-01-24T14:29:29.760 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-radosgw-2:18.0.0-1954.g1a671c19.el8.x86_64 23/25 2023-01-24T14:29:29.781 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-1954.g1a671c19.el8.x86_64 23/25 2023-01-24T14:29:29.781 INFO:teuthology.orchestra.run.smithi179.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2023-01-24T14:29:29.971 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2023-01-24T14:29:29.972 INFO:teuthology.orchestra.run.smithi179.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2023-01-24T14:29:30.677 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2023-01-24T14:29:30.677 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-base-2:18.0.0-1954.g1a671c19.el8.x86_64 1/25 2023-01-24T14:29:30.677 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-common-2:18.0.0-1954.g1a671c19.el8.x86_64 2/25 2023-01-24T14:29:30.678 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-radosgw-2:18.0.0-1954.g1a671c19.el8.x86_64 3/25 2023-01-24T14:29:30.678 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-selinux-2:18.0.0-1954.g1a671c19.el8.x86_64 4/25 2023-01-24T14:29:30.678 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : libcephfs2-2:18.0.0-1954.g1a671c19.el8.x86_64 5/25 2023-01-24T14:29:30.678 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : libradosstriper1-2:18.0.0-1954.g1a671c19.el8.x86_6 6/25 2023-01-24T14:29:30.678 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : librgw2-2:18.0.0-1954.g1a671c19.el8.x86_64 7/25 2023-01-24T14:29:30.679 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-ceph-argparse-2:18.0.0-1954.g1a671c19.el8. 8/25 2023-01-24T14:29:30.679 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-ceph-common-2:18.0.0-1954.g1a671c19.el8.x8 9/25 2023-01-24T14:29:30.679 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-cephfs-2:18.0.0-1954.g1a671c19.el8.x86_64 10/25 2023-01-24T14:29:30.679 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-rados-2:18.0.0-1954.g1a671c19.el8.x86_64 11/25 2023-01-24T14:29:30.680 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-rbd-2:18.0.0-1954.g1a671c19.el8.x86_64 12/25 2023-01-24T14:29:30.680 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-rgw-2:18.0.0-1954.g1a671c19.el8.x86_64 13/25 2023-01-24T14:29:30.680 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 14/25 2023-01-24T14:29:30.680 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 15/25 2023-01-24T14:29:30.680 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 16/25 2023-01-24T14:29:30.681 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 17/25 2023-01-24T14:29:30.681 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 18/25 2023-01-24T14:29:30.681 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 19/25 2023-01-24T14:29:30.681 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 20/25 2023-01-24T14:29:30.681 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 21/25 2023-01-24T14:29:30.682 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : librados2-2:18.0.0-1954.g1a671c19.el8.x86_64 22/25 2023-01-24T14:29:30.682 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2023-01-24T14:29:30.682 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : librbd1-2:18.0.0-1954.g1a671c19.el8.x86_64 24/25 2023-01-24T14:29:31.513 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2023-01-24T14:29:31.513 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-01-24T14:29:31.513 INFO:teuthology.orchestra.run.smithi179.stdout:Upgraded: 2023-01-24T14:29:31.513 INFO:teuthology.orchestra.run.smithi179.stdout: librados2-2:18.0.0-1954.g1a671c19.el8.x86_64 2023-01-24T14:29:31.514 INFO:teuthology.orchestra.run.smithi179.stdout: librbd1-2:18.0.0-1954.g1a671c19.el8.x86_64 2023-01-24T14:29:31.514 INFO:teuthology.orchestra.run.smithi179.stdout:Installed: 2023-01-24T14:29:31.514 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-base-2:18.0.0-1954.g1a671c19.el8.x86_64 2023-01-24T14:29:31.514 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-common-2:18.0.0-1954.g1a671c19.el8.x86_64 2023-01-24T14:29:31.515 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-radosgw-2:18.0.0-1954.g1a671c19.el8.x86_64 2023-01-24T14:29:31.515 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-selinux-2:18.0.0-1954.g1a671c19.el8.x86_64 2023-01-24T14:29:31.515 INFO:teuthology.orchestra.run.smithi179.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-01-24T14:29:31.515 INFO:teuthology.orchestra.run.smithi179.stdout: libcephfs2-2:18.0.0-1954.g1a671c19.el8.x86_64 2023-01-24T14:29:31.515 INFO:teuthology.orchestra.run.smithi179.stdout: liboath-2.6.2-3.el8.x86_64 2023-01-24T14:29:31.516 INFO:teuthology.orchestra.run.smithi179.stdout: librabbitmq-0.9.0-3.el8.x86_64 2023-01-24T14:29:31.516 INFO:teuthology.orchestra.run.smithi179.stdout: libradosstriper1-2:18.0.0-1954.g1a671c19.el8.x86_64 2023-01-24T14:29:31.516 INFO:teuthology.orchestra.run.smithi179.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-01-24T14:29:31.516 INFO:teuthology.orchestra.run.smithi179.stdout: librgw2-2:18.0.0-1954.g1a671c19.el8.x86_64 2023-01-24T14:29:31.516 INFO:teuthology.orchestra.run.smithi179.stdout: libunwind-1.3.1-3.el8.x86_64 2023-01-24T14:29:31.517 INFO:teuthology.orchestra.run.smithi179.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-01-24T14:29:31.517 INFO:teuthology.orchestra.run.smithi179.stdout: python3-ceph-argparse-2:18.0.0-1954.g1a671c19.el8.x86_64 2023-01-24T14:29:31.517 INFO:teuthology.orchestra.run.smithi179.stdout: python3-ceph-common-2:18.0.0-1954.g1a671c19.el8.x86_64 2023-01-24T14:29:31.517 INFO:teuthology.orchestra.run.smithi179.stdout: python3-cephfs-2:18.0.0-1954.g1a671c19.el8.x86_64 2023-01-24T14:29:31.517 INFO:teuthology.orchestra.run.smithi179.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-01-24T14:29:31.518 INFO:teuthology.orchestra.run.smithi179.stdout: python3-rados-2:18.0.0-1954.g1a671c19.el8.x86_64 2023-01-24T14:29:31.518 INFO:teuthology.orchestra.run.smithi179.stdout: python3-rbd-2:18.0.0-1954.g1a671c19.el8.x86_64 2023-01-24T14:29:31.518 INFO:teuthology.orchestra.run.smithi179.stdout: python3-rgw-2:18.0.0-1954.g1a671c19.el8.x86_64 2023-01-24T14:29:31.518 INFO:teuthology.orchestra.run.smithi179.stdout: thrift-0.13.0-2.el8.x86_64 2023-01-24T14:29:31.518 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-01-24T14:29:31.519 INFO:teuthology.orchestra.run.smithi179.stdout:Complete! 2023-01-24T14:29:31.860 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum -y install ceph-test 2023-01-24T14:29:32.593 INFO:teuthology.orchestra.run.smithi179.stdout:Last metadata expiration check: 0:00:45 ago on Tue 24 Jan 2023 02:28:47 PM UTC. 2023-01-24T14:29:33.734 INFO:teuthology.orchestra.run.smithi179.stdout:Dependencies resolved. 2023-01-24T14:29:33.736 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2023-01-24T14:29:33.736 INFO:teuthology.orchestra.run.smithi179.stdout: Package Arch Version Repository Size 2023-01-24T14:29:33.736 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2023-01-24T14:29:33.736 INFO:teuthology.orchestra.run.smithi179.stdout:Installing: 2023-01-24T14:29:33.737 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-test x86_64 2:18.0.0-1954.g1a671c19.el8 ceph 149 M 2023-01-24T14:29:33.737 INFO:teuthology.orchestra.run.smithi179.stdout:Installing dependencies: 2023-01-24T14:29:33.737 INFO:teuthology.orchestra.run.smithi179.stdout: cryptopp x86_64 8.6.0-1.el8 epel 1.3 M 2023-01-24T14:29:33.737 INFO:teuthology.orchestra.run.smithi179.stdout: jq x86_64 1.6-6.el8 CentOS-AppStream 202 k 2023-01-24T14:29:33.737 INFO:teuthology.orchestra.run.smithi179.stdout: libasan6 x86_64 11.2.1-9.1.el8 CentOS-AppStream 402 k 2023-01-24T14:29:33.738 INFO:teuthology.orchestra.run.smithi179.stdout: libatomic x86_64 8.5.0-18.el8 CentOS-BaseOS 25 k 2023-01-24T14:29:33.738 INFO:teuthology.orchestra.run.smithi179.stdout: libcephsqlite x86_64 2:18.0.0-1954.g1a671c19.el8 ceph 134 k 2023-01-24T14:29:33.738 INFO:teuthology.orchestra.run.smithi179.stdout: libubsan x86_64 8.5.0-18.el8 CentOS-BaseOS 152 k 2023-01-24T14:29:33.738 INFO:teuthology.orchestra.run.smithi179.stdout: oniguruma x86_64 6.8.2-2.el8 CentOS-AppStream 187 k 2023-01-24T14:29:33.738 INFO:teuthology.orchestra.run.smithi179.stdout: socat x86_64 1.7.4.1-1.el8 CentOS-AppStream 323 k 2023-01-24T14:29:33.739 INFO:teuthology.orchestra.run.smithi179.stdout: xmlstarlet x86_64 1.6.1-20.el8 CentOS-AppStream 71 k 2023-01-24T14:29:33.739 INFO:teuthology.orchestra.run.smithi179.stdout: yaml-cpp x86_64 0.6.2-3.el8 epel 171 k 2023-01-24T14:29:33.739 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-01-24T14:29:33.739 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction Summary 2023-01-24T14:29:33.739 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2023-01-24T14:29:33.740 INFO:teuthology.orchestra.run.smithi179.stdout:Install 11 Packages 2023-01-24T14:29:33.740 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-01-24T14:29:33.741 INFO:teuthology.orchestra.run.smithi179.stdout:Total download size: 152 M 2023-01-24T14:29:33.741 INFO:teuthology.orchestra.run.smithi179.stdout:Installed size: 1.7 G 2023-01-24T14:29:33.741 INFO:teuthology.orchestra.run.smithi179.stdout:Downloading Packages: 2023-01-24T14:29:33.817 INFO:teuthology.orchestra.run.smithi179.stdout:(1/11): jq-1.6-6.el8.x86_64.rpm 2.6 MB/s | 202 kB 00:00 2023-01-24T14:29:33.850 INFO:teuthology.orchestra.run.smithi179.stdout:(2/11): libasan6-11.2.1-9.1.el8.x86_64.rpm 12 MB/s | 402 kB 00:00 2023-01-24T14:29:33.884 INFO:teuthology.orchestra.run.smithi179.stdout:(3/11): oniguruma-6.8.2-2.el8.x86_64.rpm 5.5 MB/s | 187 kB 00:00 2023-01-24T14:29:33.918 INFO:teuthology.orchestra.run.smithi179.stdout:(4/11): socat-1.7.4.1-1.el8.x86_64.rpm 9.5 MB/s | 323 kB 00:00 2023-01-24T14:29:33.934 INFO:teuthology.orchestra.run.smithi179.stdout:(5/11): xmlstarlet-1.6.1-20.el8.x86_64.rpm 4.2 MB/s | 71 kB 00:00 2023-01-24T14:29:33.960 INFO:teuthology.orchestra.run.smithi179.stdout:(6/11): libatomic-8.5.0-18.el8.x86_64.rpm 1.0 MB/s | 25 kB 00:00 2023-01-24T14:29:33.985 INFO:teuthology.orchestra.run.smithi179.stdout:(7/11): libcephsqlite-18.0.0-1954.g1a671c19.el8 549 kB/s | 134 kB 00:00 2023-01-24T14:29:34.010 INFO:teuthology.orchestra.run.smithi179.stdout:(8/11): libubsan-8.5.0-18.el8.x86_64.rpm 2.9 MB/s | 152 kB 00:00 2023-01-24T14:29:34.394 INFO:teuthology.orchestra.run.smithi179.stdout:(9/11): yaml-cpp-0.6.2-3.el8.x86_64.rpm 445 kB/s | 171 kB 00:00 2023-01-24T14:29:35.436 INFO:teuthology.orchestra.run.smithi179.stdout:(10/11): cryptopp-8.6.0-1.el8.x86_64.rpm 899 kB/s | 1.3 MB 00:01 2023-01-24T14:29:59.367 INFO:teuthology.orchestra.run.smithi179.stdout:(11/11): ceph-test-18.0.0-1954.g1a671c19.el8.x8 5.8 MB/s | 149 MB 00:25 2023-01-24T14:29:59.369 INFO:teuthology.orchestra.run.smithi179.stdout:-------------------------------------------------------------------------------- 2023-01-24T14:29:59.369 INFO:teuthology.orchestra.run.smithi179.stdout:Total 5.9 MB/s | 152 MB 00:25 2023-01-24T14:29:59.438 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction check 2023-01-24T14:29:59.461 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction check succeeded. 2023-01-24T14:29:59.461 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction test 2023-01-24T14:30:00.159 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction test succeeded. 2023-01-24T14:30:00.162 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction 2023-01-24T14:30:01.181 INFO:teuthology.orchestra.run.smithi179.stdout: Preparing : 1/1 2023-01-24T14:30:01.498 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : yaml-cpp-0.6.2-3.el8.x86_64 1/11 2023-01-24T14:30:01.677 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : cryptopp-8.6.0-1.el8.x86_64 2/11 2023-01-24T14:30:01.806 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : libubsan-8.5.0-18.el8.x86_64 3/11 2023-01-24T14:30:01.872 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: libubsan-8.5.0-18.el8.x86_64 3/11 2023-01-24T14:30:01.981 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : libatomic-8.5.0-18.el8.x86_64 4/11 2023-01-24T14:30:02.054 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: libatomic-8.5.0-18.el8.x86_64 4/11 2023-01-24T14:30:02.277 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 5/11 2023-01-24T14:30:02.492 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 6/11 2023-01-24T14:30:02.673 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 7/11 2023-01-24T14:30:02.750 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 7/11 2023-01-24T14:30:02.978 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : jq-1.6-6.el8.x86_64 8/11 2023-01-24T14:30:03.123 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : libasan6-11.2.1-9.1.el8.x86_64 9/11 2023-01-24T14:30:03.179 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: libasan6-11.2.1-9.1.el8.x86_64 9/11 2023-01-24T14:30:03.339 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : libcephsqlite-2:18.0.0-1954.g1a671c19.el8.x86_64 10/11 2023-01-24T14:30:26.216 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: libcephsqlite-2:18.0.0-1954.g1a671c19.el8.x86_64 10/11 2023-01-24T14:30:26.450 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-test-2:18.0.0-1954.g1a671c19.el8.x86_64 11/11 2023-01-24T14:30:26.760 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: ceph-test-2:18.0.0-1954.g1a671c19.el8.x86_64 11/11 2023-01-24T14:30:26.760 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-test-2:18.0.0-1954.g1a671c19.el8.x86_64 1/11 2023-01-24T14:30:26.761 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : libcephsqlite-2:18.0.0-1954.g1a671c19.el8.x86_64 2/11 2023-01-24T14:30:26.761 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : jq-1.6-6.el8.x86_64 3/11 2023-01-24T14:30:26.761 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : libasan6-11.2.1-9.1.el8.x86_64 4/11 2023-01-24T14:30:26.761 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 5/11 2023-01-24T14:30:26.761 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 6/11 2023-01-24T14:30:26.762 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 7/11 2023-01-24T14:30:26.762 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : libatomic-8.5.0-18.el8.x86_64 8/11 2023-01-24T14:30:26.762 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : libubsan-8.5.0-18.el8.x86_64 9/11 2023-01-24T14:30:26.762 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : cryptopp-8.6.0-1.el8.x86_64 10/11 2023-01-24T14:30:27.300 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : yaml-cpp-0.6.2-3.el8.x86_64 11/11 2023-01-24T14:30:27.300 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-01-24T14:30:27.300 INFO:teuthology.orchestra.run.smithi179.stdout:Installed: 2023-01-24T14:30:27.300 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-test-2:18.0.0-1954.g1a671c19.el8.x86_64 2023-01-24T14:30:27.301 INFO:teuthology.orchestra.run.smithi179.stdout: cryptopp-8.6.0-1.el8.x86_64 2023-01-24T14:30:27.301 INFO:teuthology.orchestra.run.smithi179.stdout: jq-1.6-6.el8.x86_64 2023-01-24T14:30:27.301 INFO:teuthology.orchestra.run.smithi179.stdout: libasan6-11.2.1-9.1.el8.x86_64 2023-01-24T14:30:27.301 INFO:teuthology.orchestra.run.smithi179.stdout: libatomic-8.5.0-18.el8.x86_64 2023-01-24T14:30:27.301 INFO:teuthology.orchestra.run.smithi179.stdout: libcephsqlite-2:18.0.0-1954.g1a671c19.el8.x86_64 2023-01-24T14:30:27.302 INFO:teuthology.orchestra.run.smithi179.stdout: libubsan-8.5.0-18.el8.x86_64 2023-01-24T14:30:27.302 INFO:teuthology.orchestra.run.smithi179.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-01-24T14:30:27.302 INFO:teuthology.orchestra.run.smithi179.stdout: socat-1.7.4.1-1.el8.x86_64 2023-01-24T14:30:27.302 INFO:teuthology.orchestra.run.smithi179.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-01-24T14:30:27.303 INFO:teuthology.orchestra.run.smithi179.stdout: yaml-cpp-0.6.2-3.el8.x86_64 2023-01-24T14:30:27.303 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-01-24T14:30:27.303 INFO:teuthology.orchestra.run.smithi179.stdout:Complete! 2023-01-24T14:30:27.584 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum -y install ceph 2023-01-24T14:30:28.312 INFO:teuthology.orchestra.run.smithi179.stdout:Last metadata expiration check: 0:01:41 ago on Tue 24 Jan 2023 02:28:47 PM UTC. 2023-01-24T14:30:29.468 INFO:teuthology.orchestra.run.smithi179.stdout:Dependencies resolved. 2023-01-24T14:30:29.472 INFO:teuthology.orchestra.run.smithi179.stdout:======================================================================================= 2023-01-24T14:30:29.472 INFO:teuthology.orchestra.run.smithi179.stdout: Package Arch Version Repository Size 2023-01-24T14:30:29.472 INFO:teuthology.orchestra.run.smithi179.stdout:======================================================================================= 2023-01-24T14:30:29.473 INFO:teuthology.orchestra.run.smithi179.stdout:Installing: 2023-01-24T14:30:29.473 INFO:teuthology.orchestra.run.smithi179.stdout: ceph x86_64 2:18.0.0-1954.g1a671c19.el8 ceph 6.4 k 2023-01-24T14:30:29.473 INFO:teuthology.orchestra.run.smithi179.stdout:Installing dependencies: 2023-01-24T14:30:29.474 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-mds x86_64 2:18.0.0-1954.g1a671c19.el8 ceph 2.2 M 2023-01-24T14:30:29.474 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-mgr x86_64 2:18.0.0-1954.g1a671c19.el8 ceph 1.5 M 2023-01-24T14:30:29.475 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-mgr-modules-core noarch 2:18.0.0-1954.g1a671c19.el8 ceph-noarch 236 k 2023-01-24T14:30:29.475 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-mon x86_64 2:18.0.0-1954.g1a671c19.el8 ceph 7.4 M 2023-01-24T14:30:29.475 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-osd x86_64 2:18.0.0-1954.g1a671c19.el8 ceph 151 M 2023-01-24T14:30:29.476 INFO:teuthology.orchestra.run.smithi179.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-01-24T14:30:29.476 INFO:teuthology.orchestra.run.smithi179.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-01-24T14:30:29.476 INFO:teuthology.orchestra.run.smithi179.stdout: python3-cffi x86_64 1.11.5-5.el8 CentOS-BaseOS 237 k 2023-01-24T14:30:29.477 INFO:teuthology.orchestra.run.smithi179.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-01-24T14:30:29.477 INFO:teuthology.orchestra.run.smithi179.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-01-24T14:30:29.477 INFO:teuthology.orchestra.run.smithi179.stdout: python3-cryptography x86_64 3.2.1-5.el8 CentOS-BaseOS 559 k 2023-01-24T14:30:29.478 INFO:teuthology.orchestra.run.smithi179.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-01-24T14:30:29.478 INFO:teuthology.orchestra.run.smithi179.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-01-24T14:30:29.478 INFO:teuthology.orchestra.run.smithi179.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-01-24T14:30:29.479 INFO:teuthology.orchestra.run.smithi179.stdout: python3-mako noarch 1.0.6-13.el8 CentOS-AppStream 157 k 2023-01-24T14:30:29.480 INFO:teuthology.orchestra.run.smithi179.stdout: python3-markupsafe x86_64 0.23-19.el8 CentOS-AppStream 39 k 2023-01-24T14:30:29.480 INFO:teuthology.orchestra.run.smithi179.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-01-24T14:30:29.480 INFO:teuthology.orchestra.run.smithi179.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-01-24T14:30:29.480 INFO:teuthology.orchestra.run.smithi179.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-01-24T14:30:29.480 INFO:teuthology.orchestra.run.smithi179.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 CentOS-AppStream 103 k 2023-01-24T14:30:29.481 INFO:teuthology.orchestra.run.smithi179.stdout: python3-pycparser noarch 2.14-14.el8 CentOS-BaseOS 109 k 2023-01-24T14:30:29.481 INFO:teuthology.orchestra.run.smithi179.stdout: python3-pytz noarch 2017.2-9.el8 CentOS-AppStream 54 k 2023-01-24T14:30:29.481 INFO:teuthology.orchestra.run.smithi179.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-01-24T14:30:29.481 INFO:teuthology.orchestra.run.smithi179.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-01-24T14:30:29.482 INFO:teuthology.orchestra.run.smithi179.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-01-24T14:30:29.482 INFO:teuthology.orchestra.run.smithi179.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-01-24T14:30:29.482 INFO:teuthology.orchestra.run.smithi179.stdout: python3-waitress noarch 1.2.1-2.el8.1 epel 241 k 2023-01-24T14:30:29.482 INFO:teuthology.orchestra.run.smithi179.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-01-24T14:30:29.482 INFO:teuthology.orchestra.run.smithi179.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-01-24T14:30:29.483 INFO:teuthology.orchestra.run.smithi179.stdout: python3-werkzeug noarch 0.12.2-4.el8 CentOS-AppStream 457 k 2023-01-24T14:30:29.483 INFO:teuthology.orchestra.run.smithi179.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-01-24T14:30:29.483 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-01-24T14:30:29.483 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction Summary 2023-01-24T14:30:29.483 INFO:teuthology.orchestra.run.smithi179.stdout:======================================================================================= 2023-01-24T14:30:29.484 INFO:teuthology.orchestra.run.smithi179.stdout:Install 32 Packages 2023-01-24T14:30:29.484 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-01-24T14:30:29.485 INFO:teuthology.orchestra.run.smithi179.stdout:Total download size: 165 M 2023-01-24T14:30:29.485 INFO:teuthology.orchestra.run.smithi179.stdout:Installed size: 2.2 G 2023-01-24T14:30:29.485 INFO:teuthology.orchestra.run.smithi179.stdout:Downloading Packages: 2023-01-24T14:30:29.605 INFO:teuthology.orchestra.run.smithi179.stdout:(1/32): ceph-18.0.0-1954.g1a671c19.el8.x86_64.r 50 kB/s | 6.4 kB 00:00 2023-01-24T14:30:29.830 INFO:teuthology.orchestra.run.smithi179.stdout:(2/32): ceph-mds-18.0.0-1954.g1a671c19.el8.x86_ 6.2 MB/s | 2.2 MB 00:00 2023-01-24T14:30:29.872 INFO:teuthology.orchestra.run.smithi179.stdout:(3/32): ceph-mgr-18.0.0-1954.g1a671c19.el8.x86_ 3.7 MB/s | 1.5 MB 00:00 2023-01-24T14:30:29.965 INFO:teuthology.orchestra.run.smithi179.stdout:(4/32): ceph-mgr-modules-core-18.0.0-1954.g1a67 2.5 MB/s | 236 kB 00:00 2023-01-24T14:30:30.023 INFO:teuthology.orchestra.run.smithi179.stdout:(5/32): python3-mako-1.0.6-13.el8.noarch.rpm 2.6 MB/s | 157 kB 00:00 2023-01-24T14:30:30.048 INFO:teuthology.orchestra.run.smithi179.stdout:(6/32): python3-markupsafe-0.23-19.el8.x86_64.r 1.5 MB/s | 39 kB 00:00 2023-01-24T14:30:30.098 INFO:teuthology.orchestra.run.smithi179.stdout:(7/32): python3-pyOpenSSL-19.0.0-1.el8.noarch.r 2.0 MB/s | 103 kB 00:00 2023-01-24T14:30:30.132 INFO:teuthology.orchestra.run.smithi179.stdout:(8/32): python3-pytz-2017.2-9.el8.noarch.rpm 1.6 MB/s | 54 kB 00:00 2023-01-24T14:30:30.166 INFO:teuthology.orchestra.run.smithi179.stdout:(9/32): python3-werkzeug-0.12.2-4.el8.noarch.rp 13 MB/s | 457 kB 00:00 2023-01-24T14:30:30.191 INFO:teuthology.orchestra.run.smithi179.stdout:(10/32): python3-cffi-1.11.5-5.el8.x86_64.rpm 9.3 MB/s | 237 kB 00:00 2023-01-24T14:30:30.225 INFO:teuthology.orchestra.run.smithi179.stdout:(11/32): python3-cryptography-3.2.1-5.el8.x86_6 16 MB/s | 559 kB 00:00 2023-01-24T14:30:30.250 INFO:teuthology.orchestra.run.smithi179.stdout:(12/32): python3-pycparser-2.14-14.el8.noarch.r 4.3 MB/s | 109 kB 00:00 2023-01-24T14:30:30.606 INFO:teuthology.orchestra.run.smithi179.stdout:(13/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 123 kB/s | 44 kB 00:00 2023-01-24T14:30:30.759 INFO:teuthology.orchestra.run.smithi179.stdout:(14/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 1.2 MB/s | 185 kB 00:00 2023-01-24T14:30:30.868 INFO:teuthology.orchestra.run.smithi179.stdout:(15/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 1.6 MB/s | 173 kB 00:00 2023-01-24T14:30:31.043 INFO:teuthology.orchestra.run.smithi179.stdout:(16/32): python3-cherrypy-18.4.0-1.el8.noarch.r 2.1 MB/s | 384 kB 00:00 2023-01-24T14:30:31.185 INFO:teuthology.orchestra.run.smithi179.stdout:(17/32): ceph-mon-18.0.0-1954.g1a671c19.el8.x86 4.7 MB/s | 7.4 MB 00:01 2023-01-24T14:30:31.202 INFO:teuthology.orchestra.run.smithi179.stdout:(18/32): python3-jaraco-6.2-6.el8.noarch.rpm 70 kB/s | 11 kB 00:00 2023-01-24T14:30:31.227 INFO:teuthology.orchestra.run.smithi179.stdout:(19/32): python3-jaraco-functools-2.0-4.el8.noa 421 kB/s | 18 kB 00:00 2023-01-24T14:30:31.269 INFO:teuthology.orchestra.run.smithi179.stdout:(20/32): python3-more-itertools-7.2.0-3.el8.noa 1.4 MB/s | 59 kB 00:00 2023-01-24T14:30:31.344 INFO:teuthology.orchestra.run.smithi179.stdout:(21/32): python3-logutils-0.3.5-11.el8.noarch.r 343 kB/s | 49 kB 00:00 2023-01-24T14:30:31.386 INFO:teuthology.orchestra.run.smithi179.stdout:(22/32): python3-portend-2.6-1.el8.noarch.rpm 373 kB/s | 16 kB 00:00 2023-01-24T14:30:31.420 INFO:teuthology.orchestra.run.smithi179.stdout:(23/32): python3-pecan-1.3.2-9.el8.noarch.rpm 1.8 MB/s | 283 kB 00:00 2023-01-24T14:30:31.438 INFO:teuthology.orchestra.run.smithi179.stdout:(24/32): python3-simplegeneric-0.8.1-17.el8.noa 372 kB/s | 19 kB 00:00 2023-01-24T14:30:31.471 INFO:teuthology.orchestra.run.smithi179.stdout:(25/32): python3-singledispatch-3.4.0.3-18.el8. 477 kB/s | 24 kB 00:00 2023-01-24T14:30:31.496 INFO:teuthology.orchestra.run.smithi179.stdout:(26/32): python3-tempora-1.14.1-5.el8.noarch.rp 499 kB/s | 29 kB 00:00 2023-01-24T14:30:31.521 INFO:teuthology.orchestra.run.smithi179.stdout:(27/32): python3-trustme-0.6.0-4.el8.noarch.rpm 546 kB/s | 27 kB 00:00 2023-01-24T14:30:31.638 INFO:teuthology.orchestra.run.smithi179.stdout:(28/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 2.1 MB/s | 251 kB 00:00 2023-01-24T14:30:31.664 INFO:teuthology.orchestra.run.smithi179.stdout:(29/32): python3-waitress-1.2.1-2.el8.1.noarch. 1.4 MB/s | 241 kB 00:00 2023-01-24T14:30:31.689 INFO:teuthology.orchestra.run.smithi179.stdout:(30/32): python3-webtest-2.0.33-1.el8.noarch.rp 1.7 MB/s | 85 kB 00:00 2023-01-24T14:30:31.714 INFO:teuthology.orchestra.run.smithi179.stdout:(31/32): python3-zc-lockfile-2.0-2.el8.noarch.r 457 kB/s | 23 kB 00:00 2023-01-24T14:38:10.398 INFO:teuthology.orchestra.run.smithi179.stdout:[MIRROR] ceph-osd-18.0.0-1954.g1a671c19.el8.x86_64.rpm: Curl error (18): Transferred a partial file for https://1.chacra.ceph.com/r/ceph/main/1a671c198559f54389cd5afa22085a0c7b29a1ef/centos/8/flavors/crimson/x86_64/ceph-osd-18.0.0-1954.g1a671c19.el8.x86_64.rpm [transfer closed with 8801389 bytes remaining to read] 2023-01-24T14:39:34.207 INFO:teuthology.orchestra.run.smithi179.stdout:(32/32): ceph-osd-18.0.0-1954.g1a671c19.el8.x86 283 kB/s | 151 MB 09:04 2023-01-24T14:39:34.208 INFO:teuthology.orchestra.run.smithi179.stdout:-------------------------------------------------------------------------------- 2023-01-24T14:39:34.208 INFO:teuthology.orchestra.run.smithi179.stdout:Total 311 kB/s | 165 MB 09:04 2023-01-24T14:39:34.286 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction check 2023-01-24T14:39:34.326 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction check succeeded. 2023-01-24T14:39:34.327 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction test 2023-01-24T14:39:35.234 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction test succeeded. 2023-01-24T14:39:35.237 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction 2023-01-24T14:39:37.025 INFO:teuthology.orchestra.run.smithi179.stdout: Preparing : 1/1 2023-01-24T14:39:37.245 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2023-01-24T14:39:37.456 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2023-01-24T14:39:37.694 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2023-01-24T14:39:37.946 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-waitress-1.2.1-2.el8.1.noarch 4/32 2023-01-24T14:39:38.127 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2023-01-24T14:39:38.299 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2023-01-24T14:39:38.492 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2023-01-24T14:39:38.671 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2023-01-24T14:39:38.885 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2023-01-24T14:39:39.173 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2023-01-24T14:39:39.419 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2023-01-24T14:39:39.649 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2023-01-24T14:39:40.072 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 13/32 2023-01-24T14:39:40.351 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-cryptography-3.2.1-5.el8.x86_64 14/32 2023-01-24T14:39:40.555 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2023-01-24T14:39:40.821 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2023-01-24T14:39:41.065 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2023-01-24T14:39:41.373 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2023-01-24T14:39:41.578 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2023-01-24T14:39:41.824 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 20/32 2023-01-24T14:39:42.061 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2023-01-24T14:39:42.319 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2023-01-24T14:39:42.593 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2023-01-24T14:39:42.816 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2023-01-24T14:39:43.155 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 25/32 2023-01-24T14:39:43.512 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2023-01-24T14:39:43.927 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-1954.g1a671c19.el8. 27/32 2023-01-24T14:39:44.157 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-mgr-2:18.0.0-1954.g1a671c19.el8.x86_64 28/32 2023-01-24T14:40:11.178 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: ceph-mgr-2:18.0.0-1954.g1a671c19.el8.x86_64 28/32 2023-01-24T14:40:11.374 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-osd-2:18.0.0-1954.g1a671c19.el8.x86_64 29/32 2023-01-24T14:40:12.135 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: ceph-osd-2:18.0.0-1954.g1a671c19.el8.x86_64 29/32 2023-01-24T14:40:12.360 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-mon-2:18.0.0-1954.g1a671c19.el8.x86_64 30/32 2023-01-24T14:40:12.577 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: ceph-mon-2:18.0.0-1954.g1a671c19.el8.x86_64 30/32 2023-01-24T14:40:12.769 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-mds-2:18.0.0-1954.g1a671c19.el8.x86_64 31/32 2023-01-24T14:40:12.786 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: ceph-mds-2:18.0.0-1954.g1a671c19.el8.x86_64 31/32 2023-01-24T14:40:12.942 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-2:18.0.0-1954.g1a671c19.el8.x86_64 32/32 2023-01-24T14:40:13.378 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: ceph-2:18.0.0-1954.g1a671c19.el8.x86_64 32/32 2023-01-24T14:40:13.378 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-2:18.0.0-1954.g1a671c19.el8.x86_64 1/32 2023-01-24T14:40:13.379 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-mds-2:18.0.0-1954.g1a671c19.el8.x86_64 2/32 2023-01-24T14:40:13.379 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-mgr-2:18.0.0-1954.g1a671c19.el8.x86_64 3/32 2023-01-24T14:40:13.379 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-mon-2:18.0.0-1954.g1a671c19.el8.x86_64 4/32 2023-01-24T14:40:13.379 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-osd-2:18.0.0-1954.g1a671c19.el8.x86_64 5/32 2023-01-24T14:40:13.379 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-1954.g1a671c19.el8. 6/32 2023-01-24T14:40:13.380 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 7/32 2023-01-24T14:40:13.380 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2023-01-24T14:40:13.380 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2023-01-24T14:40:13.380 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 10/32 2023-01-24T14:40:13.380 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2023-01-24T14:40:13.381 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 12/32 2023-01-24T14:40:13.381 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-cryptography-3.2.1-5.el8.x86_64 13/32 2023-01-24T14:40:13.381 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2023-01-24T14:40:13.381 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2023-01-24T14:40:13.381 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2023-01-24T14:40:13.382 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2023-01-24T14:40:13.382 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2023-01-24T14:40:13.382 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2023-01-24T14:40:13.382 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2023-01-24T14:40:13.382 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2023-01-24T14:40:13.383 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2023-01-24T14:40:13.383 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2023-01-24T14:40:13.383 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2023-01-24T14:40:13.383 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2023-01-24T14:40:13.383 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2023-01-24T14:40:13.384 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2023-01-24T14:40:13.384 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2023-01-24T14:40:13.384 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-waitress-1.2.1-2.el8.1.noarch 29/32 2023-01-24T14:40:13.385 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2023-01-24T14:40:13.385 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2023-01-24T14:40:14.624 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2023-01-24T14:40:14.624 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-01-24T14:40:14.624 INFO:teuthology.orchestra.run.smithi179.stdout:Installed: 2023-01-24T14:40:14.625 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-2:18.0.0-1954.g1a671c19.el8.x86_64 2023-01-24T14:40:14.625 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-mds-2:18.0.0-1954.g1a671c19.el8.x86_64 2023-01-24T14:40:14.625 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-mgr-2:18.0.0-1954.g1a671c19.el8.x86_64 2023-01-24T14:40:14.625 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-mgr-modules-core-2:18.0.0-1954.g1a671c19.el8.noarch 2023-01-24T14:40:14.625 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-mon-2:18.0.0-1954.g1a671c19.el8.x86_64 2023-01-24T14:40:14.625 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-osd-2:18.0.0-1954.g1a671c19.el8.x86_64 2023-01-24T14:40:14.626 INFO:teuthology.orchestra.run.smithi179.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-01-24T14:40:14.626 INFO:teuthology.orchestra.run.smithi179.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-01-24T14:40:14.626 INFO:teuthology.orchestra.run.smithi179.stdout: python3-cffi-1.11.5-5.el8.x86_64 2023-01-24T14:40:14.626 INFO:teuthology.orchestra.run.smithi179.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-01-24T14:40:14.627 INFO:teuthology.orchestra.run.smithi179.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-01-24T14:40:14.627 INFO:teuthology.orchestra.run.smithi179.stdout: python3-cryptography-3.2.1-5.el8.x86_64 2023-01-24T14:40:14.627 INFO:teuthology.orchestra.run.smithi179.stdout: python3-jaraco-6.2-6.el8.noarch 2023-01-24T14:40:14.627 INFO:teuthology.orchestra.run.smithi179.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-01-24T14:40:14.627 INFO:teuthology.orchestra.run.smithi179.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-01-24T14:40:14.628 INFO:teuthology.orchestra.run.smithi179.stdout: python3-mako-1.0.6-13.el8.noarch 2023-01-24T14:40:14.628 INFO:teuthology.orchestra.run.smithi179.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-01-24T14:40:14.628 INFO:teuthology.orchestra.run.smithi179.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-01-24T14:40:14.628 INFO:teuthology.orchestra.run.smithi179.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-01-24T14:40:14.628 INFO:teuthology.orchestra.run.smithi179.stdout: python3-portend-2.6-1.el8.noarch 2023-01-24T14:40:14.629 INFO:teuthology.orchestra.run.smithi179.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-01-24T14:40:14.629 INFO:teuthology.orchestra.run.smithi179.stdout: python3-pycparser-2.14-14.el8.noarch 2023-01-24T14:40:14.629 INFO:teuthology.orchestra.run.smithi179.stdout: python3-pytz-2017.2-9.el8.noarch 2023-01-24T14:40:14.629 INFO:teuthology.orchestra.run.smithi179.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-01-24T14:40:14.629 INFO:teuthology.orchestra.run.smithi179.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-01-24T14:40:14.630 INFO:teuthology.orchestra.run.smithi179.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-01-24T14:40:14.630 INFO:teuthology.orchestra.run.smithi179.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-01-24T14:40:14.630 INFO:teuthology.orchestra.run.smithi179.stdout: python3-waitress-1.2.1-2.el8.1.noarch 2023-01-24T14:40:14.630 INFO:teuthology.orchestra.run.smithi179.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-01-24T14:40:14.631 INFO:teuthology.orchestra.run.smithi179.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-01-24T14:40:14.631 INFO:teuthology.orchestra.run.smithi179.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-01-24T14:40:14.631 INFO:teuthology.orchestra.run.smithi179.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-01-24T14:40:14.631 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-01-24T14:40:14.632 INFO:teuthology.orchestra.run.smithi179.stdout:Complete! 2023-01-24T14:40:14.847 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum -y install ceph-base 2023-01-24T14:40:15.568 INFO:teuthology.orchestra.run.smithi179.stdout:Last metadata expiration check: 0:11:28 ago on Tue 24 Jan 2023 02:28:47 PM UTC. 2023-01-24T14:40:16.604 INFO:teuthology.orchestra.run.smithi179.stdout:Package ceph-base-2:18.0.0-1954.g1a671c19.el8.x86_64 is already installed. 2023-01-24T14:40:16.704 INFO:teuthology.orchestra.run.smithi179.stdout:Dependencies resolved. 2023-01-24T14:40:16.705 INFO:teuthology.orchestra.run.smithi179.stdout:Nothing to do. 2023-01-24T14:40:16.705 INFO:teuthology.orchestra.run.smithi179.stdout:Complete! 2023-01-24T14:40:16.801 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum -y install cephadm 2023-01-24T14:40:17.519 INFO:teuthology.orchestra.run.smithi179.stdout:Last metadata expiration check: 0:11:30 ago on Tue 24 Jan 2023 02:28:47 PM UTC. 2023-01-24T14:40:18.662 INFO:teuthology.orchestra.run.smithi179.stdout:Dependencies resolved. 2023-01-24T14:40:18.663 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2023-01-24T14:40:18.663 INFO:teuthology.orchestra.run.smithi179.stdout: Package Arch Version Repository Size 2023-01-24T14:40:18.663 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2023-01-24T14:40:18.663 INFO:teuthology.orchestra.run.smithi179.stdout:Installing: 2023-01-24T14:40:18.664 INFO:teuthology.orchestra.run.smithi179.stdout: cephadm noarch 2:18.0.0-1954.g1a671c19.el8 ceph-noarch 82 k 2023-01-24T14:40:18.664 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-01-24T14:40:18.664 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction Summary 2023-01-24T14:40:18.664 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2023-01-24T14:40:18.665 INFO:teuthology.orchestra.run.smithi179.stdout:Install 1 Package 2023-01-24T14:40:18.665 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-01-24T14:40:18.665 INFO:teuthology.orchestra.run.smithi179.stdout:Total download size: 82 k 2023-01-24T14:40:18.665 INFO:teuthology.orchestra.run.smithi179.stdout:Installed size: 374 k 2023-01-24T14:40:18.665 INFO:teuthology.orchestra.run.smithi179.stdout:Downloading Packages: 2023-01-24T14:40:18.955 INFO:teuthology.orchestra.run.smithi179.stdout:cephadm-18.0.0-1954.g1a671c19.el8.noarch.rpm 283 kB/s | 82 kB 00:00 2023-01-24T14:40:18.955 INFO:teuthology.orchestra.run.smithi179.stdout:-------------------------------------------------------------------------------- 2023-01-24T14:40:18.956 INFO:teuthology.orchestra.run.smithi179.stdout:Total 282 kB/s | 82 kB 00:00 2023-01-24T14:40:18.956 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction check 2023-01-24T14:40:18.963 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction check succeeded. 2023-01-24T14:40:18.964 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction test 2023-01-24T14:40:18.965 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction test succeeded. 2023-01-24T14:40:18.968 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction 2023-01-24T14:40:19.014 INFO:teuthology.orchestra.run.smithi179.stdout: Preparing : 1/1 2023-01-24T14:40:27.861 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: cephadm-2:18.0.0-1954.g1a671c19.el8.noarch 1/1 2023-01-24T14:40:29.830 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : cephadm-2:18.0.0-1954.g1a671c19.el8.noarch 1/1 2023-01-24T14:40:30.081 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: cephadm-2:18.0.0-1954.g1a671c19.el8.noarch 1/1 2023-01-24T14:40:31.158 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : cephadm-2:18.0.0-1954.g1a671c19.el8.noarch 1/1 2023-01-24T14:40:31.159 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-01-24T14:40:31.159 INFO:teuthology.orchestra.run.smithi179.stdout:Installed: 2023-01-24T14:40:31.159 INFO:teuthology.orchestra.run.smithi179.stdout: cephadm-2:18.0.0-1954.g1a671c19.el8.noarch 2023-01-24T14:40:31.160 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-01-24T14:40:31.160 INFO:teuthology.orchestra.run.smithi179.stdout:Complete! 2023-01-24T14:40:32.944 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum -y install ceph-immutable-object-cache 2023-01-24T14:40:33.678 INFO:teuthology.orchestra.run.smithi179.stdout:Last metadata expiration check: 0:11:46 ago on Tue 24 Jan 2023 02:28:47 PM UTC. 2023-01-24T14:40:34.831 INFO:teuthology.orchestra.run.smithi179.stdout:Dependencies resolved. 2023-01-24T14:40:34.832 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2023-01-24T14:40:34.832 INFO:teuthology.orchestra.run.smithi179.stdout: Package Arch Version Repo Size 2023-01-24T14:40:34.833 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2023-01-24T14:40:34.833 INFO:teuthology.orchestra.run.smithi179.stdout:Installing: 2023-01-24T14:40:34.833 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-1954.g1a671c19.el8 ceph 157 k 2023-01-24T14:40:34.833 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-01-24T14:40:34.833 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction Summary 2023-01-24T14:40:34.834 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2023-01-24T14:40:34.834 INFO:teuthology.orchestra.run.smithi179.stdout:Install 1 Package 2023-01-24T14:40:34.834 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-01-24T14:40:34.834 INFO:teuthology.orchestra.run.smithi179.stdout:Total download size: 157 k 2023-01-24T14:40:34.834 INFO:teuthology.orchestra.run.smithi179.stdout:Installed size: 540 k 2023-01-24T14:40:34.835 INFO:teuthology.orchestra.run.smithi179.stdout:Downloading Packages: 2023-01-24T14:40:35.404 INFO:teuthology.orchestra.run.smithi179.stdout:ceph-immutable-object-cache-18.0.0-1954.g1a671c 275 kB/s | 157 kB 00:00 2023-01-24T14:40:35.405 INFO:teuthology.orchestra.run.smithi179.stdout:-------------------------------------------------------------------------------- 2023-01-24T14:40:35.405 INFO:teuthology.orchestra.run.smithi179.stdout:Total 274 kB/s | 157 kB 00:00 2023-01-24T14:40:35.405 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction check 2023-01-24T14:40:35.416 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction check succeeded. 2023-01-24T14:40:35.416 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction test 2023-01-24T14:40:35.466 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction test succeeded. 2023-01-24T14:40:35.469 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction 2023-01-24T14:40:35.626 INFO:teuthology.orchestra.run.smithi179.stdout: Preparing : 1/1 2023-01-24T14:40:35.832 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-1954.g1a671c19. 1/1 2023-01-24T14:40:36.279 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-1954.g1a671c19. 1/1 2023-01-24T14:40:36.777 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-1954.g1a671c19. 1/1 2023-01-24T14:40:36.778 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-01-24T14:40:36.778 INFO:teuthology.orchestra.run.smithi179.stdout:Installed: 2023-01-24T14:40:36.778 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-immutable-object-cache-2:18.0.0-1954.g1a671c19.el8.x86_64 2023-01-24T14:40:36.779 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-01-24T14:40:36.779 INFO:teuthology.orchestra.run.smithi179.stdout:Complete! 2023-01-24T14:40:36.949 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum -y install ceph-mgr 2023-01-24T14:40:37.681 INFO:teuthology.orchestra.run.smithi179.stdout:Last metadata expiration check: 0:11:50 ago on Tue 24 Jan 2023 02:28:47 PM UTC. 2023-01-24T14:40:38.714 INFO:teuthology.orchestra.run.smithi179.stdout:Package ceph-mgr-2:18.0.0-1954.g1a671c19.el8.x86_64 is already installed. 2023-01-24T14:40:38.814 INFO:teuthology.orchestra.run.smithi179.stdout:Dependencies resolved. 2023-01-24T14:40:38.815 INFO:teuthology.orchestra.run.smithi179.stdout:Nothing to do. 2023-01-24T14:40:38.815 INFO:teuthology.orchestra.run.smithi179.stdout:Complete! 2023-01-24T14:40:38.897 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum -y install ceph-mgr-dashboard 2023-01-24T14:40:39.619 INFO:teuthology.orchestra.run.smithi179.stdout:Last metadata expiration check: 0:11:52 ago on Tue 24 Jan 2023 02:28:47 PM UTC. 2023-01-24T14:40:40.749 INFO:teuthology.orchestra.run.smithi179.stdout:Dependencies resolved. 2023-01-24T14:40:40.750 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2023-01-24T14:40:40.750 INFO:teuthology.orchestra.run.smithi179.stdout: Package Arch Version Repository Size 2023-01-24T14:40:40.751 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2023-01-24T14:40:40.751 INFO:teuthology.orchestra.run.smithi179.stdout:Installing: 2023-01-24T14:40:40.751 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-mgr-dashboard noarch 2:18.0.0-1954.g1a671c19.el8 ceph-noarch 1.7 M 2023-01-24T14:40:40.751 INFO:teuthology.orchestra.run.smithi179.stdout:Installing dependencies: 2023-01-24T14:40:40.752 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-grafana-dashboards noarch 2:18.0.0-1954.g1a671c19.el8 ceph-noarch 23 k 2023-01-24T14:40:40.752 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-prometheus-alerts noarch 2:18.0.0-1954.g1a671c19.el8 ceph-noarch 15 k 2023-01-24T14:40:40.752 INFO:teuthology.orchestra.run.smithi179.stdout: python3-jwt noarch 1.6.1-2.el8 CentOS-BaseOS 43 k 2023-01-24T14:40:40.752 INFO:teuthology.orchestra.run.smithi179.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-01-24T14:40:40.752 INFO:teuthology.orchestra.run.smithi179.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-01-24T14:40:40.753 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-01-24T14:40:40.753 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction Summary 2023-01-24T14:40:40.753 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2023-01-24T14:40:40.753 INFO:teuthology.orchestra.run.smithi179.stdout:Install 6 Packages 2023-01-24T14:40:40.753 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-01-24T14:40:40.754 INFO:teuthology.orchestra.run.smithi179.stdout:Total download size: 2.0 M 2023-01-24T14:40:40.754 INFO:teuthology.orchestra.run.smithi179.stdout:Installed size: 8.2 M 2023-01-24T14:40:40.755 INFO:teuthology.orchestra.run.smithi179.stdout:Downloading Packages: 2023-01-24T14:40:41.018 INFO:teuthology.orchestra.run.smithi179.stdout:(1/6): ceph-grafana-dashboards-18.0.0-1954.g1a6 87 kB/s | 23 kB 00:00 2023-01-24T14:40:41.043 INFO:teuthology.orchestra.run.smithi179.stdout:(2/6): ceph-prometheus-alerts-18.0.0-1954.g1a67 50 kB/s | 15 kB 00:00 2023-01-24T14:40:41.102 INFO:teuthology.orchestra.run.smithi179.stdout:(3/6): python3-jwt-1.6.1-2.el8.noarch.rpm 510 kB/s | 43 kB 00:00 2023-01-24T14:40:41.244 INFO:teuthology.orchestra.run.smithi179.stdout:(4/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 168 kB/s | 34 kB 00:00 2023-01-24T14:40:41.377 INFO:teuthology.orchestra.run.smithi179.stdout:(5/6): python3-routes-2.4.1-12.el8.noarch.rpm 714 kB/s | 196 kB 00:00 2023-01-24T14:40:41.544 INFO:teuthology.orchestra.run.smithi179.stdout:(6/6): ceph-mgr-dashboard-18.0.0-1954.g1a671c19 2.2 MB/s | 1.7 MB 00:00 2023-01-24T14:40:41.546 INFO:teuthology.orchestra.run.smithi179.stdout:-------------------------------------------------------------------------------- 2023-01-24T14:40:41.547 INFO:teuthology.orchestra.run.smithi179.stdout:Total 2.5 MB/s | 2.0 MB 00:00 2023-01-24T14:40:41.557 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction check 2023-01-24T14:40:41.566 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction check succeeded. 2023-01-24T14:40:41.566 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction test 2023-01-24T14:40:41.647 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction test succeeded. 2023-01-24T14:40:41.651 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction 2023-01-24T14:40:41.929 INFO:teuthology.orchestra.run.smithi179.stdout: Preparing : 1/1 2023-01-24T14:40:42.169 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2023-01-24T14:40:42.406 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2023-01-24T14:40:42.600 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2023-01-24T14:40:42.737 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-1954.g1a671c19.el8.n 4/6 2023-01-24T14:40:43.119 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-1954.g1a671c19.el8. 5/6 2023-01-24T14:40:43.394 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-1954.g1a671c19.el8.noarc 6/6 2023-01-24T14:40:43.706 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-1954.g1a671c19.el8.noarc 6/6 2023-01-24T14:40:43.706 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-1954.g1a671c19.el8. 1/6 2023-01-24T14:40:43.706 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-1954.g1a671c19.el8.noarc 2/6 2023-01-24T14:40:43.706 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-1954.g1a671c19.el8.n 3/6 2023-01-24T14:40:43.706 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2023-01-24T14:40:43.707 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2023-01-24T14:40:44.188 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2023-01-24T14:40:44.188 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-01-24T14:40:44.189 INFO:teuthology.orchestra.run.smithi179.stdout:Installed: 2023-01-24T14:40:44.189 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-grafana-dashboards-2:18.0.0-1954.g1a671c19.el8.noarch 2023-01-24T14:40:44.190 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-mgr-dashboard-2:18.0.0-1954.g1a671c19.el8.noarch 2023-01-24T14:40:44.190 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-prometheus-alerts-2:18.0.0-1954.g1a671c19.el8.noarch 2023-01-24T14:40:44.190 INFO:teuthology.orchestra.run.smithi179.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-01-24T14:40:44.191 INFO:teuthology.orchestra.run.smithi179.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-01-24T14:40:44.191 INFO:teuthology.orchestra.run.smithi179.stdout: python3-routes-2.4.1-12.el8.noarch 2023-01-24T14:40:44.191 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-01-24T14:40:44.192 INFO:teuthology.orchestra.run.smithi179.stdout:Complete! 2023-01-24T14:40:44.330 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum -y install ceph-mgr-diskprediction-local 2023-01-24T14:40:45.050 INFO:teuthology.orchestra.run.smithi179.stdout:Last metadata expiration check: 0:11:58 ago on Tue 24 Jan 2023 02:28:47 PM UTC. 2023-01-24T14:40:46.187 INFO:teuthology.orchestra.run.smithi179.stdout:Dependencies resolved. 2023-01-24T14:40:46.188 INFO:teuthology.orchestra.run.smithi179.stdout:===================================================================================================== 2023-01-24T14:40:46.188 INFO:teuthology.orchestra.run.smithi179.stdout: Package Arch Version Repository Size 2023-01-24T14:40:46.188 INFO:teuthology.orchestra.run.smithi179.stdout:===================================================================================================== 2023-01-24T14:40:46.188 INFO:teuthology.orchestra.run.smithi179.stdout:Installing: 2023-01-24T14:40:46.189 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-1954.g1a671c19.el8 ceph-noarch 7.4 M 2023-01-24T14:40:46.189 INFO:teuthology.orchestra.run.smithi179.stdout:Installing dependencies: 2023-01-24T14:40:46.189 INFO:teuthology.orchestra.run.smithi179.stdout: libgfortran x86_64 8.5.0-18.el8 CentOS-BaseOS 645 k 2023-01-24T14:40:46.189 INFO:teuthology.orchestra.run.smithi179.stdout: libquadmath x86_64 8.5.0-18.el8 CentOS-BaseOS 172 k 2023-01-24T14:40:46.190 INFO:teuthology.orchestra.run.smithi179.stdout: openblas x86_64 0.3.15-4.el8 CentOS-AppStream 4.7 M 2023-01-24T14:40:46.190 INFO:teuthology.orchestra.run.smithi179.stdout: openblas-threads x86_64 0.3.15-4.el8 CentOS-AppStream 4.9 M 2023-01-24T14:40:46.190 INFO:teuthology.orchestra.run.smithi179.stdout: python3-numpy x86_64 1:1.14.3-10.el8 CentOS-AppStream 3.7 M 2023-01-24T14:40:46.190 INFO:teuthology.orchestra.run.smithi179.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 CentOS-AppStream 225 k 2023-01-24T14:40:46.190 INFO:teuthology.orchestra.run.smithi179.stdout: python3-scipy x86_64 1.0.0-21.module_el8.5.0+771+e5d9a225 CentOS-AppStream 14 M 2023-01-24T14:40:46.191 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-01-24T14:40:46.191 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction Summary 2023-01-24T14:40:46.191 INFO:teuthology.orchestra.run.smithi179.stdout:===================================================================================================== 2023-01-24T14:40:46.191 INFO:teuthology.orchestra.run.smithi179.stdout:Install 8 Packages 2023-01-24T14:40:46.191 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-01-24T14:40:46.192 INFO:teuthology.orchestra.run.smithi179.stdout:Total download size: 35 M 2023-01-24T14:40:46.192 INFO:teuthology.orchestra.run.smithi179.stdout:Installed size: 210 M 2023-01-24T14:40:46.193 INFO:teuthology.orchestra.run.smithi179.stdout:Downloading Packages: 2023-01-24T14:40:46.443 INFO:teuthology.orchestra.run.smithi179.stdout:(1/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 19 MB/s | 4.9 MB 00:00 2023-01-24T14:40:46.518 INFO:teuthology.orchestra.run.smithi179.stdout:(2/8): openblas-0.3.15-4.el8.x86_64.rpm 14 MB/s | 4.7 MB 00:00 2023-01-24T14:40:46.551 INFO:teuthology.orchestra.run.smithi179.stdout:(3/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 6.6 MB/s | 225 kB 00:00 2023-01-24T14:40:46.652 INFO:teuthology.orchestra.run.smithi179.stdout:(4/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 18 MB/s | 3.7 MB 00:00 2023-01-24T14:40:46.694 INFO:teuthology.orchestra.run.smithi179.stdout:(5/8): libgfortran-8.5.0-18.el8.x86_64.rpm 15 MB/s | 645 kB 00:00 2023-01-24T14:40:46.727 INFO:teuthology.orchestra.run.smithi179.stdout:(6/8): libquadmath-8.5.0-18.el8.x86_64.rpm 5.0 MB/s | 172 kB 00:00 2023-01-24T14:40:46.969 INFO:teuthology.orchestra.run.smithi179.stdout:(7/8): python3-scipy-1.0.0-21.module_el8.5.0+77 33 MB/s | 14 MB 00:00 2023-01-24T14:40:47.869 INFO:teuthology.orchestra.run.smithi179.stdout:(8/8): ceph-mgr-diskprediction-local-18.0.0-195 4.4 MB/s | 7.4 MB 00:01 2023-01-24T14:40:47.872 INFO:teuthology.orchestra.run.smithi179.stdout:-------------------------------------------------------------------------------- 2023-01-24T14:40:47.872 INFO:teuthology.orchestra.run.smithi179.stdout:Total 21 MB/s | 35 MB 00:01 2023-01-24T14:40:48.103 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction check 2023-01-24T14:40:48.121 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction check succeeded. 2023-01-24T14:40:48.121 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction test 2023-01-24T14:40:48.532 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction test succeeded. 2023-01-24T14:40:48.536 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction 2023-01-24T14:40:49.099 INFO:teuthology.orchestra.run.smithi179.stdout: Preparing : 1/1 2023-01-24T14:40:49.229 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : libquadmath-8.5.0-18.el8.x86_64 1/8 2023-01-24T14:40:49.364 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: libquadmath-8.5.0-18.el8.x86_64 1/8 2023-01-24T14:40:49.496 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : libgfortran-8.5.0-18.el8.x86_64 2/8 2023-01-24T14:40:50.093 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: libgfortran-8.5.0-18.el8.x86_64 2/8 2023-01-24T14:40:50.246 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2023-01-24T14:40:50.862 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2023-01-24T14:40:51.113 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2023-01-24T14:40:51.789 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2023-01-24T14:40:52.150 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2023-01-24T14:40:52.396 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-01-24T14:40:54.190 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-01-24T14:40:55.460 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 7/8 2023-01-24T14:40:55.678 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-1954.g1a671c1 8/8 2023-01-24T14:40:56.006 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-1954.g1a671c1 8/8 2023-01-24T14:40:56.007 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-1954.g1a671c1 1/8 2023-01-24T14:40:56.007 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2023-01-24T14:40:56.007 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2023-01-24T14:40:56.007 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2023-01-24T14:40:56.008 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2023-01-24T14:40:56.008 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 6/8 2023-01-24T14:40:56.008 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : libgfortran-8.5.0-18.el8.x86_64 7/8 2023-01-24T14:40:56.508 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : libquadmath-8.5.0-18.el8.x86_64 8/8 2023-01-24T14:40:56.509 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-01-24T14:40:56.509 INFO:teuthology.orchestra.run.smithi179.stdout:Installed: 2023-01-24T14:40:56.509 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-mgr-diskprediction-local-2:18.0.0-1954.g1a671c19.el8.noarch 2023-01-24T14:40:56.510 INFO:teuthology.orchestra.run.smithi179.stdout: libgfortran-8.5.0-18.el8.x86_64 2023-01-24T14:40:56.510 INFO:teuthology.orchestra.run.smithi179.stdout: libquadmath-8.5.0-18.el8.x86_64 2023-01-24T14:40:56.510 INFO:teuthology.orchestra.run.smithi179.stdout: openblas-0.3.15-4.el8.x86_64 2023-01-24T14:40:56.511 INFO:teuthology.orchestra.run.smithi179.stdout: openblas-threads-0.3.15-4.el8.x86_64 2023-01-24T14:40:56.511 INFO:teuthology.orchestra.run.smithi179.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-01-24T14:40:56.511 INFO:teuthology.orchestra.run.smithi179.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-01-24T14:40:56.511 INFO:teuthology.orchestra.run.smithi179.stdout: python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x86_64 2023-01-24T14:40:56.512 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-01-24T14:40:56.512 INFO:teuthology.orchestra.run.smithi179.stdout:Complete! 2023-01-24T14:40:56.629 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum -y install ceph-mgr-rook 2023-01-24T14:40:57.346 INFO:teuthology.orchestra.run.smithi179.stdout:Last metadata expiration check: 0:12:10 ago on Tue 24 Jan 2023 02:28:47 PM UTC. 2023-01-24T14:40:58.492 INFO:teuthology.orchestra.run.smithi179.stdout:Dependencies resolved. 2023-01-24T14:40:58.494 INFO:teuthology.orchestra.run.smithi179.stdout:======================================================================================== 2023-01-24T14:40:58.494 INFO:teuthology.orchestra.run.smithi179.stdout: Package Arch Version Repository Size 2023-01-24T14:40:58.494 INFO:teuthology.orchestra.run.smithi179.stdout:======================================================================================== 2023-01-24T14:40:58.495 INFO:teuthology.orchestra.run.smithi179.stdout:Installing: 2023-01-24T14:40:58.495 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-mgr-rook noarch 2:18.0.0-1954.g1a671c19.el8 ceph-noarch 47 k 2023-01-24T14:40:58.495 INFO:teuthology.orchestra.run.smithi179.stdout:Installing dependencies: 2023-01-24T14:40:58.495 INFO:teuthology.orchestra.run.smithi179.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-01-24T14:40:58.496 INFO:teuthology.orchestra.run.smithi179.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-01-24T14:40:58.496 INFO:teuthology.orchestra.run.smithi179.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-01-24T14:40:58.496 INFO:teuthology.orchestra.run.smithi179.stdout: python3-jsonpatch noarch 1.21-2.el8 CentOS-AppStream 27 k 2023-01-24T14:40:58.496 INFO:teuthology.orchestra.run.smithi179.stdout: python3-jsonpointer noarch 1.10-11.el8 CentOS-AppStream 20 k 2023-01-24T14:40:58.496 INFO:teuthology.orchestra.run.smithi179.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-01-24T14:40:58.497 INFO:teuthology.orchestra.run.smithi179.stdout: python3-oauthlib noarch 2.1.0-1.el8 CentOS-BaseOS 155 k 2023-01-24T14:40:58.497 INFO:teuthology.orchestra.run.smithi179.stdout: python3-pyasn1 noarch 0.3.7-6.el8 CentOS-AppStream 126 k 2023-01-24T14:40:58.497 INFO:teuthology.orchestra.run.smithi179.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 CentOS-AppStream 110 k 2023-01-24T14:40:58.497 INFO:teuthology.orchestra.run.smithi179.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 CentOS-BaseOS 43 k 2023-01-24T14:40:58.497 INFO:teuthology.orchestra.run.smithi179.stdout: python3-rsa noarch 4.8-1.el8 epel 61 k 2023-01-24T14:40:58.498 INFO:teuthology.orchestra.run.smithi179.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-01-24T14:40:58.498 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-01-24T14:40:58.498 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction Summary 2023-01-24T14:40:58.498 INFO:teuthology.orchestra.run.smithi179.stdout:======================================================================================== 2023-01-24T14:40:58.498 INFO:teuthology.orchestra.run.smithi179.stdout:Install 13 Packages 2023-01-24T14:40:58.499 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-01-24T14:40:58.499 INFO:teuthology.orchestra.run.smithi179.stdout:Total download size: 1.9 M 2023-01-24T14:40:58.500 INFO:teuthology.orchestra.run.smithi179.stdout:Installed size: 25 M 2023-01-24T14:40:58.500 INFO:teuthology.orchestra.run.smithi179.stdout:Downloading Packages: 2023-01-24T14:40:58.592 INFO:teuthology.orchestra.run.smithi179.stdout:(1/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 281 kB/s | 27 kB 00:00 2023-01-24T14:40:58.626 INFO:teuthology.orchestra.run.smithi179.stdout:(2/13): python3-jsonpointer-1.10-11.el8.noarch. 156 kB/s | 20 kB 00:00 2023-01-24T14:40:58.651 INFO:teuthology.orchestra.run.smithi179.stdout:(3/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 2.3 MB/s | 126 kB 00:00 2023-01-24T14:40:58.676 INFO:teuthology.orchestra.run.smithi179.stdout:(4/13): python3-pyasn1-modules-0.3.7-6.el8.noar 2.1 MB/s | 110 kB 00:00 2023-01-24T14:40:58.702 INFO:teuthology.orchestra.run.smithi179.stdout:(5/13): python3-requests-oauthlib-1.0.0-1.el8.n 1.7 MB/s | 43 kB 00:00 2023-01-24T14:40:58.727 INFO:teuthology.orchestra.run.smithi179.stdout:(6/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 2.0 MB/s | 155 kB 00:00 2023-01-24T14:40:58.836 INFO:teuthology.orchestra.run.smithi179.stdout:(7/13): ceph-mgr-rook-18.0.0-1954.g1a671c19.el8 140 kB/s | 47 kB 00:00 2023-01-24T14:40:58.861 INFO:teuthology.orchestra.run.smithi179.stdout:(8/13): python3-certifi-2018.10.15-7.el8.noarch 122 kB/s | 16 kB 00:00 2023-01-24T14:40:58.886 INFO:teuthology.orchestra.run.smithi179.stdout:(9/13): python3-cachetools-3.1.1-4.el8.noarch.r 179 kB/s | 33 kB 00:00 2023-01-24T14:40:58.928 INFO:teuthology.orchestra.run.smithi179.stdout:(10/13): python3-google-auth-1.1.1-10.el8.noarc 880 kB/s | 81 kB 00:00 2023-01-24T14:40:59.020 INFO:teuthology.orchestra.run.smithi179.stdout:(11/13): python3-websocket-client-0.56.0-5.el8. 667 kB/s | 61 kB 00:00 2023-01-24T14:40:59.045 INFO:teuthology.orchestra.run.smithi179.stdout:(12/13): python3-rsa-4.8-1.el8.noarch.rpm 387 kB/s | 61 kB 00:00 2023-01-24T14:40:59.387 INFO:teuthology.orchestra.run.smithi179.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 2.2 MB/s | 1.2 MB 00:00 2023-01-24T14:40:59.389 INFO:teuthology.orchestra.run.smithi179.stdout:-------------------------------------------------------------------------------- 2023-01-24T14:40:59.389 INFO:teuthology.orchestra.run.smithi179.stdout:Total 2.2 MB/s | 1.9 MB 00:00 2023-01-24T14:40:59.440 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction check 2023-01-24T14:40:59.455 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction check succeeded. 2023-01-24T14:40:59.456 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction test 2023-01-24T14:40:59.553 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction test succeeded. 2023-01-24T14:40:59.556 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction 2023-01-24T14:41:00.070 INFO:teuthology.orchestra.run.smithi179.stdout: Preparing : 1/1 2023-01-24T14:41:00.314 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2023-01-24T14:41:00.504 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2023-01-24T14:41:00.789 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-rsa-4.8-1.el8.noarch 3/13 2023-01-24T14:41:01.010 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2023-01-24T14:41:01.427 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2023-01-24T14:41:01.676 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2023-01-24T14:41:02.053 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2023-01-24T14:41:02.275 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2023-01-24T14:41:03.313 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2023-01-24T14:41:03.601 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2023-01-24T14:41:03.843 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2023-01-24T14:41:04.052 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2023-01-24T14:41:04.228 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-mgr-rook-2:18.0.0-1954.g1a671c19.el8.noarch 13/13 2023-01-24T14:41:04.522 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-1954.g1a671c19.el8.noarch 13/13 2023-01-24T14:41:04.522 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-mgr-rook-2:18.0.0-1954.g1a671c19.el8.noarch 1/13 2023-01-24T14:41:04.523 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2023-01-24T14:41:04.523 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2023-01-24T14:41:04.523 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2023-01-24T14:41:04.523 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2023-01-24T14:41:04.523 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2023-01-24T14:41:04.524 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2023-01-24T14:41:04.524 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2023-01-24T14:41:04.524 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2023-01-24T14:41:04.524 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2023-01-24T14:41:04.525 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2023-01-24T14:41:04.525 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-rsa-4.8-1.el8.noarch 12/13 2023-01-24T14:41:05.078 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2023-01-24T14:41:05.078 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-01-24T14:41:05.078 INFO:teuthology.orchestra.run.smithi179.stdout:Installed: 2023-01-24T14:41:05.078 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-mgr-rook-2:18.0.0-1954.g1a671c19.el8.noarch 2023-01-24T14:41:05.079 INFO:teuthology.orchestra.run.smithi179.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-01-24T14:41:05.079 INFO:teuthology.orchestra.run.smithi179.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-01-24T14:41:05.079 INFO:teuthology.orchestra.run.smithi179.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-01-24T14:41:05.079 INFO:teuthology.orchestra.run.smithi179.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-01-24T14:41:05.080 INFO:teuthology.orchestra.run.smithi179.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-01-24T14:41:05.080 INFO:teuthology.orchestra.run.smithi179.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-01-24T14:41:05.080 INFO:teuthology.orchestra.run.smithi179.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-01-24T14:41:05.080 INFO:teuthology.orchestra.run.smithi179.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-01-24T14:41:05.080 INFO:teuthology.orchestra.run.smithi179.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-01-24T14:41:05.081 INFO:teuthology.orchestra.run.smithi179.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-01-24T14:41:05.081 INFO:teuthology.orchestra.run.smithi179.stdout: python3-rsa-4.8-1.el8.noarch 2023-01-24T14:41:05.081 INFO:teuthology.orchestra.run.smithi179.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-01-24T14:41:05.081 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-01-24T14:41:05.081 INFO:teuthology.orchestra.run.smithi179.stdout:Complete! 2023-01-24T14:41:05.210 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum -y install ceph-mgr-cephadm 2023-01-24T14:41:05.938 INFO:teuthology.orchestra.run.smithi179.stdout:Last metadata expiration check: 0:12:18 ago on Tue 24 Jan 2023 02:28:47 PM UTC. 2023-01-24T14:41:07.043 INFO:teuthology.orchestra.run.smithi179.stdout:Dependencies resolved. 2023-01-24T14:41:07.043 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2023-01-24T14:41:07.044 INFO:teuthology.orchestra.run.smithi179.stdout: Package Arch Version Repository Size 2023-01-24T14:41:07.044 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2023-01-24T14:41:07.044 INFO:teuthology.orchestra.run.smithi179.stdout:Installing: 2023-01-24T14:41:07.044 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-mgr-cephadm noarch 2:18.0.0-1954.g1a671c19.el8 ceph-noarch 120 k 2023-01-24T14:41:07.045 INFO:teuthology.orchestra.run.smithi179.stdout:Installing dependencies: 2023-01-24T14:41:07.045 INFO:teuthology.orchestra.run.smithi179.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-01-24T14:41:07.045 INFO:teuthology.orchestra.run.smithi179.stdout: python3-babel noarch 2.5.1-7.el8 CentOS-AppStream 4.8 M 2023-01-24T14:41:07.045 INFO:teuthology.orchestra.run.smithi179.stdout: python3-jinja2 noarch 2.10.1-3.el8 CentOS-AppStream 538 k 2023-01-24T14:41:07.045 INFO:teuthology.orchestra.run.smithi179.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-01-24T14:41:07.046 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-01-24T14:41:07.046 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction Summary 2023-01-24T14:41:07.046 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2023-01-24T14:41:07.046 INFO:teuthology.orchestra.run.smithi179.stdout:Install 5 Packages 2023-01-24T14:41:07.046 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-01-24T14:41:07.047 INFO:teuthology.orchestra.run.smithi179.stdout:Total download size: 6.0 M 2023-01-24T14:41:07.047 INFO:teuthology.orchestra.run.smithi179.stdout:Installed size: 26 M 2023-01-24T14:41:07.048 INFO:teuthology.orchestra.run.smithi179.stdout:Downloading Packages: 2023-01-24T14:41:07.121 INFO:teuthology.orchestra.run.smithi179.stdout:(1/5): python3-jinja2-2.10.1-3.el8.noarch.rpm 7.1 MB/s | 538 kB 00:00 2023-01-24T14:41:07.221 INFO:teuthology.orchestra.run.smithi179.stdout:(2/5): python3-babel-2.5.1-7.el8.noarch.rpm 27 MB/s | 4.8 MB 00:00 2023-01-24T14:41:07.272 INFO:teuthology.orchestra.run.smithi179.stdout:(3/5): ceph-mgr-cephadm-18.0.0-1954.g1a671c19.e 533 kB/s | 120 kB 00:00 2023-01-24T14:41:07.371 INFO:teuthology.orchestra.run.smithi179.stdout:(4/5): python3-natsort-7.1.1-2.el8.noarch.rpm 401 kB/s | 60 kB 00:00 2023-01-24T14:41:07.463 INFO:teuthology.orchestra.run.smithi179.stdout:(5/5): python3-asyncssh-2.7.0-2.el8.noarch.rpm 1.4 MB/s | 476 kB 00:00 2023-01-24T14:41:07.465 INFO:teuthology.orchestra.run.smithi179.stdout:-------------------------------------------------------------------------------- 2023-01-24T14:41:07.465 INFO:teuthology.orchestra.run.smithi179.stdout:Total 14 MB/s | 6.0 MB 00:00 2023-01-24T14:41:07.516 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction check 2023-01-24T14:41:07.527 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction check succeeded. 2023-01-24T14:41:07.527 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction test 2023-01-24T14:41:07.649 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction test succeeded. 2023-01-24T14:41:07.653 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction 2023-01-24T14:41:07.933 INFO:teuthology.orchestra.run.smithi179.stdout: Preparing : 1/1 2023-01-24T14:41:08.248 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 1/5 2023-01-24T14:41:09.205 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 2/5 2023-01-24T14:41:09.606 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 3/5 2023-01-24T14:41:09.863 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 4/5 2023-01-24T14:41:10.093 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-1954.g1a671c19.el8.noarch 5/5 2023-01-24T14:41:10.395 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-1954.g1a671c19.el8.noarch 5/5 2023-01-24T14:41:10.396 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-1954.g1a671c19.el8.noarch 1/5 2023-01-24T14:41:10.396 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/5 2023-01-24T14:41:10.396 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 3/5 2023-01-24T14:41:10.397 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 4/5 2023-01-24T14:41:10.878 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 5/5 2023-01-24T14:41:10.879 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-01-24T14:41:10.879 INFO:teuthology.orchestra.run.smithi179.stdout:Installed: 2023-01-24T14:41:10.879 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-mgr-cephadm-2:18.0.0-1954.g1a671c19.el8.noarch 2023-01-24T14:41:10.879 INFO:teuthology.orchestra.run.smithi179.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-01-24T14:41:10.880 INFO:teuthology.orchestra.run.smithi179.stdout: python3-babel-2.5.1-7.el8.noarch 2023-01-24T14:41:10.880 INFO:teuthology.orchestra.run.smithi179.stdout: python3-jinja2-2.10.1-3.el8.noarch 2023-01-24T14:41:10.880 INFO:teuthology.orchestra.run.smithi179.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-01-24T14:41:10.880 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-01-24T14:41:10.880 INFO:teuthology.orchestra.run.smithi179.stdout:Complete! 2023-01-24T14:41:11.014 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum -y install ceph-fuse 2023-01-24T14:41:11.737 INFO:teuthology.orchestra.run.smithi179.stdout:Last metadata expiration check: 0:12:24 ago on Tue 24 Jan 2023 02:28:47 PM UTC. 2023-01-24T14:41:12.885 INFO:teuthology.orchestra.run.smithi179.stdout:Dependencies resolved. 2023-01-24T14:41:12.886 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2023-01-24T14:41:12.887 INFO:teuthology.orchestra.run.smithi179.stdout: Package Arch Version Repository Size 2023-01-24T14:41:12.887 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2023-01-24T14:41:12.887 INFO:teuthology.orchestra.run.smithi179.stdout:Installing: 2023-01-24T14:41:12.887 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-fuse x86_64 2:18.0.0-1954.g1a671c19.el8 ceph 814 k 2023-01-24T14:41:12.887 INFO:teuthology.orchestra.run.smithi179.stdout:Installing dependencies: 2023-01-24T14:41:12.888 INFO:teuthology.orchestra.run.smithi179.stdout: fuse x86_64 2.9.7-16.el8 CentOS-BaseOS 83 k 2023-01-24T14:41:12.888 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-01-24T14:41:12.888 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction Summary 2023-01-24T14:41:12.888 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2023-01-24T14:41:12.889 INFO:teuthology.orchestra.run.smithi179.stdout:Install 2 Packages 2023-01-24T14:41:12.889 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-01-24T14:41:12.889 INFO:teuthology.orchestra.run.smithi179.stdout:Total download size: 897 k 2023-01-24T14:41:12.889 INFO:teuthology.orchestra.run.smithi179.stdout:Installed size: 3.3 M 2023-01-24T14:41:12.889 INFO:teuthology.orchestra.run.smithi179.stdout:Downloading Packages: 2023-01-24T14:41:12.972 INFO:teuthology.orchestra.run.smithi179.stdout:(1/2): fuse-2.9.7-16.el8.x86_64.rpm 986 kB/s | 83 kB 00:00 2023-01-24T14:41:15.146 INFO:teuthology.orchestra.run.smithi179.stdout:(2/2): ceph-fuse-18.0.0-1954.g1a671c19.el8.x86_ 361 kB/s | 814 kB 00:02 2023-01-24T14:41:15.147 INFO:teuthology.orchestra.run.smithi179.stdout:-------------------------------------------------------------------------------- 2023-01-24T14:41:15.148 INFO:teuthology.orchestra.run.smithi179.stdout:Total 397 kB/s | 897 kB 00:02 2023-01-24T14:41:15.159 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction check 2023-01-24T14:41:15.180 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction check succeeded. 2023-01-24T14:41:15.181 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction test 2023-01-24T14:41:15.276 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction test succeeded. 2023-01-24T14:41:15.279 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction 2023-01-24T14:41:15.461 INFO:teuthology.orchestra.run.smithi179.stdout: Preparing : 1/1 2023-01-24T14:41:15.742 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : fuse-2.9.7-16.el8.x86_64 1/2 2023-01-24T14:41:15.966 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-fuse-2:18.0.0-1954.g1a671c19.el8.x86_64 2/2 2023-01-24T14:41:16.403 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: ceph-fuse-2:18.0.0-1954.g1a671c19.el8.x86_64 2/2 2023-01-24T14:41:16.404 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-fuse-2:18.0.0-1954.g1a671c19.el8.x86_64 1/2 2023-01-24T14:41:16.987 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : fuse-2.9.7-16.el8.x86_64 2/2 2023-01-24T14:41:16.987 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-01-24T14:41:16.987 INFO:teuthology.orchestra.run.smithi179.stdout:Installed: 2023-01-24T14:41:16.987 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-fuse-2:18.0.0-1954.g1a671c19.el8.x86_64 fuse-2.9.7-16.el8.x86_64 2023-01-24T14:41:16.988 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-01-24T14:41:16.988 INFO:teuthology.orchestra.run.smithi179.stdout:Complete! 2023-01-24T14:41:17.236 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum -y install ceph-volume 2023-01-24T14:41:17.975 INFO:teuthology.orchestra.run.smithi179.stdout:Last metadata expiration check: 0:12:30 ago on Tue 24 Jan 2023 02:28:47 PM UTC. 2023-01-24T14:41:19.123 INFO:teuthology.orchestra.run.smithi179.stdout:Dependencies resolved. 2023-01-24T14:41:19.124 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2023-01-24T14:41:19.125 INFO:teuthology.orchestra.run.smithi179.stdout: Package Arch Version Repository Size 2023-01-24T14:41:19.125 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2023-01-24T14:41:19.125 INFO:teuthology.orchestra.run.smithi179.stdout:Installing: 2023-01-24T14:41:19.125 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-volume noarch 2:18.0.0-1954.g1a671c19.el8 ceph-noarch 263 k 2023-01-24T14:41:19.125 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-01-24T14:41:19.126 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction Summary 2023-01-24T14:41:19.126 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2023-01-24T14:41:19.126 INFO:teuthology.orchestra.run.smithi179.stdout:Install 1 Package 2023-01-24T14:41:19.126 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-01-24T14:41:19.127 INFO:teuthology.orchestra.run.smithi179.stdout:Total download size: 263 k 2023-01-24T14:41:19.127 INFO:teuthology.orchestra.run.smithi179.stdout:Installed size: 1.3 M 2023-01-24T14:41:19.127 INFO:teuthology.orchestra.run.smithi179.stdout:Downloading Packages: 2023-01-24T14:41:19.501 INFO:teuthology.orchestra.run.smithi179.stdout:ceph-volume-18.0.0-1954.g1a671c19.el8.noarch.rp 699 kB/s | 263 kB 00:00 2023-01-24T14:41:19.502 INFO:teuthology.orchestra.run.smithi179.stdout:-------------------------------------------------------------------------------- 2023-01-24T14:41:19.502 INFO:teuthology.orchestra.run.smithi179.stdout:Total 697 kB/s | 263 kB 00:00 2023-01-24T14:41:19.503 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction check 2023-01-24T14:41:19.512 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction check succeeded. 2023-01-24T14:41:19.513 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction test 2023-01-24T14:41:19.603 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction test succeeded. 2023-01-24T14:41:19.606 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction 2023-01-24T14:41:19.787 INFO:teuthology.orchestra.run.smithi179.stdout: Preparing : 1/1 2023-01-24T14:41:20.050 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-volume-2:18.0.0-1954.g1a671c19.el8.noarch 1/1 2023-01-24T14:41:20.512 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: ceph-volume-2:18.0.0-1954.g1a671c19.el8.noarch 1/1 2023-01-24T14:41:21.032 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-volume-2:18.0.0-1954.g1a671c19.el8.noarch 1/1 2023-01-24T14:41:21.054 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-01-24T14:41:21.055 INFO:teuthology.orchestra.run.smithi179.stdout:Installed: 2023-01-24T14:41:21.055 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-volume-2:18.0.0-1954.g1a671c19.el8.noarch 2023-01-24T14:41:21.055 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-01-24T14:41:21.056 INFO:teuthology.orchestra.run.smithi179.stdout:Complete! 2023-01-24T14:41:21.264 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum -y install librados-devel 2023-01-24T14:41:22.009 INFO:teuthology.orchestra.run.smithi179.stdout:Last metadata expiration check: 0:12:35 ago on Tue 24 Jan 2023 02:28:47 PM UTC. 2023-01-24T14:41:23.153 INFO:teuthology.orchestra.run.smithi179.stdout:Dependencies resolved. 2023-01-24T14:41:23.153 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2023-01-24T14:41:23.153 INFO:teuthology.orchestra.run.smithi179.stdout: Package Arch Version Repo Size 2023-01-24T14:41:23.154 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2023-01-24T14:41:23.154 INFO:teuthology.orchestra.run.smithi179.stdout:Installing: 2023-01-24T14:41:23.154 INFO:teuthology.orchestra.run.smithi179.stdout: librados-devel x86_64 2:18.0.0-1954.g1a671c19.el8 ceph 139 k 2023-01-24T14:41:23.154 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-01-24T14:41:23.154 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction Summary 2023-01-24T14:41:23.155 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2023-01-24T14:41:23.155 INFO:teuthology.orchestra.run.smithi179.stdout:Install 1 Package 2023-01-24T14:41:23.155 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-01-24T14:41:23.155 INFO:teuthology.orchestra.run.smithi179.stdout:Total download size: 139 k 2023-01-24T14:41:23.155 INFO:teuthology.orchestra.run.smithi179.stdout:Installed size: 512 k 2023-01-24T14:41:23.156 INFO:teuthology.orchestra.run.smithi179.stdout:Downloading Packages: 2023-01-24T14:41:23.798 INFO:teuthology.orchestra.run.smithi179.stdout:librados-devel-18.0.0-1954.g1a671c19.el8.x86_64 215 kB/s | 139 kB 00:00 2023-01-24T14:41:23.799 INFO:teuthology.orchestra.run.smithi179.stdout:-------------------------------------------------------------------------------- 2023-01-24T14:41:23.799 INFO:teuthology.orchestra.run.smithi179.stdout:Total 214 kB/s | 139 kB 00:00 2023-01-24T14:41:23.800 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction check 2023-01-24T14:41:23.805 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction check succeeded. 2023-01-24T14:41:23.805 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction test 2023-01-24T14:41:23.857 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction test succeeded. 2023-01-24T14:41:23.861 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction 2023-01-24T14:41:23.974 INFO:teuthology.orchestra.run.smithi179.stdout: Preparing : 1/1 2023-01-24T14:41:24.164 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : librados-devel-2:18.0.0-1954.g1a671c19.el8.x86_64 1/1 2023-01-24T14:41:24.490 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: librados-devel-2:18.0.0-1954.g1a671c19.el8.x86_64 1/1 2023-01-24T14:41:24.995 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : librados-devel-2:18.0.0-1954.g1a671c19.el8.x86_64 1/1 2023-01-24T14:41:24.996 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-01-24T14:41:24.996 INFO:teuthology.orchestra.run.smithi179.stdout:Installed: 2023-01-24T14:41:24.996 INFO:teuthology.orchestra.run.smithi179.stdout: librados-devel-2:18.0.0-1954.g1a671c19.el8.x86_64 2023-01-24T14:41:24.997 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-01-24T14:41:24.997 INFO:teuthology.orchestra.run.smithi179.stdout:Complete! 2023-01-24T14:41:25.208 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum -y install libcephfs2 2023-01-24T14:41:25.944 INFO:teuthology.orchestra.run.smithi179.stdout:Last metadata expiration check: 0:12:38 ago on Tue 24 Jan 2023 02:28:47 PM UTC. 2023-01-24T14:41:26.988 INFO:teuthology.orchestra.run.smithi179.stdout:Package libcephfs2-2:18.0.0-1954.g1a671c19.el8.x86_64 is already installed. 2023-01-24T14:41:27.089 INFO:teuthology.orchestra.run.smithi179.stdout:Dependencies resolved. 2023-01-24T14:41:27.090 INFO:teuthology.orchestra.run.smithi179.stdout:Nothing to do. 2023-01-24T14:41:27.091 INFO:teuthology.orchestra.run.smithi179.stdout:Complete! 2023-01-24T14:41:27.188 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum -y install libcephfs-devel 2023-01-24T14:41:27.935 INFO:teuthology.orchestra.run.smithi179.stdout:Last metadata expiration check: 0:12:40 ago on Tue 24 Jan 2023 02:28:47 PM UTC. 2023-01-24T14:41:29.089 INFO:teuthology.orchestra.run.smithi179.stdout:Dependencies resolved. 2023-01-24T14:41:29.089 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2023-01-24T14:41:29.089 INFO:teuthology.orchestra.run.smithi179.stdout: Package Arch Version Repo Size 2023-01-24T14:41:29.090 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2023-01-24T14:41:29.090 INFO:teuthology.orchestra.run.smithi179.stdout:Installing: 2023-01-24T14:41:29.090 INFO:teuthology.orchestra.run.smithi179.stdout: libcephfs-devel x86_64 2:18.0.0-1954.g1a671c19.el8 ceph 30 k 2023-01-24T14:41:29.090 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-01-24T14:41:29.090 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction Summary 2023-01-24T14:41:29.091 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2023-01-24T14:41:29.091 INFO:teuthology.orchestra.run.smithi179.stdout:Install 1 Package 2023-01-24T14:41:29.091 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-01-24T14:41:29.091 INFO:teuthology.orchestra.run.smithi179.stdout:Total download size: 30 k 2023-01-24T14:41:29.091 INFO:teuthology.orchestra.run.smithi179.stdout:Installed size: 137 k 2023-01-24T14:41:29.092 INFO:teuthology.orchestra.run.smithi179.stdout:Downloading Packages: 2023-01-24T14:41:29.471 INFO:teuthology.orchestra.run.smithi179.stdout:libcephfs-devel-18.0.0-1954.g1a671c19.el8.x86_6 80 kB/s | 30 kB 00:00 2023-01-24T14:41:29.471 INFO:teuthology.orchestra.run.smithi179.stdout:-------------------------------------------------------------------------------- 2023-01-24T14:41:29.472 INFO:teuthology.orchestra.run.smithi179.stdout:Total 79 kB/s | 30 kB 00:00 2023-01-24T14:41:29.472 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction check 2023-01-24T14:41:29.475 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction check succeeded. 2023-01-24T14:41:29.475 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction test 2023-01-24T14:41:29.499 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction test succeeded. 2023-01-24T14:41:29.502 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction 2023-01-24T14:41:29.564 INFO:teuthology.orchestra.run.smithi179.stdout: Preparing : 1/1 2023-01-24T14:41:29.763 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : libcephfs-devel-2:18.0.0-1954.g1a671c19.el8.x86_64 1/1 2023-01-24T14:41:30.036 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: libcephfs-devel-2:18.0.0-1954.g1a671c19.el8.x86_64 1/1 2023-01-24T14:41:30.501 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : libcephfs-devel-2:18.0.0-1954.g1a671c19.el8.x86_64 1/1 2023-01-24T14:41:30.502 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-01-24T14:41:30.502 INFO:teuthology.orchestra.run.smithi179.stdout:Installed: 2023-01-24T14:41:30.503 INFO:teuthology.orchestra.run.smithi179.stdout: libcephfs-devel-2:18.0.0-1954.g1a671c19.el8.x86_64 2023-01-24T14:41:30.503 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-01-24T14:41:30.503 INFO:teuthology.orchestra.run.smithi179.stdout:Complete! 2023-01-24T14:41:30.619 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum -y install librados2 2023-01-24T14:41:31.339 INFO:teuthology.orchestra.run.smithi179.stdout:Last metadata expiration check: 0:12:44 ago on Tue 24 Jan 2023 02:28:47 PM UTC. 2023-01-24T14:41:32.378 INFO:teuthology.orchestra.run.smithi179.stdout:Package librados2-2:18.0.0-1954.g1a671c19.el8.x86_64 is already installed. 2023-01-24T14:41:32.479 INFO:teuthology.orchestra.run.smithi179.stdout:Dependencies resolved. 2023-01-24T14:41:32.480 INFO:teuthology.orchestra.run.smithi179.stdout:Nothing to do. 2023-01-24T14:41:32.480 INFO:teuthology.orchestra.run.smithi179.stdout:Complete! 2023-01-24T14:41:32.578 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum -y install librbd1 2023-01-24T14:41:33.306 INFO:teuthology.orchestra.run.smithi179.stdout:Last metadata expiration check: 0:12:46 ago on Tue 24 Jan 2023 02:28:47 PM UTC. 2023-01-24T14:41:34.354 INFO:teuthology.orchestra.run.smithi179.stdout:Package librbd1-2:18.0.0-1954.g1a671c19.el8.x86_64 is already installed. 2023-01-24T14:41:34.457 INFO:teuthology.orchestra.run.smithi179.stdout:Dependencies resolved. 2023-01-24T14:41:34.458 INFO:teuthology.orchestra.run.smithi179.stdout:Nothing to do. 2023-01-24T14:41:34.459 INFO:teuthology.orchestra.run.smithi179.stdout:Complete! 2023-01-24T14:41:34.555 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum -y install python3-rados 2023-01-24T14:41:35.287 INFO:teuthology.orchestra.run.smithi179.stdout:Last metadata expiration check: 0:12:48 ago on Tue 24 Jan 2023 02:28:47 PM UTC. 2023-01-24T14:41:36.348 INFO:teuthology.orchestra.run.smithi179.stdout:Package python3-rados-2:18.0.0-1954.g1a671c19.el8.x86_64 is already installed. 2023-01-24T14:41:36.449 INFO:teuthology.orchestra.run.smithi179.stdout:Dependencies resolved. 2023-01-24T14:41:36.450 INFO:teuthology.orchestra.run.smithi179.stdout:Nothing to do. 2023-01-24T14:41:36.450 INFO:teuthology.orchestra.run.smithi179.stdout:Complete! 2023-01-24T14:41:36.551 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum -y install python3-rgw 2023-01-24T14:41:37.273 INFO:teuthology.orchestra.run.smithi179.stdout:Last metadata expiration check: 0:12:50 ago on Tue 24 Jan 2023 02:28:47 PM UTC. 2023-01-24T14:41:38.324 INFO:teuthology.orchestra.run.smithi179.stdout:Package python3-rgw-2:18.0.0-1954.g1a671c19.el8.x86_64 is already installed. 2023-01-24T14:41:38.427 INFO:teuthology.orchestra.run.smithi179.stdout:Dependencies resolved. 2023-01-24T14:41:38.428 INFO:teuthology.orchestra.run.smithi179.stdout:Nothing to do. 2023-01-24T14:41:38.428 INFO:teuthology.orchestra.run.smithi179.stdout:Complete! 2023-01-24T14:41:38.514 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum -y install python3-cephfs 2023-01-24T14:41:39.254 INFO:teuthology.orchestra.run.smithi179.stdout:Last metadata expiration check: 0:12:52 ago on Tue 24 Jan 2023 02:28:47 PM UTC. 2023-01-24T14:41:40.295 INFO:teuthology.orchestra.run.smithi179.stdout:Package python3-cephfs-2:18.0.0-1954.g1a671c19.el8.x86_64 is already installed. 2023-01-24T14:41:40.397 INFO:teuthology.orchestra.run.smithi179.stdout:Dependencies resolved. 2023-01-24T14:41:40.398 INFO:teuthology.orchestra.run.smithi179.stdout:Nothing to do. 2023-01-24T14:41:40.398 INFO:teuthology.orchestra.run.smithi179.stdout:Complete! 2023-01-24T14:41:40.499 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum -y install python3-rbd 2023-01-24T14:41:41.237 INFO:teuthology.orchestra.run.smithi179.stdout:Last metadata expiration check: 0:12:54 ago on Tue 24 Jan 2023 02:28:47 PM UTC. 2023-01-24T14:41:42.277 INFO:teuthology.orchestra.run.smithi179.stdout:Package python3-rbd-2:18.0.0-1954.g1a671c19.el8.x86_64 is already installed. 2023-01-24T14:41:42.378 INFO:teuthology.orchestra.run.smithi179.stdout:Dependencies resolved. 2023-01-24T14:41:42.379 INFO:teuthology.orchestra.run.smithi179.stdout:Nothing to do. 2023-01-24T14:41:42.379 INFO:teuthology.orchestra.run.smithi179.stdout:Complete! 2023-01-24T14:41:42.476 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum -y install rbd-fuse 2023-01-24T14:41:43.194 INFO:teuthology.orchestra.run.smithi179.stdout:Last metadata expiration check: 0:12:56 ago on Tue 24 Jan 2023 02:28:47 PM UTC. 2023-01-24T14:41:44.334 INFO:teuthology.orchestra.run.smithi179.stdout:Dependencies resolved. 2023-01-24T14:41:44.335 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2023-01-24T14:41:44.335 INFO:teuthology.orchestra.run.smithi179.stdout: Package Architecture Version Repository Size 2023-01-24T14:41:44.335 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2023-01-24T14:41:44.336 INFO:teuthology.orchestra.run.smithi179.stdout:Installing: 2023-01-24T14:41:44.336 INFO:teuthology.orchestra.run.smithi179.stdout: rbd-fuse x86_64 2:18.0.0-1954.g1a671c19.el8 ceph 91 k 2023-01-24T14:41:44.336 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-01-24T14:41:44.336 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction Summary 2023-01-24T14:41:44.336 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2023-01-24T14:41:44.337 INFO:teuthology.orchestra.run.smithi179.stdout:Install 1 Package 2023-01-24T14:41:44.337 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-01-24T14:41:44.338 INFO:teuthology.orchestra.run.smithi179.stdout:Total download size: 91 k 2023-01-24T14:41:44.338 INFO:teuthology.orchestra.run.smithi179.stdout:Installed size: 303 k 2023-01-24T14:41:44.338 INFO:teuthology.orchestra.run.smithi179.stdout:Downloading Packages: 2023-01-24T14:41:44.702 INFO:teuthology.orchestra.run.smithi179.stdout:rbd-fuse-18.0.0-1954.g1a671c19.el8.x86_64.rpm 249 kB/s | 91 kB 00:00 2023-01-24T14:41:44.703 INFO:teuthology.orchestra.run.smithi179.stdout:-------------------------------------------------------------------------------- 2023-01-24T14:41:44.703 INFO:teuthology.orchestra.run.smithi179.stdout:Total 248 kB/s | 91 kB 00:00 2023-01-24T14:41:44.703 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction check 2023-01-24T14:41:44.713 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction check succeeded. 2023-01-24T14:41:44.713 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction test 2023-01-24T14:41:44.765 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction test succeeded. 2023-01-24T14:41:44.769 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction 2023-01-24T14:41:44.890 INFO:teuthology.orchestra.run.smithi179.stdout: Preparing : 1/1 2023-01-24T14:41:45.154 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : rbd-fuse-2:18.0.0-1954.g1a671c19.el8.x86_64 1/1 2023-01-24T14:41:45.454 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: rbd-fuse-2:18.0.0-1954.g1a671c19.el8.x86_64 1/1 2023-01-24T14:41:46.015 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : rbd-fuse-2:18.0.0-1954.g1a671c19.el8.x86_64 1/1 2023-01-24T14:41:46.015 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-01-24T14:41:46.015 INFO:teuthology.orchestra.run.smithi179.stdout:Installed: 2023-01-24T14:41:46.015 INFO:teuthology.orchestra.run.smithi179.stdout: rbd-fuse-2:18.0.0-1954.g1a671c19.el8.x86_64 2023-01-24T14:41:46.016 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-01-24T14:41:46.016 INFO:teuthology.orchestra.run.smithi179.stdout:Complete! 2023-01-24T14:41:46.212 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum -y install rbd-mirror 2023-01-24T14:41:46.940 INFO:teuthology.orchestra.run.smithi179.stdout:Last metadata expiration check: 0:12:59 ago on Tue 24 Jan 2023 02:28:47 PM UTC. 2023-01-24T14:41:48.080 INFO:teuthology.orchestra.run.smithi179.stdout:Dependencies resolved. 2023-01-24T14:41:48.081 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2023-01-24T14:41:48.082 INFO:teuthology.orchestra.run.smithi179.stdout: Package Arch Version Repo Size 2023-01-24T14:41:48.082 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2023-01-24T14:41:48.082 INFO:teuthology.orchestra.run.smithi179.stdout:Installing: 2023-01-24T14:41:48.082 INFO:teuthology.orchestra.run.smithi179.stdout: rbd-mirror x86_64 2:18.0.0-1954.g1a671c19.el8 ceph 3.3 M 2023-01-24T14:41:48.082 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-01-24T14:41:48.083 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction Summary 2023-01-24T14:41:48.083 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2023-01-24T14:41:48.083 INFO:teuthology.orchestra.run.smithi179.stdout:Install 1 Package 2023-01-24T14:41:48.083 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-01-24T14:41:48.084 INFO:teuthology.orchestra.run.smithi179.stdout:Total download size: 3.3 M 2023-01-24T14:41:48.084 INFO:teuthology.orchestra.run.smithi179.stdout:Installed size: 18 M 2023-01-24T14:41:48.084 INFO:teuthology.orchestra.run.smithi179.stdout:Downloading Packages: 2023-01-24T14:41:48.964 INFO:teuthology.orchestra.run.smithi179.stdout:rbd-mirror-18.0.0-1954.g1a671c19.el8.x86_64.rpm 3.8 MB/s | 3.3 MB 00:00 2023-01-24T14:41:48.965 INFO:teuthology.orchestra.run.smithi179.stdout:-------------------------------------------------------------------------------- 2023-01-24T14:41:48.965 INFO:teuthology.orchestra.run.smithi179.stdout:Total 3.8 MB/s | 3.3 MB 00:00 2023-01-24T14:41:48.966 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction check 2023-01-24T14:41:48.978 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction check succeeded. 2023-01-24T14:41:48.978 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction test 2023-01-24T14:41:49.040 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction test succeeded. 2023-01-24T14:41:49.044 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction 2023-01-24T14:41:49.494 INFO:teuthology.orchestra.run.smithi179.stdout: Preparing : 1/1 2023-01-24T14:41:49.672 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : rbd-mirror-2:18.0.0-1954.g1a671c19.el8.x86_64 1/1 2023-01-24T14:41:50.115 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: rbd-mirror-2:18.0.0-1954.g1a671c19.el8.x86_64 1/1 2023-01-24T14:41:50.578 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : rbd-mirror-2:18.0.0-1954.g1a671c19.el8.x86_64 1/1 2023-01-24T14:41:50.578 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-01-24T14:41:50.579 INFO:teuthology.orchestra.run.smithi179.stdout:Installed: 2023-01-24T14:41:50.579 INFO:teuthology.orchestra.run.smithi179.stdout: rbd-mirror-2:18.0.0-1954.g1a671c19.el8.x86_64 2023-01-24T14:41:50.579 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-01-24T14:41:50.579 INFO:teuthology.orchestra.run.smithi179.stdout:Complete! 2023-01-24T14:41:50.761 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum -y install rbd-nbd 2023-01-24T14:41:51.496 INFO:teuthology.orchestra.run.smithi179.stdout:Last metadata expiration check: 0:13:04 ago on Tue 24 Jan 2023 02:28:47 PM UTC. 2023-01-24T14:41:52.641 INFO:teuthology.orchestra.run.smithi179.stdout:Dependencies resolved. 2023-01-24T14:41:52.641 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2023-01-24T14:41:52.641 INFO:teuthology.orchestra.run.smithi179.stdout: Package Architecture Version Repository Size 2023-01-24T14:41:52.642 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2023-01-24T14:41:52.642 INFO:teuthology.orchestra.run.smithi179.stdout:Installing: 2023-01-24T14:41:52.642 INFO:teuthology.orchestra.run.smithi179.stdout: rbd-nbd x86_64 2:18.0.0-1954.g1a671c19.el8 ceph 177 k 2023-01-24T14:41:52.642 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-01-24T14:41:52.642 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction Summary 2023-01-24T14:41:52.643 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2023-01-24T14:41:52.643 INFO:teuthology.orchestra.run.smithi179.stdout:Install 1 Package 2023-01-24T14:41:52.643 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-01-24T14:41:52.644 INFO:teuthology.orchestra.run.smithi179.stdout:Total download size: 177 k 2023-01-24T14:41:52.644 INFO:teuthology.orchestra.run.smithi179.stdout:Installed size: 626 k 2023-01-24T14:41:52.644 INFO:teuthology.orchestra.run.smithi179.stdout:Downloading Packages: 2023-01-24T14:41:53.009 INFO:teuthology.orchestra.run.smithi179.stdout:rbd-nbd-18.0.0-1954.g1a671c19.el8.x86_64.rpm 483 kB/s | 177 kB 00:00 2023-01-24T14:41:53.010 INFO:teuthology.orchestra.run.smithi179.stdout:-------------------------------------------------------------------------------- 2023-01-24T14:41:53.010 INFO:teuthology.orchestra.run.smithi179.stdout:Total 481 kB/s | 177 kB 00:00 2023-01-24T14:41:53.010 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction check 2023-01-24T14:41:53.020 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction check succeeded. 2023-01-24T14:41:53.021 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction test 2023-01-24T14:41:53.071 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction test succeeded. 2023-01-24T14:41:53.075 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction 2023-01-24T14:41:53.191 INFO:teuthology.orchestra.run.smithi179.stdout: Preparing : 1/1 2023-01-24T14:41:53.376 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : rbd-nbd-2:18.0.0-1954.g1a671c19.el8.x86_64 1/1 2023-01-24T14:41:53.676 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: rbd-nbd-2:18.0.0-1954.g1a671c19.el8.x86_64 1/1 2023-01-24T14:41:54.175 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : rbd-nbd-2:18.0.0-1954.g1a671c19.el8.x86_64 1/1 2023-01-24T14:41:54.175 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-01-24T14:41:54.175 INFO:teuthology.orchestra.run.smithi179.stdout:Installed: 2023-01-24T14:41:54.176 INFO:teuthology.orchestra.run.smithi179.stdout: rbd-nbd-2:18.0.0-1954.g1a671c19.el8.x86_64 2023-01-24T14:41:54.176 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-01-24T14:41:54.176 INFO:teuthology.orchestra.run.smithi179.stdout:Complete! 2023-01-24T14:41:54.389 DEBUG:teuthology.parallel:result is None 2023-01-24T14:41:54.390 DEBUG:teuthology.orchestra.run.smithi179:> sudo lsb_release -is 2023-01-24T14:41:54.428 INFO:teuthology.orchestra.run.smithi179.stdout:CentOSStream 2023-01-24T14:41:54.429 DEBUG:teuthology.misc:System to be installed: CentOSStream 2023-01-24T14:41:54.429 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F8%2Fx86_64&sha1=1a671c198559f54389cd5afa22085a0c7b29a1ef 2023-01-24T14:41:54.589 DEBUG:teuthology.orchestra.run.smithi179:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-01-24T14:41:54.611 INFO:teuthology.orchestra.run.smithi179.stdout:18.0.0-1954.g1a671c19.el8 2023-01-24T14:41:54.612 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-1954.g1a671c19.el8 2023-01-24T14:41:54.613 INFO:teuthology.task.install:The correct ceph version 18.0.0-1954.g1a671c19 is installed. 2023-01-24T14:41:54.613 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-01-24T14:41:54.613 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2023-01-24T14:41:54.614 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-01-24T14:41:54.679 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-01-24T14:41:54.679 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2023-01-24T14:41:54.680 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd of=/usr/bin/daemon-helper 2023-01-24T14:41:54.749 DEBUG:teuthology.orchestra.run.smithi179:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-01-24T14:41:54.816 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-01-24T14:41:54.817 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2023-01-24T14:41:54.817 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd of=/usr/bin/adjust-ulimits 2023-01-24T14:41:54.884 DEBUG:teuthology.orchestra.run.smithi179:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-01-24T14:41:54.952 INFO:teuthology.run_tasks:Running task ceph... 2023-01-24T14:41:55.025 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2023-01-24T14:41:55.026 DEBUG:teuthology.orchestra.run.smithi179:> sudo chmod 777 /var/log/ceph 2023-01-24T14:41:55.054 INFO:tasks.ceph:Disabling ceph logrotate... 2023-01-24T14:41:55.055 DEBUG:teuthology.orchestra.run.smithi179:> sudo rm -f -- /etc/logrotate.d/ceph 2023-01-24T14:41:55.122 INFO:tasks.ceph:Creating extra log directories... 2023-01-24T14:41:55.122 DEBUG:teuthology.orchestra.run.smithi179:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-01-24T14:41:55.198 INFO:tasks.ceph:Creating ceph cluster ceph... 2023-01-24T14:41:55.198 INFO:tasks.ceph:config {'conf': {'mon': {'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10, 'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'osd shutdown pgref assert': True}, 'global': {'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-01-24T14:41:55.199 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/rzarzynski-2023-01-24_14:08:39-crimson-rados-main-distro-default-smithi/7135317', 'branch': 'main', 'description': 'crimson-rados/rbd/{centos_latest clusters/fixed-1 crimson_qa_overrides deploy/ceph tasks/rbd_lock_and_fence}', 'email': None, 'first_in_suite': False, 'job_id': '7135317', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'rzarzynski-2023-01-24_14:08:39-crimson-rados-main-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'os_type': 'centos', 'os_version': '8.stream', 'overrides': {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'global': {'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '1a671c198559f54389cd5afa22085a0c7b29a1ef'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'global': {'osd crush chooseleaf type': 0, 'osd pool default pg num': 128, 'osd pool default pgp num': 128, 'osd pool default size': 2}, 'mon': {'osd default pool size': 2}}}, 'install': {'ceph': {'flavor': 'crimson', 'sha1': '1a671c198559f54389cd5afa22085a0c7b29a1ef'}}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'workunit': {'branch': 'main', 'env': {'CRIMSON_COMPAT': '1'}, 'sha1': '510284b66513490445619d1430aa869868c71a09'}}, 'owner': 'scheduled_rzarzynski@teuthology', 'priority': 60, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0']], 'sha1': '1a671c198559f54389cd5afa22085a0c7b29a1ef', 'sleep_before_teardown': 0, 'suite': 'crimson-rados', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_510284b66513490445619d1430aa869868c71a09/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '510284b66513490445619d1430aa869868c71a09', 'targets': {'smithi179.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDhcVdRI0vM1zDtZgKT9dJkSvZ2ScPJA++AreyF5LS+WNJw95QHg7bH3ztqfaElQcRF/7W0nP9mvKzB6JNgCtOodGC0AfXVksSzrrpGlzizloEFQ7ZHLZOv7WlhwmAq3+2hoeSGsEW9nDTWZgkNHRk6ytj3Ic7w0rmBloX41oZeCiSBJDFIHRMOvTf61J1lWSXucJYKTqiFM9/XzLr1zPZov3bTj29nHP8E+AqSXI3lV76oShfa5gPgkpvM0VJhbNQcnUMxe5r+C5OjvWGwbup8UNUMsJuD0QjDl7jP3giakJZ/aK2D3V4phnGPAn80VkK7CadnRiTNyuM/+6pcV37C5W3+2xc7Kdy+2Ykpp+ufQ44ORL6Ibpll6xQGlGG/Fpn0gvINGM/ArivhRN/5jt5Mj4TUVbjii3Q+beU2QzugtsP3kJ/Vo/ExPnM9mt3w6aProdoJ4s6EowljAU+dV7pjMIgKtrXqr/xAEmAN6PyWBeaS2mIhXi03WTr6JpDHDk8='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'conf': {'mon': {'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10, 'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'osd shutdown pgref assert': True}, 'global': {'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '1a671c198559f54389cd5afa22085a0c7b29a1ef', 'cluster': 'ceph'}}, {'workunit': {'clients': {'client.0': ['rbd/test_lock_fence.sh']}}}], 'teuthology_branch': 'main', 'teuthology_sha1': '8e1bba12e584b7ae912e92e94afe9c075ab51fa3', 'timestamp': '2023-01-24_14:08:39', 'tube': 'smithi', 'user': 'rzarzynski', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.17735'} 2023-01-24T14:41:55.199 DEBUG:teuthology.orchestra.run.smithi179:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-01-24T14:41:55.248 DEBUG:teuthology.orchestra.run.smithi179:> sudo install -d -m0777 -- /var/run/ceph 2023-01-24T14:41:55.316 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2023-01-24T14:41:55.316 DEBUG:teuthology.orchestra.run.smithi179:> dd if=/scratch_devs of=/dev/stdout 2023-01-24T14:41:55.373 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-01-24T14:41:55.374 DEBUG:teuthology.orchestra.run.smithi179:> stat /dev/vg_nvme/lv_1 2023-01-24T14:41:55.431 INFO:teuthology.orchestra.run.smithi179.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-01-24T14:41:55.431 INFO:teuthology.orchestra.run.smithi179.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-01-24T14:41:55.432 INFO:teuthology.orchestra.run.smithi179.stdout:Device: 6h/6d Inode: 374865 Links: 1 2023-01-24T14:41:55.432 INFO:teuthology.orchestra.run.smithi179.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-01-24T14:41:55.432 INFO:teuthology.orchestra.run.smithi179.stdout:Context: system_u:object_r:device_t:s0 2023-01-24T14:41:55.432 INFO:teuthology.orchestra.run.smithi179.stdout:Access: 2023-01-24 14:41:49.857180911 +0000 2023-01-24T14:41:55.433 INFO:teuthology.orchestra.run.smithi179.stdout:Modify: 2023-01-24 14:27:58.997385312 +0000 2023-01-24T14:41:55.433 INFO:teuthology.orchestra.run.smithi179.stdout:Change: 2023-01-24 14:27:58.997385312 +0000 2023-01-24T14:41:55.433 INFO:teuthology.orchestra.run.smithi179.stdout: Birth: - 2023-01-24T14:41:55.433 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-01-24T14:41:55.498 INFO:teuthology.orchestra.run.smithi179.stderr:1+0 records in 2023-01-24T14:41:55.499 INFO:teuthology.orchestra.run.smithi179.stderr:1+0 records out 2023-01-24T14:41:55.499 INFO:teuthology.orchestra.run.smithi179.stderr:512 bytes copied, 0.00024949 s, 2.1 MB/s 2023-01-24T14:41:55.499 DEBUG:teuthology.orchestra.run.smithi179:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-01-24T14:41:55.557 DEBUG:teuthology.orchestra.run.smithi179:> stat /dev/vg_nvme/lv_2 2023-01-24T14:41:55.615 INFO:teuthology.orchestra.run.smithi179.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-01-24T14:41:55.616 INFO:teuthology.orchestra.run.smithi179.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-01-24T14:41:55.616 INFO:teuthology.orchestra.run.smithi179.stdout:Device: 6h/6d Inode: 374906 Links: 1 2023-01-24T14:41:55.616 INFO:teuthology.orchestra.run.smithi179.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-01-24T14:41:55.616 INFO:teuthology.orchestra.run.smithi179.stdout:Context: system_u:object_r:device_t:s0 2023-01-24T14:41:55.617 INFO:teuthology.orchestra.run.smithi179.stdout:Access: 2023-01-24 14:41:49.857180911 +0000 2023-01-24T14:41:55.617 INFO:teuthology.orchestra.run.smithi179.stdout:Modify: 2023-01-24 14:27:59.507374099 +0000 2023-01-24T14:41:55.617 INFO:teuthology.orchestra.run.smithi179.stdout:Change: 2023-01-24 14:27:59.507374099 +0000 2023-01-24T14:41:55.617 INFO:teuthology.orchestra.run.smithi179.stdout: Birth: - 2023-01-24T14:41:55.618 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-01-24T14:41:55.681 INFO:teuthology.orchestra.run.smithi179.stderr:1+0 records in 2023-01-24T14:41:55.682 INFO:teuthology.orchestra.run.smithi179.stderr:1+0 records out 2023-01-24T14:41:55.682 INFO:teuthology.orchestra.run.smithi179.stderr:512 bytes copied, 0.000213348 s, 2.4 MB/s 2023-01-24T14:41:55.684 DEBUG:teuthology.orchestra.run.smithi179:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-01-24T14:41:55.743 DEBUG:teuthology.orchestra.run.smithi179:> stat /dev/vg_nvme/lv_3 2023-01-24T14:41:55.801 INFO:teuthology.orchestra.run.smithi179.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-01-24T14:41:55.801 INFO:teuthology.orchestra.run.smithi179.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-01-24T14:41:55.802 INFO:teuthology.orchestra.run.smithi179.stdout:Device: 6h/6d Inode: 377885 Links: 1 2023-01-24T14:41:55.802 INFO:teuthology.orchestra.run.smithi179.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-01-24T14:41:55.802 INFO:teuthology.orchestra.run.smithi179.stdout:Context: system_u:object_r:device_t:s0 2023-01-24T14:41:55.803 INFO:teuthology.orchestra.run.smithi179.stdout:Access: 2023-01-24 14:41:49.857180911 +0000 2023-01-24T14:41:55.803 INFO:teuthology.orchestra.run.smithi179.stdout:Modify: 2023-01-24 14:28:00.013362975 +0000 2023-01-24T14:41:55.803 INFO:teuthology.orchestra.run.smithi179.stdout:Change: 2023-01-24 14:28:00.013362975 +0000 2023-01-24T14:41:55.803 INFO:teuthology.orchestra.run.smithi179.stdout: Birth: - 2023-01-24T14:41:55.804 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-01-24T14:41:55.868 INFO:teuthology.orchestra.run.smithi179.stderr:1+0 records in 2023-01-24T14:41:55.868 INFO:teuthology.orchestra.run.smithi179.stderr:1+0 records out 2023-01-24T14:41:55.869 INFO:teuthology.orchestra.run.smithi179.stderr:512 bytes copied, 0.000232306 s, 2.2 MB/s 2023-01-24T14:41:55.870 DEBUG:teuthology.orchestra.run.smithi179:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-01-24T14:41:55.928 DEBUG:teuthology.orchestra.run.smithi179:> stat /dev/vg_nvme/lv_4 2023-01-24T14:41:55.987 INFO:teuthology.orchestra.run.smithi179.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-01-24T14:41:55.987 INFO:teuthology.orchestra.run.smithi179.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-01-24T14:41:55.987 INFO:teuthology.orchestra.run.smithi179.stdout:Device: 6h/6d Inode: 368393 Links: 1 2023-01-24T14:41:55.987 INFO:teuthology.orchestra.run.smithi179.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-01-24T14:41:55.988 INFO:teuthology.orchestra.run.smithi179.stdout:Context: system_u:object_r:device_t:s0 2023-01-24T14:41:55.988 INFO:teuthology.orchestra.run.smithi179.stdout:Access: 2023-01-24 14:41:49.857180911 +0000 2023-01-24T14:41:55.988 INFO:teuthology.orchestra.run.smithi179.stdout:Modify: 2023-01-24 14:28:00.688348137 +0000 2023-01-24T14:41:55.988 INFO:teuthology.orchestra.run.smithi179.stdout:Change: 2023-01-24 14:28:00.688348137 +0000 2023-01-24T14:41:55.988 INFO:teuthology.orchestra.run.smithi179.stdout: Birth: - 2023-01-24T14:41:55.989 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-01-24T14:41:56.053 INFO:teuthology.orchestra.run.smithi179.stderr:1+0 records in 2023-01-24T14:41:56.053 INFO:teuthology.orchestra.run.smithi179.stderr:1+0 records out 2023-01-24T14:41:56.053 INFO:teuthology.orchestra.run.smithi179.stderr:512 bytes copied, 0.000178105 s, 2.9 MB/s 2023-01-24T14:41:56.054 DEBUG:teuthology.orchestra.run.smithi179:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-01-24T14:41:56.113 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-01-24T14:41:56.114 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi179.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2023-01-24T14:41:56.114 INFO:tasks.ceph:Generating config... 2023-01-24T14:41:56.115 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2023-01-24T14:41:56.116 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2023-01-24T14:41:56.116 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2023-01-24T14:41:56.116 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2023-01-24T14:41:56.117 INFO:tasks.ceph:[mon] paxos service trim min = 10 2023-01-24T14:41:56.117 INFO:tasks.ceph:[mon] debug mon = 20 2023-01-24T14:41:56.117 INFO:tasks.ceph:[mon] debug ms = 1 2023-01-24T14:41:56.117 INFO:tasks.ceph:[mon] debug paxos = 20 2023-01-24T14:41:56.118 INFO:tasks.ceph:[osd] debug monc = 20 2023-01-24T14:41:56.118 INFO:tasks.ceph:[osd] debug ms = 1 2023-01-24T14:41:56.118 INFO:tasks.ceph:[osd] debug osd = 20 2023-01-24T14:41:56.119 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2023-01-24T14:41:56.119 INFO:tasks.ceph:[global] ms client mode = crc 2023-01-24T14:41:56.119 INFO:tasks.ceph:[global] ms cluster mode = crc 2023-01-24T14:41:56.119 INFO:tasks.ceph:[global] ms mon client mode = crc 2023-01-24T14:41:56.120 INFO:tasks.ceph:[global] ms mon cluster mode = crc 2023-01-24T14:41:56.120 INFO:tasks.ceph:[global] ms mon service mode = crc 2023-01-24T14:41:56.120 INFO:tasks.ceph:[global] ms service mode = crc 2023-01-24T14:41:56.120 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-01-24T14:41:56.121 INFO:tasks.ceph:[mgr] debug ms = 1 2023-01-24T14:41:56.121 INFO:tasks.ceph:Setting up mon.a... 2023-01-24T14:41:56.121 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2023-01-24T14:41:56.220 INFO:teuthology.orchestra.run.smithi179.stdout:creating /etc/ceph/ceph.keyring 2023-01-24T14:41:56.223 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2023-01-24T14:41:56.277 DEBUG:teuthology.orchestra.run.smithi179:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-01-24T14:41:56.304 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.179')] 2023-01-24T14:41:56.305 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.179', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}} 2023-01-24T14:41:56.306 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2023-01-24T14:41:56.306 DEBUG:teuthology.orchestra.run.smithi179:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-01-24T14:41:56.363 DEBUG:teuthology.orchestra.run.smithi179:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.179 --print /home/ubuntu/cephtest/ceph.monmap 2023-01-24T14:41:56.454 INFO:teuthology.orchestra.run.smithi179.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2023-01-24T14:41:56.455 INFO:teuthology.orchestra.run.smithi179.stdout:monmaptool: generated fsid 91764032-a053-4a42-a007-59362bb9c17e 2023-01-24T14:41:56.455 INFO:teuthology.orchestra.run.smithi179.stdout:setting min_mon_release = pacific 2023-01-24T14:41:56.455 INFO:teuthology.orchestra.run.smithi179.stdout:epoch 0 2023-01-24T14:41:56.455 INFO:teuthology.orchestra.run.smithi179.stdout:fsid 91764032-a053-4a42-a007-59362bb9c17e 2023-01-24T14:41:56.455 INFO:teuthology.orchestra.run.smithi179.stdout:last_changed 2023-01-24T14:41:56.454230+0000 2023-01-24T14:41:56.456 INFO:teuthology.orchestra.run.smithi179.stdout:created 2023-01-24T14:41:56.454230+0000 2023-01-24T14:41:56.456 INFO:teuthology.orchestra.run.smithi179.stdout:min_mon_release 16 (pacific) 2023-01-24T14:41:56.456 INFO:teuthology.orchestra.run.smithi179.stdout:election_strategy: 1 2023-01-24T14:41:56.456 INFO:teuthology.orchestra.run.smithi179.stdout:0: [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] mon.a 2023-01-24T14:41:56.456 INFO:teuthology.orchestra.run.smithi179.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (1 monitors) 2023-01-24T14:41:56.458 DEBUG:teuthology.orchestra.run.smithi179:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-01-24T14:41:56.475 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 91764032-a053-4a42-a007-59362bb9c17e... 2023-01-24T14:41:56.476 DEBUG:teuthology.orchestra.run.smithi179:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-01-24T14:41:56.561 INFO:teuthology.orchestra.run.smithi179.stdout:[global] 2023-01-24T14:41:56.562 INFO:teuthology.orchestra.run.smithi179.stdout: chdir = "" 2023-01-24T14:41:56.562 INFO:teuthology.orchestra.run.smithi179.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-01-24T14:41:56.562 INFO:teuthology.orchestra.run.smithi179.stdout: auth supported = cephx 2023-01-24T14:41:56.562 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-01-24T14:41:56.563 INFO:teuthology.orchestra.run.smithi179.stdout: filestore xattr use omap = true 2023-01-24T14:41:56.563 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-01-24T14:41:56.563 INFO:teuthology.orchestra.run.smithi179.stdout: mon clock drift allowed = 1.000 2023-01-24T14:41:56.563 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-01-24T14:41:56.563 INFO:teuthology.orchestra.run.smithi179.stdout: osd crush chooseleaf type = 0 2023-01-24T14:41:56.564 INFO:teuthology.orchestra.run.smithi179.stdout: auth debug = true 2023-01-24T14:41:56.564 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-01-24T14:41:56.564 INFO:teuthology.orchestra.run.smithi179.stdout: ms die on old message = true 2023-01-24T14:41:56.564 INFO:teuthology.orchestra.run.smithi179.stdout: ms die on bug = true 2023-01-24T14:41:56.564 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-01-24T14:41:56.565 INFO:teuthology.orchestra.run.smithi179.stdout: mon max pg per osd = 10000 # >= luminous 2023-01-24T14:41:56.565 INFO:teuthology.orchestra.run.smithi179.stdout: mon pg warn max object skew = 0 2023-01-24T14:41:56.565 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-01-24T14:41:56.565 INFO:teuthology.orchestra.run.smithi179.stdout: # disable pg_autoscaler by default for new pools 2023-01-24T14:41:56.565 INFO:teuthology.orchestra.run.smithi179.stdout: osd_pool_default_pg_autoscale_mode = off 2023-01-24T14:41:56.566 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-01-24T14:41:56.566 INFO:teuthology.orchestra.run.smithi179.stdout: osd pool default size = 2 2023-01-24T14:41:56.566 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-01-24T14:41:56.566 INFO:teuthology.orchestra.run.smithi179.stdout: mon osd allow primary affinity = true 2023-01-24T14:41:56.566 INFO:teuthology.orchestra.run.smithi179.stdout: mon osd allow pg remap = true 2023-01-24T14:41:56.567 INFO:teuthology.orchestra.run.smithi179.stdout: mon warn on legacy crush tunables = false 2023-01-24T14:41:56.567 INFO:teuthology.orchestra.run.smithi179.stdout: mon warn on crush straw calc version zero = false 2023-01-24T14:41:56.567 INFO:teuthology.orchestra.run.smithi179.stdout: mon warn on no sortbitwise = false 2023-01-24T14:41:56.567 INFO:teuthology.orchestra.run.smithi179.stdout: mon warn on osd down out interval zero = false 2023-01-24T14:41:56.567 INFO:teuthology.orchestra.run.smithi179.stdout: mon warn on too few osds = false 2023-01-24T14:41:56.568 INFO:teuthology.orchestra.run.smithi179.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-01-24T14:41:56.568 INFO:teuthology.orchestra.run.smithi179.stdout: mon_warn_on_pool_no_redundancy = false 2023-01-24T14:41:56.568 INFO:teuthology.orchestra.run.smithi179.stdout: mon_allow_pool_size_one = true 2023-01-24T14:41:56.568 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-01-24T14:41:56.569 INFO:teuthology.orchestra.run.smithi179.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-01-24T14:41:56.569 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-01-24T14:41:56.569 INFO:teuthology.orchestra.run.smithi179.stdout: osd default data pool replay window = 5 2023-01-24T14:41:56.569 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-01-24T14:41:56.569 INFO:teuthology.orchestra.run.smithi179.stdout: mon allow pool delete = true 2023-01-24T14:41:56.569 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-01-24T14:41:56.570 INFO:teuthology.orchestra.run.smithi179.stdout: mon cluster log file level = debug 2023-01-24T14:41:56.570 INFO:teuthology.orchestra.run.smithi179.stdout: debug asserts on shutdown = true 2023-01-24T14:41:56.570 INFO:teuthology.orchestra.run.smithi179.stdout: mon health detail to clog = false 2023-01-24T14:41:56.570 INFO:teuthology.orchestra.run.smithi179.stdout: mon host = 172.21.15.179 2023-01-24T14:41:56.570 INFO:teuthology.orchestra.run.smithi179.stdout: ms client mode = crc 2023-01-24T14:41:56.571 INFO:teuthology.orchestra.run.smithi179.stdout: ms cluster mode = crc 2023-01-24T14:41:56.571 INFO:teuthology.orchestra.run.smithi179.stdout: ms mon client mode = crc 2023-01-24T14:41:56.571 INFO:teuthology.orchestra.run.smithi179.stdout: ms mon cluster mode = crc 2023-01-24T14:41:56.571 INFO:teuthology.orchestra.run.smithi179.stdout: ms mon service mode = crc 2023-01-24T14:41:56.571 INFO:teuthology.orchestra.run.smithi179.stdout: ms service mode = crc 2023-01-24T14:41:56.572 INFO:teuthology.orchestra.run.smithi179.stdout: fsid = 91764032-a053-4a42-a007-59362bb9c17e 2023-01-24T14:41:56.572 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-01-24T14:41:56.572 INFO:teuthology.orchestra.run.smithi179.stdout:[osd] 2023-01-24T14:41:56.572 INFO:teuthology.orchestra.run.smithi179.stdout: osd journal size = 100 2023-01-24T14:41:56.572 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-01-24T14:41:56.573 INFO:teuthology.orchestra.run.smithi179.stdout: osd scrub load threshold = 5.0 2023-01-24T14:41:56.573 INFO:teuthology.orchestra.run.smithi179.stdout: osd scrub max interval = 600 2023-01-24T14:41:56.573 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-01-24T14:41:56.573 INFO:teuthology.orchestra.run.smithi179.stdout: osd recover clone overlap = true 2023-01-24T14:41:56.573 INFO:teuthology.orchestra.run.smithi179.stdout: osd recovery max chunk = 1048576 2023-01-24T14:41:56.574 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-01-24T14:41:56.574 INFO:teuthology.orchestra.run.smithi179.stdout: osd debug shutdown = true 2023-01-24T14:41:56.574 INFO:teuthology.orchestra.run.smithi179.stdout: osd debug op order = true 2023-01-24T14:41:56.574 INFO:teuthology.orchestra.run.smithi179.stdout: osd debug verify stray on activate = true 2023-01-24T14:41:56.574 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-01-24T14:41:56.575 INFO:teuthology.orchestra.run.smithi179.stdout: osd open classes on start = true 2023-01-24T14:41:56.575 INFO:teuthology.orchestra.run.smithi179.stdout: osd debug pg log writeout = true 2023-01-24T14:41:56.575 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-01-24T14:41:56.575 INFO:teuthology.orchestra.run.smithi179.stdout: osd deep scrub update digest min age = 30 2023-01-24T14:41:56.575 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-01-24T14:41:56.576 INFO:teuthology.orchestra.run.smithi179.stdout: osd map max advance = 10 2023-01-24T14:41:56.576 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-01-24T14:41:56.576 INFO:teuthology.orchestra.run.smithi179.stdout: journal zero on create = true 2023-01-24T14:41:56.576 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-01-24T14:41:56.576 INFO:teuthology.orchestra.run.smithi179.stdout: filestore ondisk finisher threads = 3 2023-01-24T14:41:56.577 INFO:teuthology.orchestra.run.smithi179.stdout: filestore apply finisher threads = 3 2023-01-24T14:41:56.577 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-01-24T14:41:56.577 INFO:teuthology.orchestra.run.smithi179.stdout: bdev debug aio = true 2023-01-24T14:41:56.578 INFO:teuthology.orchestra.run.smithi179.stdout: osd debug misdirected ops = true 2023-01-24T14:41:56.578 INFO:teuthology.orchestra.run.smithi179.stdout: debug monc = 20 2023-01-24T14:41:56.578 INFO:teuthology.orchestra.run.smithi179.stdout: debug ms = 1 2023-01-24T14:41:56.578 INFO:teuthology.orchestra.run.smithi179.stdout: debug osd = 20 2023-01-24T14:41:56.578 INFO:teuthology.orchestra.run.smithi179.stdout: osd shutdown pgref assert = True 2023-01-24T14:41:56.579 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-01-24T14:41:56.579 INFO:teuthology.orchestra.run.smithi179.stdout:[mgr] 2023-01-24T14:41:56.579 INFO:teuthology.orchestra.run.smithi179.stdout: debug ms = 1 2023-01-24T14:41:56.579 INFO:teuthology.orchestra.run.smithi179.stdout: debug mgr = 20 2023-01-24T14:41:56.579 INFO:teuthology.orchestra.run.smithi179.stdout: debug mon = 20 2023-01-24T14:41:56.580 INFO:teuthology.orchestra.run.smithi179.stdout: debug auth = 20 2023-01-24T14:41:56.580 INFO:teuthology.orchestra.run.smithi179.stdout: mon reweight min pgs per osd = 4 2023-01-24T14:41:56.580 INFO:teuthology.orchestra.run.smithi179.stdout: mon reweight min bytes per osd = 10 2023-01-24T14:41:56.580 INFO:teuthology.orchestra.run.smithi179.stdout: mgr/telemetry/nag = false 2023-01-24T14:41:56.581 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-01-24T14:41:56.581 INFO:teuthology.orchestra.run.smithi179.stdout:[mon] 2023-01-24T14:41:56.581 INFO:teuthology.orchestra.run.smithi179.stdout: debug ms = 1 2023-01-24T14:41:56.581 INFO:teuthology.orchestra.run.smithi179.stdout: debug mon = 20 2023-01-24T14:41:56.581 INFO:teuthology.orchestra.run.smithi179.stdout: debug paxos = 20 2023-01-24T14:41:56.582 INFO:teuthology.orchestra.run.smithi179.stdout: debug auth = 20 2023-01-24T14:41:56.582 INFO:teuthology.orchestra.run.smithi179.stdout: mon data avail warn = 5 2023-01-24T14:41:56.582 INFO:teuthology.orchestra.run.smithi179.stdout: mon mgr mkfs grace = 240 2023-01-24T14:41:56.582 INFO:teuthology.orchestra.run.smithi179.stdout: mon reweight min pgs per osd = 4 2023-01-24T14:41:56.582 INFO:teuthology.orchestra.run.smithi179.stdout: mon osd reporter subtree level = osd 2023-01-24T14:41:56.583 INFO:teuthology.orchestra.run.smithi179.stdout: mon osd prime pg temp = true 2023-01-24T14:41:56.583 INFO:teuthology.orchestra.run.smithi179.stdout: mon reweight min bytes per osd = 10 2023-01-24T14:41:56.583 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-01-24T14:41:56.583 INFO:teuthology.orchestra.run.smithi179.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-01-24T14:41:56.583 INFO:teuthology.orchestra.run.smithi179.stdout: auth mon ticket ttl = 660 # 11m 2023-01-24T14:41:56.584 INFO:teuthology.orchestra.run.smithi179.stdout: auth service ticket ttl = 240 # 4m 2023-01-24T14:41:56.584 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-01-24T14:41:56.584 INFO:teuthology.orchestra.run.smithi179.stdout: # don't complain about insecure global_id in the test suite 2023-01-24T14:41:56.584 INFO:teuthology.orchestra.run.smithi179.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-01-24T14:41:56.584 INFO:teuthology.orchestra.run.smithi179.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-01-24T14:41:56.585 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-01-24T14:41:56.585 INFO:teuthology.orchestra.run.smithi179.stdout: # 1m isn't quite enough 2023-01-24T14:41:56.585 INFO:teuthology.orchestra.run.smithi179.stdout: mon_down_mkfs_grace = 2m 2023-01-24T14:41:56.585 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-01-24T14:41:56.585 INFO:teuthology.orchestra.run.smithi179.stdout: mon_warn_on_filestore_osds = false 2023-01-24T14:41:56.586 INFO:teuthology.orchestra.run.smithi179.stdout: mon min osdmap epochs = 50 2023-01-24T14:41:56.586 INFO:teuthology.orchestra.run.smithi179.stdout: mon osdmap full prune interval = 2 2023-01-24T14:41:56.586 INFO:teuthology.orchestra.run.smithi179.stdout: mon osdmap full prune min = 15 2023-01-24T14:41:56.586 INFO:teuthology.orchestra.run.smithi179.stdout: mon osdmap full prune txsize = 2 2023-01-24T14:41:56.586 INFO:teuthology.orchestra.run.smithi179.stdout: paxos service trim min = 10 2023-01-24T14:41:56.587 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-01-24T14:41:56.587 INFO:teuthology.orchestra.run.smithi179.stdout:[client] 2023-01-24T14:41:56.587 INFO:teuthology.orchestra.run.smithi179.stdout: rgw cache enabled = true 2023-01-24T14:41:56.587 INFO:teuthology.orchestra.run.smithi179.stdout: rgw enable ops log = true 2023-01-24T14:41:56.587 INFO:teuthology.orchestra.run.smithi179.stdout: rgw enable usage log = true 2023-01-24T14:41:56.588 INFO:teuthology.orchestra.run.smithi179.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-01-24T14:41:56.588 INFO:teuthology.orchestra.run.smithi179.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-01-24T14:41:56.588 INFO:teuthology.orchestra.run.smithi179.stdout:[mon.a] 2023-01-24T14:41:56.589 INFO:tasks.ceph:Creating admin key on mon.a... 2023-01-24T14:41:56.589 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2023-01-24T14:41:56.694 INFO:tasks.ceph:Copying monmap to all nodes... 2023-01-24T14:41:56.695 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2023-01-24T14:41:56.695 DEBUG:teuthology.orchestra.run.smithi179:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2023-01-24T14:41:56.714 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2023-01-24T14:41:56.715 DEBUG:teuthology.orchestra.run.smithi179:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2023-01-24T14:41:56.772 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi179.front.sepia.ceph.com 2023-01-24T14:41:56.773 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2023-01-24T14:41:56.773 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd of=/etc/ceph/ceph.keyring 2023-01-24T14:41:56.774 DEBUG:teuthology.orchestra.run.smithi179:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-01-24T14:41:56.850 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2023-01-24T14:41:56.850 DEBUG:teuthology.orchestra.run.smithi179:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-01-24T14:41:56.907 INFO:tasks.ceph:Setting up mon nodes... 2023-01-24T14:41:56.908 INFO:tasks.ceph:Setting up mgr nodes... 2023-01-24T14:41:56.908 DEBUG:teuthology.orchestra.run.smithi179:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2023-01-24T14:41:57.008 INFO:teuthology.orchestra.run.smithi179.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2023-01-24T14:41:57.011 INFO:tasks.ceph:Setting up mds nodes... 2023-01-24T14:41:57.012 INFO:tasks.ceph_client:Setting up client nodes... 2023-01-24T14:41:57.012 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-01-24T14:41:57.063 INFO:teuthology.orchestra.run.smithi179.stdout:creating /etc/ceph/ceph.client.0.keyring 2023-01-24T14:41:57.076 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-01-24T14:41:57.077 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi179.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2023-01-24T14:41:57.077 DEBUG:teuthology.orchestra.run.smithi179:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2023-01-24T14:41:57.103 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-01-24T14:41:57.104 INFO:tasks.ceph:role: osd.0 2023-01-24T14:41:57.104 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi179.front.sepia.ceph.com 2023-01-24T14:41:57.104 DEBUG:teuthology.orchestra.run.smithi179:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-01-24T14:41:57.178 INFO:teuthology.orchestra.run.smithi179.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-01-24T14:41:57.179 INFO:teuthology.orchestra.run.smithi179.stdout: = sectsz=512 attr=2, projid32bit=1 2023-01-24T14:41:57.179 INFO:teuthology.orchestra.run.smithi179.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-01-24T14:41:57.179 INFO:teuthology.orchestra.run.smithi179.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-01-24T14:41:57.179 INFO:teuthology.orchestra.run.smithi179.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-01-24T14:41:57.179 INFO:teuthology.orchestra.run.smithi179.stdout: = sunit=0 swidth=0 blks 2023-01-24T14:41:57.180 INFO:teuthology.orchestra.run.smithi179.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-01-24T14:41:57.180 INFO:teuthology.orchestra.run.smithi179.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-01-24T14:41:57.180 INFO:teuthology.orchestra.run.smithi179.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-01-24T14:41:57.180 INFO:teuthology.orchestra.run.smithi179.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-01-24T14:41:57.608 INFO:teuthology.orchestra.run.smithi179.stdout:Discarding blocks...Done. 2023-01-24T14:41:57.609 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi179.front.sepia.ceph.com -o noatime 2023-01-24T14:41:57.610 DEBUG:teuthology.orchestra.run.smithi179:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2023-01-24T14:41:57.777 DEBUG:teuthology.orchestra.run.smithi179:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2023-01-24T14:41:57.813 DEBUG:teuthology.orchestra.run.smithi179:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2023-01-24T14:41:57.880 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-01-24T14:41:57.881 INFO:tasks.ceph:role: osd.1 2023-01-24T14:41:57.881 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi179.front.sepia.ceph.com 2023-01-24T14:41:57.881 DEBUG:teuthology.orchestra.run.smithi179:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-01-24T14:41:57.955 INFO:teuthology.orchestra.run.smithi179.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-01-24T14:41:57.955 INFO:teuthology.orchestra.run.smithi179.stdout: = sectsz=512 attr=2, projid32bit=1 2023-01-24T14:41:57.956 INFO:teuthology.orchestra.run.smithi179.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-01-24T14:41:57.956 INFO:teuthology.orchestra.run.smithi179.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-01-24T14:41:57.956 INFO:teuthology.orchestra.run.smithi179.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-01-24T14:41:57.956 INFO:teuthology.orchestra.run.smithi179.stdout: = sunit=0 swidth=0 blks 2023-01-24T14:41:57.957 INFO:teuthology.orchestra.run.smithi179.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-01-24T14:41:57.957 INFO:teuthology.orchestra.run.smithi179.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-01-24T14:41:57.957 INFO:teuthology.orchestra.run.smithi179.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-01-24T14:41:57.957 INFO:teuthology.orchestra.run.smithi179.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-01-24T14:41:58.381 INFO:teuthology.orchestra.run.smithi179.stdout:Discarding blocks...Done. 2023-01-24T14:41:58.383 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi179.front.sepia.ceph.com -o noatime 2023-01-24T14:41:58.383 DEBUG:teuthology.orchestra.run.smithi179:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2023-01-24T14:41:58.499 DEBUG:teuthology.orchestra.run.smithi179:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2023-01-24T14:41:58.531 DEBUG:teuthology.orchestra.run.smithi179:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2023-01-24T14:41:58.599 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-01-24T14:41:58.599 INFO:tasks.ceph:role: osd.2 2023-01-24T14:41:58.600 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi179.front.sepia.ceph.com 2023-01-24T14:41:58.600 DEBUG:teuthology.orchestra.run.smithi179:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-01-24T14:41:58.674 INFO:teuthology.orchestra.run.smithi179.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-01-24T14:41:58.675 INFO:teuthology.orchestra.run.smithi179.stdout: = sectsz=512 attr=2, projid32bit=1 2023-01-24T14:41:58.675 INFO:teuthology.orchestra.run.smithi179.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-01-24T14:41:58.675 INFO:teuthology.orchestra.run.smithi179.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-01-24T14:41:58.676 INFO:teuthology.orchestra.run.smithi179.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-01-24T14:41:58.676 INFO:teuthology.orchestra.run.smithi179.stdout: = sunit=0 swidth=0 blks 2023-01-24T14:41:58.676 INFO:teuthology.orchestra.run.smithi179.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-01-24T14:41:58.676 INFO:teuthology.orchestra.run.smithi179.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-01-24T14:41:58.676 INFO:teuthology.orchestra.run.smithi179.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-01-24T14:41:58.677 INFO:teuthology.orchestra.run.smithi179.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-01-24T14:41:59.124 INFO:teuthology.orchestra.run.smithi179.stdout:Discarding blocks...Done. 2023-01-24T14:41:59.125 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi179.front.sepia.ceph.com -o noatime 2023-01-24T14:41:59.126 DEBUG:teuthology.orchestra.run.smithi179:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2023-01-24T14:41:59.296 DEBUG:teuthology.orchestra.run.smithi179:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2023-01-24T14:41:59.332 DEBUG:teuthology.orchestra.run.smithi179:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-01-24T14:42:00.138 INFO:teuthology.orchestra.run.smithi179.stderr:WARNING: debug mode. Not for benchmarking or production 2023-01-24T14:42:00.139 INFO:teuthology.orchestra.run.smithi179.stderr:INFO 2023-01-24 14:42:00,139 seastar - Reactor backend: epoll 2023-01-24T14:42:00.147 INFO:teuthology.orchestra.run.smithi179.stderr:INFO 2023-01-24 14:42:00,147 [shard 0] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-01-24T14:42:00.147 INFO:teuthology.orchestra.run.smithi179.stderr:INFO 2023-01-24 14:42:00,147 [shard 0] seastar - IO queue uses 0.75ms latency goal for device 0 2023-01-24T14:42:00.148 INFO:teuthology.orchestra.run.smithi179.stderr:INFO 2023-01-24 14:42:00,147 [shard 0] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-01-24T14:42:00.148 INFO:teuthology.orchestra.run.smithi179.stderr:INFO 2023-01-24 14:42:00,147 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-01-24T14:42:00.149 INFO:teuthology.orchestra.run.smithi179.stderr:==100502==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-01-24T14:42:00.156 INFO:teuthology.orchestra.run.smithi179.stderr:==100502==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7fff5a76d000; bottom 0x7efde8f94000; size: 0x0101717d9000 (1105710649344) 2023-01-24T14:42:00.156 INFO:teuthology.orchestra.run.smithi179.stderr:False positive error reports may follow 2023-01-24T14:42:00.156 INFO:teuthology.orchestra.run.smithi179.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-01-24T14:42:00.190 INFO:teuthology.orchestra.run.smithi179.stderr:INFO 2023-01-24 14:42:00,190 [shard 0] osd - bypassing the config fetch due to --no-mon-config 2023-01-24T14:42:00.191 INFO:teuthology.orchestra.run.smithi179.stderr:INFO 2023-01-24 14:42:00,190 [shard 0] osd - get_early_config: set --smp 1 2023-01-24T14:42:00.675 INFO:teuthology.orchestra.run.smithi179.stderr:WARNING: debug mode. Not for benchmarking or production 2023-01-24T14:42:00.676 INFO:teuthology.orchestra.run.smithi179.stderr:INFO 2023-01-24 14:42:00,675 seastar - Reactor backend: epoll 2023-01-24T14:42:00.677 INFO:teuthology.orchestra.run.smithi179.stderr:INFO 2023-01-24 14:42:00,677 [shard 0] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-01-24T14:42:00.677 INFO:teuthology.orchestra.run.smithi179.stderr:INFO 2023-01-24 14:42:00,677 [shard 0] seastar - IO queue uses 0.75ms latency goal for device 0 2023-01-24T14:42:00.678 INFO:teuthology.orchestra.run.smithi179.stderr:INFO 2023-01-24 14:42:00,677 [shard 0] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-01-24T14:42:00.678 INFO:teuthology.orchestra.run.smithi179.stderr:INFO 2023-01-24 14:42:00,677 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-01-24T14:42:00.679 INFO:teuthology.orchestra.run.smithi179.stderr:==100500==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-01-24T14:42:00.685 INFO:teuthology.orchestra.run.smithi179.stderr:==100500==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7fff5a76d000; bottom 0x7efde8f8d000; size: 0x0101717e0000 (1105710678016) 2023-01-24T14:42:00.685 INFO:teuthology.orchestra.run.smithi179.stderr:False positive error reports may follow 2023-01-24T14:42:00.685 INFO:teuthology.orchestra.run.smithi179.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-01-24T14:42:03.595 INFO:teuthology.orchestra.run.smithi179.stdout:created object store /var/lib/ceph/osd/ceph-0 for osd.0 fsid 91764032-a053-4a42-a007-59362bb9c17e 2023-01-24T14:42:03.655 DEBUG:teuthology.orchestra.run.smithi179:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2023-01-24T14:42:03.682 DEBUG:teuthology.orchestra.run.smithi179:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-01-24T14:42:04.458 INFO:teuthology.orchestra.run.smithi179.stderr:WARNING: debug mode. Not for benchmarking or production 2023-01-24T14:42:04.459 INFO:teuthology.orchestra.run.smithi179.stderr:INFO 2023-01-24 14:42:04,459 seastar - Reactor backend: epoll 2023-01-24T14:42:04.471 INFO:teuthology.orchestra.run.smithi179.stderr:INFO 2023-01-24 14:42:04,470 [shard 0] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-01-24T14:42:04.471 INFO:teuthology.orchestra.run.smithi179.stderr:INFO 2023-01-24 14:42:04,470 [shard 0] seastar - IO queue uses 0.75ms latency goal for device 0 2023-01-24T14:42:04.472 INFO:teuthology.orchestra.run.smithi179.stderr:INFO 2023-01-24 14:42:04,470 [shard 0] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-01-24T14:42:04.472 INFO:teuthology.orchestra.run.smithi179.stderr:INFO 2023-01-24 14:42:04,471 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-01-24T14:42:04.473 INFO:teuthology.orchestra.run.smithi179.stderr:==101320==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-01-24T14:42:04.479 INFO:teuthology.orchestra.run.smithi179.stderr:==101320==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7ffd722cd000; bottom 0x7f2517c42000; size: 0x00d85a68b000 (929229746176) 2023-01-24T14:42:04.479 INFO:teuthology.orchestra.run.smithi179.stderr:False positive error reports may follow 2023-01-24T14:42:04.480 INFO:teuthology.orchestra.run.smithi179.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-01-24T14:42:04.514 INFO:teuthology.orchestra.run.smithi179.stderr:INFO 2023-01-24 14:42:04,513 [shard 0] osd - bypassing the config fetch due to --no-mon-config 2023-01-24T14:42:04.514 INFO:teuthology.orchestra.run.smithi179.stderr:INFO 2023-01-24 14:42:04,513 [shard 0] osd - get_early_config: set --smp 1 2023-01-24T14:42:05.006 INFO:teuthology.orchestra.run.smithi179.stderr:WARNING: debug mode. Not for benchmarking or production 2023-01-24T14:42:05.007 INFO:teuthology.orchestra.run.smithi179.stderr:INFO 2023-01-24 14:42:05,006 seastar - Reactor backend: epoll 2023-01-24T14:42:05.008 INFO:teuthology.orchestra.run.smithi179.stderr:INFO 2023-01-24 14:42:05,008 [shard 0] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-01-24T14:42:05.008 INFO:teuthology.orchestra.run.smithi179.stderr:INFO 2023-01-24 14:42:05,008 [shard 0] seastar - IO queue uses 0.75ms latency goal for device 0 2023-01-24T14:42:05.009 INFO:teuthology.orchestra.run.smithi179.stderr:INFO 2023-01-24 14:42:05,008 [shard 0] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-01-24T14:42:05.009 INFO:teuthology.orchestra.run.smithi179.stderr:INFO 2023-01-24 14:42:05,008 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-01-24T14:42:05.010 INFO:teuthology.orchestra.run.smithi179.stderr:==101318==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-01-24T14:42:05.016 INFO:teuthology.orchestra.run.smithi179.stderr:==101318==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7ffd722cd000; bottom 0x7f2517c3b000; size: 0x00d85a692000 (929229774848) 2023-01-24T14:42:05.016 INFO:teuthology.orchestra.run.smithi179.stderr:False positive error reports may follow 2023-01-24T14:42:05.016 INFO:teuthology.orchestra.run.smithi179.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-01-24T14:42:07.939 INFO:teuthology.orchestra.run.smithi179.stdout:created object store /var/lib/ceph/osd/ceph-1 for osd.1 fsid 91764032-a053-4a42-a007-59362bb9c17e 2023-01-24T14:42:08.027 DEBUG:teuthology.orchestra.run.smithi179:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2023-01-24T14:42:08.055 DEBUG:teuthology.orchestra.run.smithi179:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-01-24T14:42:08.830 INFO:teuthology.orchestra.run.smithi179.stderr:WARNING: debug mode. Not for benchmarking or production 2023-01-24T14:42:08.831 INFO:teuthology.orchestra.run.smithi179.stderr:INFO 2023-01-24 14:42:08,831 seastar - Reactor backend: epoll 2023-01-24T14:42:08.839 INFO:teuthology.orchestra.run.smithi179.stderr:INFO 2023-01-24 14:42:08,838 [shard 0] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-01-24T14:42:08.839 INFO:teuthology.orchestra.run.smithi179.stderr:INFO 2023-01-24 14:42:08,838 [shard 0] seastar - IO queue uses 0.75ms latency goal for device 0 2023-01-24T14:42:08.839 INFO:teuthology.orchestra.run.smithi179.stderr:INFO 2023-01-24 14:42:08,838 [shard 0] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-01-24T14:42:08.840 INFO:teuthology.orchestra.run.smithi179.stderr:INFO 2023-01-24 14:42:08,839 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-01-24T14:42:08.841 INFO:teuthology.orchestra.run.smithi179.stderr:==102138==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-01-24T14:42:08.847 INFO:teuthology.orchestra.run.smithi179.stderr:==102138==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7ffff316b000; bottom 0x7f786e57f000; size: 0x008784bec000 (582047678464) 2023-01-24T14:42:08.848 INFO:teuthology.orchestra.run.smithi179.stderr:False positive error reports may follow 2023-01-24T14:42:08.848 INFO:teuthology.orchestra.run.smithi179.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-01-24T14:42:08.882 INFO:teuthology.orchestra.run.smithi179.stderr:INFO 2023-01-24 14:42:08,881 [shard 0] osd - bypassing the config fetch due to --no-mon-config 2023-01-24T14:42:08.882 INFO:teuthology.orchestra.run.smithi179.stderr:INFO 2023-01-24 14:42:08,881 [shard 0] osd - get_early_config: set --smp 1 2023-01-24T14:42:09.378 INFO:teuthology.orchestra.run.smithi179.stderr:WARNING: debug mode. Not for benchmarking or production 2023-01-24T14:42:09.379 INFO:teuthology.orchestra.run.smithi179.stderr:INFO 2023-01-24 14:42:09,378 seastar - Reactor backend: epoll 2023-01-24T14:42:09.387 INFO:teuthology.orchestra.run.smithi179.stderr:INFO 2023-01-24 14:42:09,386 [shard 0] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-01-24T14:42:09.388 INFO:teuthology.orchestra.run.smithi179.stderr:INFO 2023-01-24 14:42:09,386 [shard 0] seastar - IO queue uses 0.75ms latency goal for device 0 2023-01-24T14:42:09.388 INFO:teuthology.orchestra.run.smithi179.stderr:INFO 2023-01-24 14:42:09,386 [shard 0] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-01-24T14:42:09.388 INFO:teuthology.orchestra.run.smithi179.stderr:INFO 2023-01-24 14:42:09,386 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-01-24T14:42:09.388 INFO:teuthology.orchestra.run.smithi179.stderr:==102136==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-01-24T14:42:09.394 INFO:teuthology.orchestra.run.smithi179.stderr:==102136==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7ffff316b000; bottom 0x7f786e578000; size: 0x008784bf3000 (582047707136) 2023-01-24T14:42:09.394 INFO:teuthology.orchestra.run.smithi179.stderr:False positive error reports may follow 2023-01-24T14:42:09.395 INFO:teuthology.orchestra.run.smithi179.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-01-24T14:42:12.306 INFO:teuthology.orchestra.run.smithi179.stdout:created object store /var/lib/ceph/osd/ceph-2 for osd.2 fsid 91764032-a053-4a42-a007-59362bb9c17e 2023-01-24T14:42:12.364 DEBUG:teuthology.orchestra.run.smithi179:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2023-01-24T14:42:12.391 INFO:tasks.ceph:Reading keys from all nodes... 2023-01-24T14:42:12.392 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2023-01-24T14:42:12.392 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2023-01-24T14:42:12.458 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2023-01-24T14:42:12.459 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2023-01-24T14:42:12.525 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2023-01-24T14:42:12.525 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2023-01-24T14:42:12.592 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2023-01-24T14:42:12.593 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2023-01-24T14:42:12.659 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2023-01-24T14:42:12.660 DEBUG:teuthology.orchestra.run.smithi179:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2023-01-24T14:42:12.718 INFO:tasks.ceph:Adding keys to all mons... 2023-01-24T14:42:12.718 DEBUG:teuthology.orchestra.run.smithi179:> sudo tee -a /etc/ceph/ceph.keyring 2023-01-24T14:42:12.783 INFO:teuthology.orchestra.run.smithi179.stdout:[mgr.x] 2023-01-24T14:42:12.783 INFO:teuthology.orchestra.run.smithi179.stdout: key = AQA17s9jEkuDABAAtV3Zgrfdnn5X2Ku3lpZkUA== 2023-01-24T14:42:12.783 INFO:teuthology.orchestra.run.smithi179.stdout:[osd.0] 2023-01-24T14:42:12.784 INFO:teuthology.orchestra.run.smithi179.stdout: key = AQA47s9jV4HAKxAAmXaqPDQBh+ZvcbTo2nqGkg== 2023-01-24T14:42:12.784 INFO:teuthology.orchestra.run.smithi179.stdout:[osd.1] 2023-01-24T14:42:12.784 INFO:teuthology.orchestra.run.smithi179.stdout: key = AQA97s9jbh3lAxAAv2QUR2H80y9bny7S5DqZ5A== 2023-01-24T14:42:12.784 INFO:teuthology.orchestra.run.smithi179.stdout:[osd.2] 2023-01-24T14:42:12.784 INFO:teuthology.orchestra.run.smithi179.stdout: key = AQBB7s9jyKdoGhAA3MtqPIeITkMYtPGlEG3KCg== 2023-01-24T14:42:12.785 INFO:teuthology.orchestra.run.smithi179.stdout:[client.0] 2023-01-24T14:42:12.785 INFO:teuthology.orchestra.run.smithi179.stdout: key = AQA17s9ju0TEAxAA2eH2w06/nrRpttDT2x01ZA== 2023-01-24T14:42:12.786 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-01-24T14:42:12.887 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-01-24T14:42:12.951 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-01-24T14:42:13.012 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-01-24T14:42:13.074 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-01-24T14:42:13.139 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-01-24T14:42:13.139 DEBUG:teuthology.orchestra.run.smithi179:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2023-01-24T14:42:13.167 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-01-24T14:42:13.282 DEBUG:teuthology.orchestra.run.smithi179:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2023-01-24T14:42:13.310 DEBUG:teuthology.orchestra.run.smithi179:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-01-24T14:42:13.367 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2023-01-24T14:42:13.368 INFO:tasks.ceph.mon.a:Restarting daemon 2023-01-24T14:42:13.368 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2023-01-24T14:42:13.410 INFO:tasks.ceph.mon.a:Started 2023-01-24T14:42:13.410 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2023-01-24T14:42:13.411 INFO:tasks.ceph.mgr.x:Restarting daemon 2023-01-24T14:42:13.411 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2023-01-24T14:42:13.413 INFO:tasks.ceph.mgr.x:Started 2023-01-24T14:42:13.413 INFO:tasks.ceph:Setting crush tunables to default 2023-01-24T14:42:13.414 DEBUG:teuthology.orchestra.run.smithi179:> sudo ceph --cluster ceph osd crush tunables default 2023-01-24T14:42:13.642 INFO:teuthology.orchestra.run.smithi179.stderr:adjusted tunables profile to default 2023-01-24T14:42:13.654 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2023-01-24T14:42:13.654 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2023-01-24T14:42:13.654 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2023-01-24T14:42:13.682 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2023-01-24T14:42:13.682 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2023-01-24T14:42:13.750 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2023-01-24T14:42:13.750 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2023-01-24T14:42:13.817 DEBUG:teuthology.orchestra.run.smithi179:> sudo ceph --cluster ceph osd new 3abca7f8-4a61-4e04-b5d4-315fb643d792 0 2023-01-24T14:42:14.078 INFO:teuthology.orchestra.run.smithi179.stdout:0 2023-01-24T14:42:14.091 DEBUG:teuthology.orchestra.run.smithi179:> sudo ceph --cluster ceph osd new a0e7d95f-3409-475d-a8d0-a6d00dbd9f99 1 2023-01-24T14:42:14.323 INFO:teuthology.orchestra.run.smithi179.stdout:1 2023-01-24T14:42:14.335 DEBUG:teuthology.orchestra.run.smithi179:> sudo ceph --cluster ceph osd new 79d1ce53-95fc-4407-96a5-cea3d9595b13 2 2023-01-24T14:42:14.556 INFO:teuthology.orchestra.run.smithi179.stdout:2 2023-01-24T14:42:14.570 INFO:tasks.ceph.osd.0:Restarting daemon 2023-01-24T14:42:14.570 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2023-01-24T14:42:14.572 INFO:tasks.ceph.osd.0:Started 2023-01-24T14:42:14.573 INFO:tasks.ceph.osd.1:Restarting daemon 2023-01-24T14:42:14.573 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2023-01-24T14:42:14.575 INFO:tasks.ceph.osd.1:Started 2023-01-24T14:42:14.576 INFO:tasks.ceph.osd.2:Restarting daemon 2023-01-24T14:42:14.576 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2023-01-24T14:42:14.578 INFO:tasks.ceph.osd.2:Started 2023-01-24T14:42:14.578 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-01-24T14:42:14.679 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-01-24T14:42:14.677+0000 7f19d9f3a1c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-01-24T14:42:14.814 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-01-24T14:42:14.812+0000 7f19d9f3a1c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-01-24T14:42:14.953 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-01-24T14:42:14.953 INFO:teuthology.orchestra.run.smithi179.stdout:{"epoch":5,"fsid":"91764032-a053-4a42-a007-59362bb9c17e","created":"2023-01-24T14:42:13.552129+0000","modified":"2023-01-24T14:42:14.554200+0000","last_up_change":"0.000000","last_in_change":"2023-01-24T14:42:14.554200+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","pools":[],"osds":[{"osd":0,"uuid":"3abca7f8-4a61-4e04-b5d4-315fb643d792","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"a0e7d95f-3409-475d-a8d0-a6d00dbd9f99","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"79d1ce53-95fc-4407-96a5-cea3d9595b13","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-01-24T14:42:14.982 INFO:tasks.ceph.ceph_manager.ceph:[] 2023-01-24T14:42:14.983 INFO:tasks.ceph:Waiting for OSDs to come up 2023-01-24T14:42:14.991 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-01-24T14:42:14.990+0000 7f19d9f3a1c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-01-24T14:42:15.236 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-01-24T14:42:15.235+0000 7f19d9f3a1c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-01-24T14:42:15.285 DEBUG:teuthology.orchestra.run.smithi179:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-01-24T14:42:15.528 INFO:tasks.ceph.osd.1.smithi179.stderr:WARNING: debug mode. Not for benchmarking or production 2023-01-24T14:42:15.534 INFO:tasks.ceph.osd.1.smithi179.stderr:INFO 2023-01-24 14:42:15,534 seastar - Reactor backend: epoll 2023-01-24T14:42:15.538 INFO:tasks.ceph.osd.0.smithi179.stderr:WARNING: debug mode. Not for benchmarking or production 2023-01-24T14:42:15.539 INFO:tasks.ceph.osd.0.smithi179.stderr:INFO 2023-01-24 14:42:15,538 seastar - Reactor backend: epoll 2023-01-24T14:42:15.559 INFO:tasks.ceph.osd.2.smithi179.stderr:WARNING: debug mode. Not for benchmarking or production 2023-01-24T14:42:15.565 INFO:tasks.ceph.osd.2.smithi179.stderr:INFO 2023-01-24 14:42:15,564 seastar - Reactor backend: epoll 2023-01-24T14:42:15.582 INFO:teuthology.misc.health.smithi179.stdout: 2023-01-24T14:42:15.582 INFO:teuthology.misc.health.smithi179.stdout:{"epoch":5,"fsid":"91764032-a053-4a42-a007-59362bb9c17e","created":"2023-01-24T14:42:13.552129+0000","modified":"2023-01-24T14:42:14.554200+0000","last_up_change":"0.000000","last_in_change":"2023-01-24T14:42:14.554200+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","pools":[],"osds":[{"osd":0,"uuid":"3abca7f8-4a61-4e04-b5d4-315fb643d792","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"a0e7d95f-3409-475d-a8d0-a6d00dbd9f99","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"79d1ce53-95fc-4407-96a5-cea3d9595b13","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-01-24T14:42:15.594 DEBUG:teuthology.misc:0 of 3 OSDs are up 2023-01-24T14:42:15.625 INFO:tasks.ceph.osd.0.smithi179.stderr:INFO 2023-01-24 14:42:15,624 [shard 0] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-01-24T14:42:15.626 INFO:tasks.ceph.osd.0.smithi179.stderr:INFO 2023-01-24 14:42:15,625 [shard 0] seastar - IO queue uses 0.75ms latency goal for device 0 2023-01-24T14:42:15.626 INFO:tasks.ceph.osd.0.smithi179.stderr:INFO 2023-01-24 14:42:15,625 [shard 0] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-01-24T14:42:15.626 INFO:tasks.ceph.osd.0.smithi179.stderr:INFO 2023-01-24 14:42:15,625 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-01-24T14:42:15.628 INFO:tasks.ceph.osd.1.smithi179.stderr:INFO 2023-01-24 14:42:15,627 [shard 0] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-01-24T14:42:15.628 INFO:tasks.ceph.osd.1.smithi179.stderr:INFO 2023-01-24 14:42:15,627 [shard 0] seastar - IO queue uses 0.75ms latency goal for device 0 2023-01-24T14:42:15.628 INFO:tasks.ceph.osd.1.smithi179.stderr:INFO 2023-01-24 14:42:15,627 [shard 0] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-01-24T14:42:15.629 INFO:tasks.ceph.osd.1.smithi179.stderr:INFO 2023-01-24 14:42:15,627 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-01-24T14:42:15.630 INFO:tasks.ceph.osd.1.smithi179.stderr:==103861==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-01-24T14:42:15.631 INFO:tasks.ceph.osd.0.smithi179.stderr:==103862==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-01-24T14:42:15.643 INFO:tasks.ceph.osd.1.smithi179.stderr:==103861==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7ffd0fb25000; bottom 0x7fc7236d5000; size: 0x0035ec450000 (231597211648) 2023-01-24T14:42:15.643 INFO:tasks.ceph.osd.1.smithi179.stderr:False positive error reports may follow 2023-01-24T14:42:15.644 INFO:tasks.ceph.osd.1.smithi179.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-01-24T14:42:15.644 INFO:tasks.ceph.osd.0.smithi179.stderr:==103862==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7ffffb17d000; bottom 0x7efe9970c000; size: 0x010161a71000 (1105444933632) 2023-01-24T14:42:15.644 INFO:tasks.ceph.osd.0.smithi179.stderr:False positive error reports may follow 2023-01-24T14:42:15.644 INFO:tasks.ceph.osd.0.smithi179.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-01-24T14:42:15.656 INFO:tasks.ceph.osd.2.smithi179.stderr:INFO 2023-01-24 14:42:15,656 [shard 0] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-01-24T14:42:15.657 INFO:tasks.ceph.osd.2.smithi179.stderr:INFO 2023-01-24 14:42:15,656 [shard 0] seastar - IO queue uses 0.75ms latency goal for device 0 2023-01-24T14:42:15.657 INFO:tasks.ceph.osd.2.smithi179.stderr:INFO 2023-01-24 14:42:15,656 [shard 0] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-01-24T14:42:15.657 INFO:tasks.ceph.osd.2.smithi179.stderr:INFO 2023-01-24 14:42:15,656 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-01-24T14:42:15.660 INFO:tasks.ceph.osd.2.smithi179.stderr:==103884==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-01-24T14:42:15.665 INFO:tasks.ceph.osd.2.smithi179.stderr:==103884==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7fffcd2a8000; bottom 0x7f833c87d000; size: 0x007c90a2b000 (535002525696) 2023-01-24T14:42:15.666 INFO:tasks.ceph.osd.2.smithi179.stderr:False positive error reports may follow 2023-01-24T14:42:15.666 INFO:tasks.ceph.osd.2.smithi179.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-01-24T14:42:15.747 INFO:tasks.ceph.osd.1.smithi179.stderr:INFO 2023-01-24 14:42:15,747 [shard 0] osd - populating config from monitor 2023-01-24T14:42:15.748 INFO:tasks.ceph.osd.2.smithi179.stderr:INFO 2023-01-24 14:42:15,747 [shard 0] osd - populating config from monitor 2023-01-24T14:42:15.749 INFO:tasks.ceph.osd.1.smithi179.stderr:INFO 2023-01-24 14:42:15,748 [shard 0] auth - AuthRegistry(0x7fc7236996b0) adding auth protocol: cephx 2023-01-24T14:42:15.749 INFO:tasks.ceph.osd.2.smithi179.stderr:INFO 2023-01-24 14:42:15,749 [shard 0] auth - AuthRegistry(0x7f833c8416b0) adding auth protocol: cephx 2023-01-24T14:42:15.750 INFO:tasks.ceph.osd.2.smithi179.stderr:INFO 2023-01-24 14:42:15,749 [shard 0] auth - AuthRegistry(0x7f833c8416b0) adding auth protocol: cephx 2023-01-24T14:42:15.751 INFO:tasks.ceph.osd.0.smithi179.stderr:INFO 2023-01-24 14:42:15,749 [shard 0] osd - populating config from monitor 2023-01-24T14:42:15.751 INFO:tasks.ceph.osd.1.smithi179.stderr:INFO 2023-01-24 14:42:15,749 [shard 0] auth - AuthRegistry(0x7fc7236996b0) adding auth protocol: cephx 2023-01-24T14:42:15.752 INFO:tasks.ceph.osd.2.smithi179.stderr:INFO 2023-01-24 14:42:15,749 [shard 0] auth - AuthRegistry(0x7f833c8416b0) adding auth protocol: cephx 2023-01-24T14:42:15.752 INFO:tasks.ceph.osd.2.smithi179.stderr:INFO 2023-01-24 14:42:15,749 [shard 0] auth - AuthRegistry(0x7f833c8416b0) adding con mode: crc 2023-01-24T14:42:15.753 INFO:tasks.ceph.osd.2.smithi179.stderr:INFO 2023-01-24 14:42:15,749 [shard 0] auth - AuthRegistry(0x7f833c8416b0) adding con mode: crc 2023-01-24T14:42:15.753 INFO:tasks.ceph.osd.2.smithi179.stderr:INFO 2023-01-24 14:42:15,749 [shard 0] auth - AuthRegistry(0x7f833c8416b0) adding con mode: crc 2023-01-24T14:42:15.753 INFO:tasks.ceph.osd.2.smithi179.stderr:INFO 2023-01-24 14:42:15,749 [shard 0] auth - AuthRegistry(0x7f833c8416b0) adding con mode: crc 2023-01-24T14:42:15.753 INFO:tasks.ceph.osd.2.smithi179.stderr:INFO 2023-01-24 14:42:15,749 [shard 0] auth - AuthRegistry(0x7f833c8416b0) adding con mode: crc 2023-01-24T14:42:15.753 INFO:tasks.ceph.osd.2.smithi179.stderr:INFO 2023-01-24 14:42:15,749 [shard 0] auth - AuthRegistry(0x7f833c8416b0) adding con mode: crc 2023-01-24T14:42:15.754 INFO:tasks.ceph.osd.1.smithi179.stderr:INFO 2023-01-24 14:42:15,749 [shard 0] auth - AuthRegistry(0x7fc7236996b0) adding auth protocol: cephx 2023-01-24T14:42:15.755 INFO:tasks.ceph.osd.1.smithi179.stderr:INFO 2023-01-24 14:42:15,749 [shard 0] auth - AuthRegistry(0x7fc7236996b0) adding con mode: crc 2023-01-24T14:42:15.755 INFO:tasks.ceph.osd.1.smithi179.stderr:INFO 2023-01-24 14:42:15,749 [shard 0] auth - AuthRegistry(0x7fc7236996b0) adding con mode: crc 2023-01-24T14:42:15.755 INFO:tasks.ceph.osd.1.smithi179.stderr:INFO 2023-01-24 14:42:15,749 [shard 0] auth - AuthRegistry(0x7fc7236996b0) adding con mode: crc 2023-01-24T14:42:15.755 INFO:tasks.ceph.osd.1.smithi179.stderr:INFO 2023-01-24 14:42:15,749 [shard 0] auth - AuthRegistry(0x7fc7236996b0) adding con mode: crc 2023-01-24T14:42:15.755 INFO:tasks.ceph.osd.1.smithi179.stderr:INFO 2023-01-24 14:42:15,749 [shard 0] auth - AuthRegistry(0x7fc7236996b0) adding con mode: crc 2023-01-24T14:42:15.756 INFO:tasks.ceph.osd.1.smithi179.stderr:INFO 2023-01-24 14:42:15,749 [shard 0] auth - AuthRegistry(0x7fc7236996b0) adding con mode: crc 2023-01-24T14:42:15.756 INFO:tasks.ceph.osd.1.smithi179.stderr:INFO 2023-01-24 14:42:15,750 [shard 0] auth - auth: KeyRing::load: loaded key file /var/lib/ceph/osd/ceph-1/keyring 2023-01-24T14:42:15.757 INFO:tasks.ceph.osd.0.smithi179.stderr:INFO 2023-01-24 14:42:15,750 [shard 0] auth - AuthRegistry(0x7efe996d06b0) adding auth protocol: cephx 2023-01-24T14:42:15.757 INFO:tasks.ceph.osd.0.smithi179.stderr:INFO 2023-01-24 14:42:15,750 [shard 0] auth - AuthRegistry(0x7efe996d06b0) adding auth protocol: cephx 2023-01-24T14:42:15.757 INFO:tasks.ceph.osd.0.smithi179.stderr:INFO 2023-01-24 14:42:15,750 [shard 0] auth - AuthRegistry(0x7efe996d06b0) adding auth protocol: cephx 2023-01-24T14:42:15.757 INFO:tasks.ceph.osd.0.smithi179.stderr:INFO 2023-01-24 14:42:15,750 [shard 0] auth - AuthRegistry(0x7efe996d06b0) adding con mode: crc 2023-01-24T14:42:15.758 INFO:tasks.ceph.osd.0.smithi179.stderr:INFO 2023-01-24 14:42:15,750 [shard 0] auth - AuthRegistry(0x7efe996d06b0) adding con mode: crc 2023-01-24T14:42:15.758 INFO:tasks.ceph.osd.0.smithi179.stderr:INFO 2023-01-24 14:42:15,750 [shard 0] auth - AuthRegistry(0x7efe996d06b0) adding con mode: crc 2023-01-24T14:42:15.758 INFO:tasks.ceph.osd.2.smithi179.stderr:INFO 2023-01-24 14:42:15,750 [shard 0] auth - auth: KeyRing::load: loaded key file /var/lib/ceph/osd/ceph-2/keyring 2023-01-24T14:42:15.759 INFO:tasks.ceph.osd.0.smithi179.stderr:INFO 2023-01-24 14:42:15,751 [shard 0] auth - AuthRegistry(0x7efe996d06b0) adding con mode: crc 2023-01-24T14:42:15.759 INFO:tasks.ceph.osd.0.smithi179.stderr:INFO 2023-01-24 14:42:15,751 [shard 0] auth - AuthRegistry(0x7efe996d06b0) adding con mode: crc 2023-01-24T14:42:15.759 INFO:tasks.ceph.osd.0.smithi179.stderr:INFO 2023-01-24 14:42:15,751 [shard 0] auth - AuthRegistry(0x7efe996d06b0) adding con mode: crc 2023-01-24T14:42:15.759 INFO:tasks.ceph.osd.0.smithi179.stderr:INFO 2023-01-24 14:42:15,752 [shard 0] auth - auth: KeyRing::load: loaded key file /var/lib/ceph/osd/ceph-0/keyring 2023-01-24T14:42:15.760 INFO:tasks.ceph.osd.2.smithi179.stderr:DEBUG 2023-01-24 14:42:15,756 [shard 0] monc - build_monmap: for_mkfs=false 2023-01-24T14:42:15.760 INFO:tasks.ceph.osd.1.smithi179.stderr:DEBUG 2023-01-24 14:42:15,756 [shard 0] monc - build_monmap: for_mkfs=false 2023-01-24T14:42:15.761 INFO:tasks.ceph.osd.0.smithi179.stderr:DEBUG 2023-01-24 14:42:15,757 [shard 0] monc - build_monmap: for_mkfs=false 2023-01-24T14:42:15.761 INFO:tasks.ceph.osd.2.smithi179.stderr:INFO 2023-01-24 14:42:15,757 [shard 0] monc - reopen_session to mon.-1 2023-01-24T14:42:15.762 INFO:tasks.ceph.osd.2.smithi179.stderr:INFO 2023-01-24 14:42:15,757 [shard 0] monc - connecting to mon.1 2023-01-24T14:42:15.762 INFO:tasks.ceph.osd.2.smithi179.stderr:INFO 2023-01-24 14:42:15,757 [shard 0] ms - [0x614000008040 client.?(temp_mon_client) - >> mon.? v2:172.21.15.179:3300/0] ProtocolV2::start_connect(): peer_addr=v2:172.21.15.179:3300/0, peer_name=mon.?, cc=15074985079572129387 policy(lossy=false, server=false, standby=false, resetcheck=true) 2023-01-24T14:42:15.762 INFO:tasks.ceph.osd.2.smithi179.stderr:INFO 2023-01-24 14:42:15,757 [shard 0] ms - [0x614000008040 client.?(temp_mon_client) - >> mon.? v2:172.21.15.179:3300/0] write_event: delay ... 2023-01-24T14:42:15.762 INFO:tasks.ceph.osd.2.smithi179.stderr:WARN 2023-01-24 14:42:15,757 [shard 0] monc - mon.0 does not have an addr compatible with me 2023-01-24T14:42:15.763 INFO:tasks.ceph.osd.1.smithi179.stderr:INFO 2023-01-24 14:42:15,758 [shard 0] monc - reopen_session to mon.-1 2023-01-24T14:42:15.763 INFO:tasks.ceph.osd.1.smithi179.stderr:INFO 2023-01-24 14:42:15,758 [shard 0] monc - connecting to mon.1 2023-01-24T14:42:15.763 INFO:tasks.ceph.osd.1.smithi179.stderr:INFO 2023-01-24 14:42:15,758 [shard 0] ms - [0x614000008040 client.?(temp_mon_client) - >> mon.? v2:172.21.15.179:3300/0] ProtocolV2::start_connect(): peer_addr=v2:172.21.15.179:3300/0, peer_name=mon.?, cc=511758048915593288 policy(lossy=false, server=false, standby=false, resetcheck=true) 2023-01-24T14:42:15.764 INFO:tasks.ceph.osd.1.smithi179.stderr:INFO 2023-01-24 14:42:15,758 [shard 0] ms - [0x614000008040 client.?(temp_mon_client) - >> mon.? v2:172.21.15.179:3300/0] write_event: delay ... 2023-01-24T14:42:15.764 INFO:tasks.ceph.osd.1.smithi179.stderr:WARN 2023-01-24 14:42:15,758 [shard 0] monc - mon.0 does not have an addr compatible with me 2023-01-24T14:42:15.765 INFO:tasks.ceph.osd.0.smithi179.stderr:INFO 2023-01-24 14:42:15,758 [shard 0] monc - reopen_session to mon.-1 2023-01-24T14:42:15.765 INFO:tasks.ceph.osd.0.smithi179.stderr:WARN 2023-01-24 14:42:15,758 [shard 0] monc - mon.0 does not have an addr compatible with me 2023-01-24T14:42:15.765 INFO:tasks.ceph.osd.0.smithi179.stderr:INFO 2023-01-24 14:42:15,758 [shard 0] monc - connecting to mon.1 2023-01-24T14:42:15.765 INFO:tasks.ceph.osd.0.smithi179.stderr:INFO 2023-01-24 14:42:15,758 [shard 0] ms - [0x614000008040 client.?(temp_mon_client) - >> mon.? v2:172.21.15.179:3300/0] ProtocolV2::start_connect(): peer_addr=v2:172.21.15.179:3300/0, peer_name=mon.?, cc=9386720941557220331 policy(lossy=false, server=false, standby=false, resetcheck=true) 2023-01-24T14:42:15.766 INFO:tasks.ceph.osd.0.smithi179.stderr:INFO 2023-01-24 14:42:15,758 [shard 0] ms - [0x614000008040 client.?(temp_mon_client) - >> mon.? v2:172.21.15.179:3300/0] write_event: delay ... 2023-01-24T14:42:15.766 INFO:tasks.ceph.osd.2.smithi179.stderr:INFO 2023-01-24 14:42:15,760 [shard 0] ms - [0x614000008040 client.?(temp_mon_client) -@64504 >> mon.? v2:172.21.15.179:3300/0] peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.179:64504/0 (socket says 172.21.15.179:64504) 2023-01-24T14:42:15.767 INFO:tasks.ceph.osd.1.smithi179.stderr:INFO 2023-01-24 14:42:15,761 [shard 0] ms - [0x614000008040 client.?(temp_mon_client) -@59662 >> mon.? v2:172.21.15.179:3300/0] peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.179:59662/0 (socket says 172.21.15.179:59662) 2023-01-24T14:42:15.767 INFO:tasks.ceph.osd.1.smithi179.stderr:INFO 2023-01-24 14:42:15,761 [shard 0] ms - [0x614000008040 client.?(temp_mon_client) 172.21.15.179:0/103861@59662 >> mon.? v2:172.21.15.179:3300/0] learned myaddr=172.21.15.179:0/103861 (unbound) 2023-01-24T14:42:15.767 INFO:tasks.ceph.osd.1.smithi179.stderr:INFO 2023-01-24 14:42:15,761 [shard 0] monc - get_auth_request(conn=[0x614000008040 client.?(temp_mon_client) 172.21.15.179:0/103861@59662 >> mon.? v2:172.21.15.179:3300/0], auth_method=0) 2023-01-24T14:42:15.767 INFO:tasks.ceph.osd.1.smithi179.stderr:INFO 2023-01-24 14:42:15,761 [shard 0] monc - method 2 preferred_modes [1] 2023-01-24T14:42:15.768 INFO:tasks.ceph.osd.0.smithi179.stderr:INFO 2023-01-24 14:42:15,761 [shard 0] ms - [0x614000008040 client.?(temp_mon_client) -@61729 >> mon.? v2:172.21.15.179:3300/0] peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.179:61729/0 (socket says 172.21.15.179:61729) 2023-01-24T14:42:15.768 INFO:tasks.ceph.osd.0.smithi179.stderr:INFO 2023-01-24 14:42:15,761 [shard 0] ms - [0x614000008040 client.?(temp_mon_client) 172.21.15.179:0/103862@61729 >> mon.? v2:172.21.15.179:3300/0] learned myaddr=172.21.15.179:0/103862 (unbound) 2023-01-24T14:42:15.769 INFO:tasks.ceph.osd.0.smithi179.stderr:INFO 2023-01-24 14:42:15,761 [shard 0] monc - get_auth_request(conn=[0x614000008040 client.?(temp_mon_client) 172.21.15.179:0/103862@61729 >> mon.? v2:172.21.15.179:3300/0], auth_method=0) 2023-01-24T14:42:15.769 INFO:tasks.ceph.osd.0.smithi179.stderr:INFO 2023-01-24 14:42:15,761 [shard 0] monc - method 2 preferred_modes [1] 2023-01-24T14:42:15.769 INFO:tasks.ceph.osd.2.smithi179.stderr:INFO 2023-01-24 14:42:15,761 [shard 0] ms - [0x614000008040 client.?(temp_mon_client) 172.21.15.179:0/103884@64504 >> mon.? v2:172.21.15.179:3300/0] learned myaddr=172.21.15.179:0/103884 (unbound) 2023-01-24T14:42:15.770 INFO:tasks.ceph.osd.2.smithi179.stderr:INFO 2023-01-24 14:42:15,761 [shard 0] monc - get_auth_request(conn=[0x614000008040 client.?(temp_mon_client) 172.21.15.179:0/103884@64504 >> mon.? v2:172.21.15.179:3300/0], auth_method=0) 2023-01-24T14:42:15.770 INFO:tasks.ceph.osd.2.smithi179.stderr:INFO 2023-01-24 14:42:15,762 [shard 0] monc - method 2 preferred_modes [1] 2023-01-24T14:42:15.802 INFO:tasks.ceph.osd.2.smithi179.stderr:INFO 2023-01-24 14:42:15,801 [shard 0] monc - responding with 36 bytes 2023-01-24T14:42:15.802 INFO:tasks.ceph.osd.1.smithi179.stderr:INFO 2023-01-24 14:42:15,802 [shard 0] monc - responding with 36 bytes 2023-01-24T14:42:15.803 INFO:tasks.ceph.osd.0.smithi179.stderr:INFO 2023-01-24 14:42:15,802 [shard 0] monc - responding with 36 bytes 2023-01-24T14:42:15.803 INFO:tasks.ceph.osd.0.smithi179.stderr:INFO 2023-01-24 14:42:15,803 [shard 0] monc - found mon.noname-a 2023-01-24T14:42:15.804 INFO:tasks.ceph.osd.0.smithi179.stderr:DEBUG 2023-01-24 14:42:15,803 [shard 0] monc - reopen_session mon connection attempts complete 2023-01-24T14:42:15.804 INFO:tasks.ceph.osd.0.smithi179.stderr:INFO 2023-01-24 14:42:15,803 [shard 0] monc - renew_rotating_keyring renewing rotating keys (they expired before 1674571305.8034706) 2023-01-24T14:42:15.805 INFO:tasks.ceph.osd.1.smithi179.stderr:INFO 2023-01-24 14:42:15,803 [shard 0] monc - found mon.noname-a 2023-01-24T14:42:15.805 INFO:tasks.ceph.osd.1.smithi179.stderr:DEBUG 2023-01-24 14:42:15,803 [shard 0] monc - reopen_session mon connection attempts complete 2023-01-24T14:42:15.805 INFO:tasks.ceph.osd.1.smithi179.stderr:INFO 2023-01-24 14:42:15,803 [shard 0] monc - renew_rotating_keyring renewing rotating keys (they expired before 1674571305.8039522) 2023-01-24T14:42:15.805 INFO:tasks.ceph.osd.1.smithi179.stderr:INFO 2023-01-24 14:42:15,803 [shard 0] monc - sending auth(proto 2 2 bytes epoch 0) v1 2023-01-24T14:42:15.806 INFO:tasks.ceph.osd.2.smithi179.stderr:INFO 2023-01-24 14:42:15,804 [shard 0] monc - found mon.noname-a 2023-01-24T14:42:15.806 INFO:tasks.ceph.osd.2.smithi179.stderr:DEBUG 2023-01-24 14:42:15,804 [shard 0] monc - reopen_session mon connection attempts complete 2023-01-24T14:42:15.806 INFO:tasks.ceph.osd.2.smithi179.stderr:INFO 2023-01-24 14:42:15,804 [shard 0] monc - renew_rotating_keyring renewing rotating keys (they expired before 1674571305.8044481) 2023-01-24T14:42:15.807 INFO:tasks.ceph.osd.2.smithi179.stderr:INFO 2023-01-24 14:42:15,804 [shard 0] monc - sending auth(proto 2 2 bytes epoch 0) v1 2023-01-24T14:42:15.807 INFO:tasks.ceph.osd.0.smithi179.stderr:INFO 2023-01-24 14:42:15,803 [shard 0] monc - sending auth(proto 2 2 bytes epoch 0) v1 2023-01-24T14:42:15.807 INFO:tasks.ceph.osd.0.smithi179.stderr:INFO 2023-01-24 14:42:15,804 [shard 0] monc - waiting 2023-01-24T14:42:15.808 INFO:tasks.ceph.osd.1.smithi179.stderr:INFO 2023-01-24 14:42:15,805 [shard 0] monc - waiting 2023-01-24T14:42:15.808 INFO:tasks.ceph.osd.2.smithi179.stderr:INFO 2023-01-24 14:42:15,805 [shard 0] monc - waiting 2023-01-24T14:42:15.845 INFO:tasks.ceph.osd.0.smithi179.stderr:WARN 2023-01-24 14:42:15,844 [shard 0] ms - [0x614000008040 client.?(temp_mon_client) 172.21.15.179:0/103862@61729 >> mon.0 v2:172.21.15.179:3300/0] UPDATE Policy(lossy=true) from server flags 2023-01-24T14:42:15.845 INFO:tasks.ceph.osd.0.smithi179.stderr:INFO 2023-01-24 14:42:15,844 [shard 0] ms - [0x614000008040 client.?(temp_mon_client) 172.21.15.179:0/103862@61729 >> mon.0 v2:172.21.15.179:3300/0] connected: gs=1, pgs=23, cs=0, client_cookie=9386720941557220331, server_cookie=0, in_seq=0, out_seq=0, out_q=2 2023-01-24T14:42:15.846 INFO:tasks.ceph.osd.1.smithi179.stderr:WARN 2023-01-24 14:42:15,845 [shard 0] ms - [0x614000008040 client.?(temp_mon_client) 172.21.15.179:0/103861@59662 >> mon.0 v2:172.21.15.179:3300/0] UPDATE Policy(lossy=true) from server flags 2023-01-24T14:42:15.846 INFO:tasks.ceph.osd.1.smithi179.stderr:INFO 2023-01-24 14:42:15,845 [shard 0] ms - [0x614000008040 client.?(temp_mon_client) 172.21.15.179:0/103861@59662 >> mon.0 v2:172.21.15.179:3300/0] connected: gs=1, pgs=22, cs=0, client_cookie=511758048915593288, server_cookie=0, in_seq=0, out_seq=0, out_q=2 2023-01-24T14:42:15.846 INFO:tasks.ceph.osd.2.smithi179.stderr:WARN 2023-01-24 14:42:15,845 [shard 0] ms - [0x614000008040 client.?(temp_mon_client) 172.21.15.179:0/103884@64504 >> mon.0 v2:172.21.15.179:3300/0] UPDATE Policy(lossy=true) from server flags 2023-01-24T14:42:15.847 INFO:tasks.ceph.osd.2.smithi179.stderr:INFO 2023-01-24 14:42:15,845 [shard 0] ms - [0x614000008040 client.?(temp_mon_client) 172.21.15.179:0/103884@64504 >> mon.0 v2:172.21.15.179:3300/0] connected: gs=1, pgs=24, cs=0, client_cookie=15074985079572129387, server_cookie=0, in_seq=0, out_seq=0, out_q=2 2023-01-24T14:42:15.847 INFO:tasks.ceph.osd.1.smithi179.stderr:INFO 2023-01-24 14:42:15,846 [shard 0] monc - got monmap 1, mon.a, is now rank 0 2023-01-24T14:42:15.848 INFO:tasks.ceph.osd.1.smithi179.stderr:INFO 2023-01-24 14:42:15,846 [shard 0] monc - handle_monmap: renewing tickets 2023-01-24T14:42:15.848 INFO:tasks.ceph.osd.1.smithi179.stderr:INFO 2023-01-24 14:42:15,846 [shard 0] monc - renew_rotating_keyring renewing rotating keys (they expired before 1674571305.846243) 2023-01-24T14:42:15.848 INFO:tasks.ceph.osd.1.smithi179.stderr:INFO 2023-01-24 14:42:15,846 [shard 0] monc - renew_rotating_keyring called too often (last: 1674571335.8039522) 2023-01-24T14:42:15.848 INFO:tasks.ceph.osd.1.smithi179.stderr:DEBUG 2023-01-24 14:42:15,846 [shard 0] monc - renew_tickets: don't need new tickets 2023-01-24T14:42:15.849 INFO:tasks.ceph.osd.1.smithi179.stderr:INFO 2023-01-24 14:42:15,846 [shard 0] monc - handle_mon_map: renewed tickets 2023-01-24T14:42:15.849 INFO:tasks.ceph.osd.2.smithi179.stderr:INFO 2023-01-24 14:42:15,846 [shard 0] monc - got monmap 1, mon.a, is now rank 0 2023-01-24T14:42:15.850 INFO:tasks.ceph.osd.2.smithi179.stderr:INFO 2023-01-24 14:42:15,846 [shard 0] monc - handle_monmap: renewing tickets 2023-01-24T14:42:15.850 INFO:tasks.ceph.osd.2.smithi179.stderr:INFO 2023-01-24 14:42:15,846 [shard 0] monc - renew_rotating_keyring renewing rotating keys (they expired before 1674571305.846684) 2023-01-24T14:42:15.850 INFO:tasks.ceph.osd.2.smithi179.stderr:INFO 2023-01-24 14:42:15,846 [shard 0] monc - renew_rotating_keyring called too often (last: 1674571335.8044481) 2023-01-24T14:42:15.850 INFO:tasks.ceph.osd.2.smithi179.stderr:DEBUG 2023-01-24 14:42:15,846 [shard 0] monc - renew_tickets: don't need new tickets 2023-01-24T14:42:15.851 INFO:tasks.ceph.osd.2.smithi179.stderr:INFO 2023-01-24 14:42:15,846 [shard 0] monc - handle_mon_map: renewed tickets 2023-01-24T14:42:15.851 INFO:tasks.ceph.osd.0.smithi179.stderr:INFO 2023-01-24 14:42:15,847 [shard 0] monc - got monmap 1, mon.a, is now rank 0 2023-01-24T14:42:15.852 INFO:tasks.ceph.osd.0.smithi179.stderr:INFO 2023-01-24 14:42:15,847 [shard 0] monc - handle_monmap: renewing tickets 2023-01-24T14:42:15.852 INFO:tasks.ceph.osd.0.smithi179.stderr:INFO 2023-01-24 14:42:15,847 [shard 0] monc - renew_rotating_keyring renewing rotating keys (they expired before 1674571305.847268) 2023-01-24T14:42:15.852 INFO:tasks.ceph.osd.0.smithi179.stderr:INFO 2023-01-24 14:42:15,847 [shard 0] monc - renew_rotating_keyring called too often (last: 1674571335.8034706) 2023-01-24T14:42:15.852 INFO:tasks.ceph.osd.0.smithi179.stderr:DEBUG 2023-01-24 14:42:15,847 [shard 0] monc - renew_tickets: don't need new tickets 2023-01-24T14:42:15.852 INFO:tasks.ceph.osd.0.smithi179.stderr:INFO 2023-01-24 14:42:15,847 [shard 0] monc - handle_mon_map: renewed tickets 2023-01-24T14:42:15.853 INFO:tasks.ceph.osd.1.smithi179.stderr:INFO 2023-01-24 14:42:15,847 [shard 0] monc - handle_auth_reply [0x614000008040 client.?(temp_mon_client) 172.21.15.179:0/103861@59662 >> mon.0 v2:172.21.15.179:3300/0] returns auth_reply(proto 2 0 (0) Success) v1: 0 2023-01-24T14:42:15.854 INFO:tasks.ceph.osd.1.smithi179.stderr:INFO 2023-01-24 14:42:15,847 [shard 0] monc - handle_auth_reply 2023-01-24T14:42:15.854 INFO:tasks.ceph.osd.1.smithi179.stderr:DEBUG 2023-01-24 14:42:15,847 [shard 0] monc - renew_tickets: don't need new tickets 2023-01-24T14:42:15.854 INFO:tasks.ceph.osd.1.smithi179.stderr:INFO 2023-01-24 14:42:15,847 [shard 0] monc - renew_rotating_keyring renewing rotating keys (they expired before 1674571305.84778) 2023-01-24T14:42:15.854 INFO:tasks.ceph.osd.1.smithi179.stderr:INFO 2023-01-24 14:42:15,847 [shard 0] monc - renew_rotating_keyring called too often (last: 1674571335.8039522) 2023-01-24T14:42:15.854 INFO:tasks.ceph.osd.1.smithi179.stderr:INFO 2023-01-24 14:42:15,847 [shard 0] monc - do_auth_single: [0x614000008040 client.?(temp_mon_client) 172.21.15.179:0/103861@59662 >> mon.0 v2:172.21.15.179:3300/0] returns auth_reply(proto 2 0 (0) Success) v1: 0 2023-01-24T14:42:15.855 INFO:tasks.ceph.osd.0.smithi179.stderr:INFO 2023-01-24 14:42:15,848 [shard 0] monc - handle_auth_reply [0x614000008040 client.?(temp_mon_client) 172.21.15.179:0/103862@61729 >> mon.0 v2:172.21.15.179:3300/0] returns auth_reply(proto 2 0 (0) Success) v1: 0 2023-01-24T14:42:15.856 INFO:tasks.ceph.osd.0.smithi179.stderr:INFO 2023-01-24 14:42:15,848 [shard 0] monc - handle_auth_reply 2023-01-24T14:42:15.856 INFO:tasks.ceph.osd.0.smithi179.stderr:DEBUG 2023-01-24 14:42:15,848 [shard 0] monc - renew_tickets: don't need new tickets 2023-01-24T14:42:15.856 INFO:tasks.ceph.osd.0.smithi179.stderr:INFO 2023-01-24 14:42:15,848 [shard 0] monc - renew_rotating_keyring renewing rotating keys (they expired before 1674571305.8482478) 2023-01-24T14:42:15.856 INFO:tasks.ceph.osd.0.smithi179.stderr:INFO 2023-01-24 14:42:15,848 [shard 0] monc - renew_rotating_keyring called too often (last: 1674571335.8034706) 2023-01-24T14:42:15.856 INFO:tasks.ceph.osd.0.smithi179.stderr:INFO 2023-01-24 14:42:15,848 [shard 0] monc - do_auth_single: [0x614000008040 client.?(temp_mon_client) 172.21.15.179:0/103862@61729 >> mon.0 v2:172.21.15.179:3300/0] returns auth_reply(proto 2 0 (0) Success) v1: 0 2023-01-24T14:42:15.857 INFO:tasks.ceph.osd.2.smithi179.stderr:INFO 2023-01-24 14:42:15,848 [shard 0] monc - handle_auth_reply [0x614000008040 client.?(temp_mon_client) 172.21.15.179:0/103884@64504 >> mon.0 v2:172.21.15.179:3300/0] returns auth_reply(proto 2 0 (0) Success) v1: 0 2023-01-24T14:42:15.858 INFO:tasks.ceph.osd.2.smithi179.stderr:INFO 2023-01-24 14:42:15,848 [shard 0] monc - handle_auth_reply 2023-01-24T14:42:15.858 INFO:tasks.ceph.osd.2.smithi179.stderr:INFO 2023-01-24 14:42:15,848 [shard 0] monc - do_auth_single: [0x614000008040 client.?(temp_mon_client) 172.21.15.179:0/103884@64504 >> mon.0 v2:172.21.15.179:3300/0] returns auth_reply(proto 2 0 (0) Success) v1: 0 2023-01-24T14:42:15.858 INFO:tasks.ceph.osd.2.smithi179.stderr:DEBUG 2023-01-24 14:42:15,848 [shard 0] monc - renew_tickets: don't need new tickets 2023-01-24T14:42:15.858 INFO:tasks.ceph.osd.2.smithi179.stderr:DEBUG 2023-01-24 14:42:15,848 [shard 0] monc - renew_rotating_keyring secrets are up-to-date (they expire after 1674571305.8486874) 2023-01-24T14:42:15.858 INFO:tasks.ceph.osd.2.smithi179.stderr:INFO 2023-01-24 14:42:15,850 [shard 0] monc - set_mon_vals no callback set 2023-01-24T14:42:15.859 INFO:tasks.ceph.osd.1.smithi179.stderr:INFO 2023-01-24 14:42:15,850 [shard 0] monc - set_mon_vals no callback set 2023-01-24T14:42:15.859 INFO:tasks.ceph.osd.0.smithi179.stderr:INFO 2023-01-24 14:42:15,851 [shard 0] monc - set_mon_vals no callback set 2023-01-24T14:42:15.860 INFO:tasks.ceph.osd.1.smithi179.stderr:DEBUG 2023-01-24 14:42:15,859 [shard 0] osd - operator(): got config from monitor, fsid 91764032-a053-4a42-a007-59362bb9c17e 2023-01-24T14:42:15.860 INFO:tasks.ceph.osd.1.smithi179.stderr:INFO 2023-01-24 14:42:15,859 [shard 0] monc - stop 2023-01-24T14:42:15.860 INFO:tasks.ceph.osd.1.smithi179.stderr:INFO 2023-01-24 14:42:15,859 [shard 0] monc - close 2023-01-24T14:42:15.860 INFO:tasks.ceph.osd.1.smithi179.stderr:INFO 2023-01-24 14:42:15,859 [shard 0] ms - [0x614000008040 client.?(temp_mon_client) 172.21.15.179:0/103861@59662 >> mon.0 v2:172.21.15.179:3300/0] closing: reset no, replace no 2023-01-24T14:42:15.861 INFO:tasks.ceph.osd.2.smithi179.stderr:DEBUG 2023-01-24 14:42:15,860 [shard 0] osd - operator(): got config from monitor, fsid 91764032-a053-4a42-a007-59362bb9c17e 2023-01-24T14:42:15.861 INFO:tasks.ceph.osd.2.smithi179.stderr:INFO 2023-01-24 14:42:15,860 [shard 0] monc - stop 2023-01-24T14:42:15.862 INFO:tasks.ceph.osd.2.smithi179.stderr:INFO 2023-01-24 14:42:15,860 [shard 0] monc - close 2023-01-24T14:42:15.862 INFO:tasks.ceph.osd.2.smithi179.stderr:INFO 2023-01-24 14:42:15,860 [shard 0] ms - [0x614000008040 client.?(temp_mon_client) 172.21.15.179:0/103884@64504 >> mon.0 v2:172.21.15.179:3300/0] closing: reset no, replace no 2023-01-24T14:42:15.862 INFO:tasks.ceph.osd.2.smithi179.stderr:INFO 2023-01-24 14:42:15,860 [shard 0] ms - [0x614000008040 client.?(temp_mon_client) 172.21.15.179:0/103884@64504 >> mon.0 v2:172.21.15.179:3300/0] execute_ready(): protocol aborted at CLOSING -- std::system_error (error crimson::net:4, read eof) 2023-01-24T14:42:15.863 INFO:tasks.ceph.osd.0.smithi179.stderr:DEBUG 2023-01-24 14:42:15,860 [shard 0] osd - operator(): got config from monitor, fsid 91764032-a053-4a42-a007-59362bb9c17e 2023-01-24T14:42:15.863 INFO:tasks.ceph.osd.0.smithi179.stderr:INFO 2023-01-24 14:42:15,860 [shard 0] monc - stop 2023-01-24T14:42:15.863 INFO:tasks.ceph.osd.0.smithi179.stderr:INFO 2023-01-24 14:42:15,860 [shard 0] monc - close 2023-01-24T14:42:15.863 INFO:tasks.ceph.osd.0.smithi179.stderr:INFO 2023-01-24 14:42:15,860 [shard 0] ms - [0x614000008040 client.?(temp_mon_client) 172.21.15.179:0/103862@61729 >> mon.0 v2:172.21.15.179:3300/0] closing: reset no, replace no 2023-01-24T14:42:15.864 INFO:tasks.ceph.osd.1.smithi179.stderr:INFO 2023-01-24 14:42:15,861 [shard 0] ms - [0x614000008040 client.?(temp_mon_client) 172.21.15.179:0/103861@59662 >> mon.0 v2:172.21.15.179:3300/0] execute_ready(): protocol aborted at CLOSING -- std::system_error (error crimson::net:4, read eof) 2023-01-24T14:42:15.864 INFO:tasks.ceph.osd.1.smithi179.stderr:INFO 2023-01-24 14:42:15,861 [shard 0] osd - get_early_config: set --smp 1 2023-01-24T14:42:15.864 INFO:tasks.ceph.osd.2.smithi179.stderr:INFO 2023-01-24 14:42:15,861 [shard 0] osd - get_early_config: set --smp 1 2023-01-24T14:42:15.865 INFO:tasks.ceph.osd.0.smithi179.stderr:INFO 2023-01-24 14:42:15,860 [shard 0] ms - [0x614000008040 client.?(temp_mon_client) 172.21.15.179:0/103862@61729 >> mon.0 v2:172.21.15.179:3300/0] execute_ready(): protocol aborted at CLOSING -- std::system_error (error crimson::net:4, read eof) 2023-01-24T14:42:15.865 INFO:tasks.ceph.osd.0.smithi179.stderr:INFO 2023-01-24 14:42:15,862 [shard 0] osd - get_early_config: set --smp 1 2023-01-24T14:42:16.087 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-01-24T14:42:16.086+0000 7f19d9f3a1c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-01-24T14:42:16.242 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-01-24T14:42:16.241+0000 7f19d9f3a1c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-01-24T14:42:16.484 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-01-24T14:42:16.482+0000 7f19d9f3a1c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-01-24T14:42:16.563 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-01-24T14:42:16.561+0000 7f19d9f3a1c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-01-24T14:42:16.714 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-01-24T14:42:16.712+0000 7f19d9f3a1c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-01-24T14:42:17.004 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-01-24T14:42:17.002+0000 7f19d9f3a1c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-01-24T14:42:17.206 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-01-24T14:42:17.204+0000 7f19d9f3a1c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-01-24T14:42:17.327 INFO:tasks.ceph.osd.0.smithi179.stderr:WARNING: debug mode. Not for benchmarking or production 2023-01-24T14:42:17.327 INFO:tasks.ceph.osd.2.smithi179.stderr:WARNING: debug mode. Not for benchmarking or production 2023-01-24T14:42:17.328 INFO:tasks.ceph.osd.0.smithi179.stderr:INFO 2023-01-24 14:42:17,327 seastar - Reactor backend: epoll 2023-01-24T14:42:17.328 INFO:tasks.ceph.osd.2.smithi179.stderr:INFO 2023-01-24 14:42:17,327 seastar - Reactor backend: epoll 2023-01-24T14:42:17.344 INFO:tasks.ceph.osd.1.smithi179.stderr:WARNING: debug mode. Not for benchmarking or production 2023-01-24T14:42:17.344 INFO:tasks.ceph.osd.1.smithi179.stderr:INFO 2023-01-24 14:42:17,343 seastar - Reactor backend: epoll 2023-01-24T14:42:17.355 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-01-24T14:42:17.353+0000 7f19d9f3a1c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-01-24T14:42:17.433 INFO:tasks.ceph.osd.1.smithi179.stderr:INFO 2023-01-24 14:42:17,432 [shard 0] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-01-24T14:42:17.433 INFO:tasks.ceph.osd.1.smithi179.stderr:INFO 2023-01-24 14:42:17,432 [shard 0] seastar - IO queue uses 0.75ms latency goal for device 0 2023-01-24T14:42:17.434 INFO:tasks.ceph.osd.1.smithi179.stderr:INFO 2023-01-24 14:42:17,432 [shard 0] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-01-24T14:42:17.434 INFO:tasks.ceph.osd.1.smithi179.stderr:INFO 2023-01-24 14:42:17,432 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-01-24T14:42:17.435 INFO:tasks.ceph.osd.0.smithi179.stderr:INFO 2023-01-24 14:42:17,434 [shard 0] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-01-24T14:42:17.435 INFO:tasks.ceph.osd.0.smithi179.stderr:INFO 2023-01-24 14:42:17,434 [shard 0] seastar - IO queue uses 0.75ms latency goal for device 0 2023-01-24T14:42:17.435 INFO:tasks.ceph.osd.0.smithi179.stderr:INFO 2023-01-24 14:42:17,435 [shard 0] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-01-24T14:42:17.436 INFO:tasks.ceph.osd.0.smithi179.stderr:INFO 2023-01-24 14:42:17,435 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-01-24T14:42:17.437 INFO:tasks.ceph.osd.0.smithi179.stderr:==103809==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-01-24T14:42:17.437 INFO:tasks.ceph.osd.1.smithi179.stderr:==103808==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-01-24T14:42:17.450 INFO:tasks.ceph.osd.0.smithi179.stderr:==103809==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7ffffb17d000; bottom 0x7efe99705000; size: 0x010161a78000 (1105444962304) 2023-01-24T14:42:17.450 INFO:tasks.ceph.osd.0.smithi179.stderr:False positive error reports may follow 2023-01-24T14:42:17.450 INFO:tasks.ceph.osd.0.smithi179.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-01-24T14:42:17.451 INFO:tasks.ceph.osd.1.smithi179.stderr:==103808==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7ffd0fb25000; bottom 0x7fc7236ce000; size: 0x0035ec457000 (231597240320) 2023-01-24T14:42:17.451 INFO:tasks.ceph.osd.1.smithi179.stderr:False positive error reports may follow 2023-01-24T14:42:17.451 INFO:tasks.ceph.osd.1.smithi179.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-01-24T14:42:17.464 INFO:tasks.ceph.osd.2.smithi179.stderr:INFO 2023-01-24 14:42:17,463 [shard 0] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-01-24T14:42:17.464 INFO:tasks.ceph.osd.2.smithi179.stderr:INFO 2023-01-24 14:42:17,464 [shard 0] seastar - IO queue uses 0.75ms latency goal for device 0 2023-01-24T14:42:17.465 INFO:tasks.ceph.osd.2.smithi179.stderr:INFO 2023-01-24 14:42:17,464 [shard 0] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-01-24T14:42:17.465 INFO:tasks.ceph.osd.2.smithi179.stderr:INFO 2023-01-24 14:42:17,464 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-01-24T14:42:17.466 INFO:tasks.ceph.osd.2.smithi179.stderr:==103807==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-01-24T14:42:17.472 INFO:tasks.ceph.osd.2.smithi179.stderr:==103807==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7fffcd2a8000; bottom 0x7f833c876000; size: 0x007c90a32000 (535002554368) 2023-01-24T14:42:17.473 INFO:tasks.ceph.osd.2.smithi179.stderr:False positive error reports may follow 2023-01-24T14:42:17.473 INFO:tasks.ceph.osd.2.smithi179.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-01-24T14:42:17.473 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-01-24T14:42:17.472+0000 7f19d9f3a1c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-01-24T14:42:17.619 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-01-24T14:42:17.617+0000 7f19d9f3a1c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-01-24T14:42:17.701 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-01-24T14:42:17.699+0000 7f19d9f3a1c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-01-24T14:42:17.928 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-01-24T14:42:17.926+0000 7f19d9f3a1c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-01-24T14:42:18.007 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-01-24T14:42:18.006+0000 7f19d9f3a1c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-01-24T14:42:18.023 INFO:tasks.ceph.osd.0.smithi179.stderr:Reactor stalled for 244 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0xce16a 0x588b4 0xc2e88 0x3da09065 0x3da0ccbd 0x3da10f92 0x3d8fb06c 0x364b8c44 0x364b9159 0x362c6842 0x362dcdef 0x362dda11 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T14:42:18.024 INFO:tasks.ceph.osd.0.smithi179.stderr:kernel callstack: 2023-01-24T14:42:18.024 INFO:tasks.ceph.osd.0.smithi179.stderr:kernel callstack: 0xffffffffffffff80 0xffffffffb1003f19 0xffffffffb100449d 0xffffffffb1a000a9 2023-01-24T14:42:18.666 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-01-24T14:42:18.664+0000 7f19d9f3a1c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-01-24T14:42:18.924 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-01-24T14:42:18.922+0000 7f19d9f3a1c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-01-24T14:42:18.998 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-01-24T14:42:18.996+0000 7f19d9f3a1c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-01-24T14:42:19.079 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-01-24T14:42:19.077+0000 7f19d9f3a1c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-01-24T14:42:19.262 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-01-24T14:42:19.260+0000 7f19d9f3a1c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-01-24T14:42:19.332 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-01-24T14:42:19.331+0000 7f19d9f3a1c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-01-24T14:42:19.486 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-01-24T14:42:19.484+0000 7f19d9f3a1c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-01-24T14:42:20.509 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-01-24T14:42:20.507+0000 7f1996580700 -1 mgr.server handle_report got status from non-daemon mon.a 2023-01-24T14:42:21.898 DEBUG:teuthology.orchestra.run.smithi179:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-01-24T14:42:22.218 INFO:teuthology.misc.health.smithi179.stdout: 2023-01-24T14:42:22.219 INFO:teuthology.misc.health.smithi179.stdout:{"epoch":8,"fsid":"91764032-a053-4a42-a007-59362bb9c17e","created":"2023-01-24T14:42:13.552129+0000","modified":"2023-01-24T14:42:20.478271+0000","last_up_change":"2023-01-24T14:42:20.478271+0000","last_in_change":"2023-01-24T14:42:14.554200+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","pools":[],"osds":[{"osd":0,"uuid":"3abca7f8-4a61-4e04-b5d4-315fb643d792","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6803","nonce":103809}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":103809}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6809","nonce":103809}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6811","nonce":103809}]},"public_addr":"172.21.15.179:6803/103809","cluster_addr":"172.21.15.179:6802/103809","heartbeat_back_addr":"172.21.15.179:6809/103809","heartbeat_front_addr":"172.21.15.179:6811/103809","state":["exists","up"]},{"osd":1,"uuid":"a0e7d95f-3409-475d-a8d0-a6d00dbd9f99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6805","nonce":103808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":103808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":103808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6810","nonce":103808}]},"public_addr":"172.21.15.179:6805/103808","cluster_addr":"172.21.15.179:6804/103808","heartbeat_back_addr":"172.21.15.179:6808/103808","heartbeat_front_addr":"172.21.15.179:6810/103808","state":["exists","up"]},{"osd":2,"uuid":"79d1ce53-95fc-4407-96a5-cea3d9595b13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":7,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6801","nonce":103807}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":103807}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6806","nonce":103807}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6807","nonce":103807}]},"public_addr":"172.21.15.179:6801/103807","cluster_addr":"172.21.15.179:6800/103807","heartbeat_back_addr":"172.21.15.179:6806/103807","heartbeat_front_addr":"172.21.15.179:6807/103807","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-01-24T14:42:22.231 DEBUG:teuthology.misc:3 of 3 OSDs are up 2023-01-24T14:42:22.231 INFO:tasks.ceph:Creating RBD pool 2023-01-24T14:42:22.232 DEBUG:teuthology.orchestra.run.smithi179:> sudo ceph --cluster ceph osd pool create rbd 8 2023-01-24T14:42:23.503 INFO:teuthology.orchestra.run.smithi179.stderr:pool 'rbd' created 2023-01-24T14:42:23.524 DEBUG:teuthology.orchestra.run.smithi179:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2023-01-24T14:42:24.518 INFO:teuthology.orchestra.run.smithi179.stderr:enabled application 'rbd' on pool 'rbd' 2023-01-24T14:42:24.530 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2023-01-24T14:42:24.531 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2023-01-24T14:42:24.531 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2023-01-24T14:42:24.532 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2023-01-24T14:42:24.532 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-01-24T14:42:24.924 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-01-24T14:42:24.937 INFO:teuthology.orchestra.run.smithi179.stdout:{"epoch":4,"active_gid":4101,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":103399},{"type":"v1","addr":"172.21.15.179:6813","nonce":103399}]},"active_addr":"172.21.15.179:6813/103399","active_change":"2023-01-24T14:42:19.498083+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":0,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.179:0","nonce":1215502231}]},{"addrvec":[{"type":"v2","addr":"172.21.15.179:0","nonce":3240780116}]},{"addrvec":[{"type":"v2","addr":"172.21.15.179:0","nonce":2988819449}]}]}} 2023-01-24T14:42:24.939 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2023-01-24T14:42:24.939 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2023-01-24T14:42:24.940 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-01-24T14:42:25.280 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-01-24T14:42:25.280 INFO:teuthology.orchestra.run.smithi179.stdout:{"epoch":11,"fsid":"91764032-a053-4a42-a007-59362bb9c17e","created":"2023-01-24T14:42:13.552129+0000","modified":"2023-01-24T14:42:24.512187+0000","last_up_change":"2023-01-24T14:42:20.478271+0000","last_in_change":"2023-01-24T14:42:14.554200+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-01-24T14:42:21.544008+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-01-24T14:42:22.569448+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"3abca7f8-4a61-4e04-b5d4-315fb643d792","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6803","nonce":103809}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":103809}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6809","nonce":103809}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6811","nonce":103809}]},"public_addr":"172.21.15.179:6803/103809","cluster_addr":"172.21.15.179:6802/103809","heartbeat_back_addr":"172.21.15.179:6809/103809","heartbeat_front_addr":"172.21.15.179:6811/103809","state":["exists","up"]},{"osd":1,"uuid":"a0e7d95f-3409-475d-a8d0-a6d00dbd9f99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6805","nonce":103808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":103808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":103808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6810","nonce":103808}]},"public_addr":"172.21.15.179:6805/103808","cluster_addr":"172.21.15.179:6804/103808","heartbeat_back_addr":"172.21.15.179:6808/103808","heartbeat_front_addr":"172.21.15.179:6810/103808","state":["exists","up"]},{"osd":2,"uuid":"79d1ce53-95fc-4407-96a5-cea3d9595b13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":7,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6801","nonce":103807}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":103807}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6806","nonce":103807}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6807","nonce":103807}]},"public_addr":"172.21.15.179:6801/103807","cluster_addr":"172.21.15.179:6800/103807","heartbeat_back_addr":"172.21.15.179:6806/103807","heartbeat_front_addr":"172.21.15.179:6807/103807","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-01-24T14:42:25.295 INFO:tasks.ceph.ceph_manager.ceph:all up! 2023-01-24T14:42:25.295 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-01-24T14:42:25.628 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-01-24T14:42:25.629 INFO:teuthology.orchestra.run.smithi179.stdout:{"epoch":12,"fsid":"91764032-a053-4a42-a007-59362bb9c17e","created":"2023-01-24T14:42:13.552129+0000","modified":"2023-01-24T14:42:25.513143+0000","last_up_change":"2023-01-24T14:42:20.478271+0000","last_in_change":"2023-01-24T14:42:14.554200+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-01-24T14:42:21.544008+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-01-24T14:42:22.569448+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"3abca7f8-4a61-4e04-b5d4-315fb643d792","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6803","nonce":103809}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":103809}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6809","nonce":103809}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6811","nonce":103809}]},"public_addr":"172.21.15.179:6803/103809","cluster_addr":"172.21.15.179:6802/103809","heartbeat_back_addr":"172.21.15.179:6809/103809","heartbeat_front_addr":"172.21.15.179:6811/103809","state":["exists","up"]},{"osd":1,"uuid":"a0e7d95f-3409-475d-a8d0-a6d00dbd9f99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6805","nonce":103808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":103808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":103808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6810","nonce":103808}]},"public_addr":"172.21.15.179:6805/103808","cluster_addr":"172.21.15.179:6804/103808","heartbeat_back_addr":"172.21.15.179:6808/103808","heartbeat_front_addr":"172.21.15.179:6810/103808","state":["exists","up"]},{"osd":2,"uuid":"79d1ce53-95fc-4407-96a5-cea3d9595b13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":7,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6801","nonce":103807}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":103807}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6806","nonce":103807}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6807","nonce":103807}]},"public_addr":"172.21.15.179:6801/103807","cluster_addr":"172.21.15.179:6800/103807","heartbeat_back_addr":"172.21.15.179:6806/103807","heartbeat_front_addr":"172.21.15.179:6807/103807","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-01-24T14:42:25.643 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2023-01-24T14:42:25.644 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2023-01-24T14:42:25.649 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2023-01-24T14:42:25.819 INFO:teuthology.orchestra.run.smithi179.stdout:34359738373 2023-01-24T14:42:25.820 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-01-24T14:42:25.837 INFO:teuthology.orchestra.run.smithi179.stdout:30064771078 2023-01-24T14:42:25.837 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-01-24T14:42:25.838 INFO:teuthology.orchestra.run.smithi179.stdout:34359738373 2023-01-24T14:42:25.839 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-01-24T14:42:26.196 INFO:teuthology.orchestra.run.smithi179.stdout:34359738373 2023-01-24T14:42:26.210 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 34359738373 for osd.1 2023-01-24T14:42:26.211 DEBUG:teuthology.parallel:result is None 2023-01-24T14:42:26.226 INFO:teuthology.orchestra.run.smithi179.stdout:30064771078 2023-01-24T14:42:26.226 INFO:teuthology.orchestra.run.smithi179.stdout:34359738373 2023-01-24T14:42:26.243 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 34359738373 for osd.0 2023-01-24T14:42:26.243 DEBUG:teuthology.parallel:result is None 2023-01-24T14:42:26.244 INFO:tasks.ceph.ceph_manager.ceph:need seq 30064771078 got 30064771078 for osd.2 2023-01-24T14:42:26.244 DEBUG:teuthology.parallel:result is None 2023-01-24T14:42:26.244 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-01-24T14:42:26.245 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-01-24T14:42:26.611 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-01-24T14:42:26.613 INFO:teuthology.orchestra.run.smithi179.stderr:dumped all 2023-01-24T14:42:26.625 INFO:teuthology.orchestra.run.smithi179.stdout:{"pg_ready":true,"pg_map":{"version":8,"stamp":"2023-01-24T14:42:25.816592+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":25,"num_read_kb":18,"num_write":83,"num_write_kb":848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":50,"ondisk_log_size":50,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":314572800,"kb_used":16168,"kb_used_data":792,"kb_used_omap":0,"kb_used_meta":14848,"kb_avail":314556632,"statfs":{"total":322122547200,"available":322105991168,"internally_reserved":0,"allocated":811008,"data_stored":701470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":15204352},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-01-24T14:42:24.664412+0000","last_change":"2023-01-24T14:42:24.664412+0000","last_active":"2023-01-24T14:42:24.664412+0000","last_peered":"2023-01-24T14:42:24.664412+0000","last_clean":"2023-01-24T14:42:24.664412+0000","last_became_active":"2023-01-24T14:42:24.661922+0000","last_became_peered":"2023-01-24T14:42:24.661922+0000","last_unstale":"2023-01-24T14:42:24.664412+0000","last_undegraded":"2023-01-24T14:42:24.664412+0000","last_fullsized":"2023-01-24T14:42:24.664412+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T14:42:23.493256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T14:42:23.493256+0000","last_clean_scrub_stamp":"2023-01-24T14:42:23.493256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-01-24T14:42:24.669889+0000","last_change":"2023-01-24T14:42:24.669889+0000","last_active":"2023-01-24T14:42:24.669889+0000","last_peered":"2023-01-24T14:42:24.669889+0000","last_clean":"2023-01-24T14:42:24.669889+0000","last_became_active":"2023-01-24T14:42:24.667022+0000","last_became_peered":"2023-01-24T14:42:24.667022+0000","last_unstale":"2023-01-24T14:42:24.669889+0000","last_undegraded":"2023-01-24T14:42:24.669889+0000","last_fullsized":"2023-01-24T14:42:24.669889+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T14:42:23.493256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T14:42:23.493256+0000","last_clean_scrub_stamp":"2023-01-24T14:42:23.493256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-01-24T14:42:24.658975+0000","last_change":"2023-01-24T14:42:24.658975+0000","last_active":"2023-01-24T14:42:24.658975+0000","last_peered":"2023-01-24T14:42:24.658975+0000","last_clean":"2023-01-24T14:42:24.658975+0000","last_became_active":"2023-01-24T14:42:24.655994+0000","last_became_peered":"2023-01-24T14:42:24.655994+0000","last_unstale":"2023-01-24T14:42:24.658975+0000","last_undegraded":"2023-01-24T14:42:24.658975+0000","last_fullsized":"2023-01-24T14:42:24.658975+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T14:42:23.493256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T14:42:23.493256+0000","last_clean_scrub_stamp":"2023-01-24T14:42:23.493256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-01-24T14:42:24.704164+0000","last_change":"2023-01-24T14:42:24.705611+0000","last_active":"2023-01-24T14:42:24.704164+0000","last_peered":"2023-01-24T14:42:24.704164+0000","last_clean":"2023-01-24T14:42:23.493256+0000","last_became_active":"2023-01-24T14:42:24.697628+0000","last_became_peered":"2023-01-24T14:42:24.697628+0000","last_unstale":"2023-01-24T14:42:24.704164+0000","last_undegraded":"2023-01-24T14:42:24.704164+0000","last_fullsized":"2023-01-24T14:42:24.704164+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T14:42:23.493256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T14:42:23.493256+0000","last_clean_scrub_stamp":"2023-01-24T14:42:23.493256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"12'50","reported_seq":61,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-01-24T14:42:25.709280+0000","last_change":"2023-01-24T14:42:23.597018+0000","last_active":"2023-01-24T14:42:25.709280+0000","last_peered":"2023-01-24T14:42:25.709280+0000","last_clean":"2023-01-24T14:42:25.709280+0000","last_became_active":"2023-01-24T14:42:23.584003+0000","last_became_peered":"2023-01-24T14:42:23.584003+0000","last_unstale":"2023-01-24T14:42:25.709280+0000","last_undegraded":"2023-01-24T14:42:25.709280+0000","last_fullsized":"2023-01-24T14:42:25.709280+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T14:42:22.489089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T14:42:22.489089+0000","last_clean_scrub_stamp":"2023-01-24T14:42:22.489089+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":25,"num_read_kb":18,"num_write":83,"num_write_kb":848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-01-24T14:42:24.641821+0000","last_change":"2023-01-24T14:42:24.641821+0000","last_active":"2023-01-24T14:42:24.641821+0000","last_peered":"2023-01-24T14:42:24.641821+0000","last_clean":"2023-01-24T14:42:24.641821+0000","last_became_active":"2023-01-24T14:42:24.637489+0000","last_became_peered":"2023-01-24T14:42:24.637489+0000","last_unstale":"2023-01-24T14:42:24.641821+0000","last_undegraded":"2023-01-24T14:42:24.641821+0000","last_fullsized":"2023-01-24T14:42:24.641821+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T14:42:23.493256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T14:42:23.493256+0000","last_clean_scrub_stamp":"2023-01-24T14:42:23.493256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-01-24T14:42:24.677226+0000","last_change":"2023-01-24T14:42:24.677952+0000","last_active":"2023-01-24T14:42:24.677226+0000","last_peered":"2023-01-24T14:42:24.677226+0000","last_clean":"2023-01-24T14:42:23.493256+0000","last_became_active":"2023-01-24T14:42:24.673637+0000","last_became_peered":"2023-01-24T14:42:24.673637+0000","last_unstale":"2023-01-24T14:42:24.677226+0000","last_undegraded":"2023-01-24T14:42:24.677226+0000","last_fullsized":"2023-01-24T14:42:24.677226+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T14:42:23.493256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T14:42:23.493256+0000","last_clean_scrub_stamp":"2023-01-24T14:42:23.493256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-01-24T14:42:24.686289+0000","last_change":"2023-01-24T14:42:24.686669+0000","last_active":"2023-01-24T14:42:24.686289+0000","last_peered":"2023-01-24T14:42:24.686289+0000","last_clean":"2023-01-24T14:42:23.493256+0000","last_became_active":"2023-01-24T14:42:24.683545+0000","last_became_peered":"2023-01-24T14:42:24.683545+0000","last_unstale":"2023-01-24T14:42:24.686289+0000","last_undegraded":"2023-01-24T14:42:24.686289+0000","last_fullsized":"2023-01-24T14:42:24.686289+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T14:42:23.493256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T14:42:23.493256+0000","last_clean_scrub_stamp":"2023-01-24T14:42:23.493256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-01-24T14:42:24.657421+0000","last_change":"2023-01-24T14:42:24.657421+0000","last_active":"2023-01-24T14:42:24.657421+0000","last_peered":"2023-01-24T14:42:24.657421+0000","last_clean":"2023-01-24T14:42:24.657421+0000","last_became_active":"2023-01-24T14:42:24.653175+0000","last_became_peered":"2023-01-24T14:42:24.653175+0000","last_unstale":"2023-01-24T14:42:24.657421+0000","last_undegraded":"2023-01-24T14:42:24.657421+0000","last_fullsized":"2023-01-24T14:42:24.657421+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T14:42:23.493256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T14:42:23.493256+0000","last_clean_scrub_stamp":"2023-01-24T14:42:23.493256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":25,"num_read_kb":18,"num_write":83,"num_write_kb":848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":50,"ondisk_log_size":50,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":2,"up_from":7,"seq":30064771078,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":4920,"kb_used_data":48,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":104852680,"statfs":{"total":107374182400,"available":107369144320,"internally_reserved":0,"allocated":49152,"data_stored":15018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":8,"seq":34359738373,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":5624,"kb_used_data":372,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":104851976,"statfs":{"total":107374182400,"available":107368423424,"internally_reserved":0,"allocated":380928,"data_stored":343226,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":8,"seq":34359738373,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":5624,"kb_used_data":372,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":104851976,"statfs":{"total":107374182400,"available":107368423424,"internally_reserved":0,"allocated":380928,"data_stored":343226,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2023-01-24T14:42:26.626 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-01-24T14:42:26.954 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-01-24T14:42:26.956 INFO:teuthology.orchestra.run.smithi179.stderr:dumped all 2023-01-24T14:42:26.968 INFO:teuthology.orchestra.run.smithi179.stdout:{"pg_ready":true,"pg_map":{"version":8,"stamp":"2023-01-24T14:42:25.816592+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":25,"num_read_kb":18,"num_write":83,"num_write_kb":848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":50,"ondisk_log_size":50,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":314572800,"kb_used":16168,"kb_used_data":792,"kb_used_omap":0,"kb_used_meta":14848,"kb_avail":314556632,"statfs":{"total":322122547200,"available":322105991168,"internally_reserved":0,"allocated":811008,"data_stored":701470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":15204352},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-01-24T14:42:24.664412+0000","last_change":"2023-01-24T14:42:24.664412+0000","last_active":"2023-01-24T14:42:24.664412+0000","last_peered":"2023-01-24T14:42:24.664412+0000","last_clean":"2023-01-24T14:42:24.664412+0000","last_became_active":"2023-01-24T14:42:24.661922+0000","last_became_peered":"2023-01-24T14:42:24.661922+0000","last_unstale":"2023-01-24T14:42:24.664412+0000","last_undegraded":"2023-01-24T14:42:24.664412+0000","last_fullsized":"2023-01-24T14:42:24.664412+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T14:42:23.493256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T14:42:23.493256+0000","last_clean_scrub_stamp":"2023-01-24T14:42:23.493256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-01-24T14:42:24.669889+0000","last_change":"2023-01-24T14:42:24.669889+0000","last_active":"2023-01-24T14:42:24.669889+0000","last_peered":"2023-01-24T14:42:24.669889+0000","last_clean":"2023-01-24T14:42:24.669889+0000","last_became_active":"2023-01-24T14:42:24.667022+0000","last_became_peered":"2023-01-24T14:42:24.667022+0000","last_unstale":"2023-01-24T14:42:24.669889+0000","last_undegraded":"2023-01-24T14:42:24.669889+0000","last_fullsized":"2023-01-24T14:42:24.669889+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T14:42:23.493256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T14:42:23.493256+0000","last_clean_scrub_stamp":"2023-01-24T14:42:23.493256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-01-24T14:42:24.658975+0000","last_change":"2023-01-24T14:42:24.658975+0000","last_active":"2023-01-24T14:42:24.658975+0000","last_peered":"2023-01-24T14:42:24.658975+0000","last_clean":"2023-01-24T14:42:24.658975+0000","last_became_active":"2023-01-24T14:42:24.655994+0000","last_became_peered":"2023-01-24T14:42:24.655994+0000","last_unstale":"2023-01-24T14:42:24.658975+0000","last_undegraded":"2023-01-24T14:42:24.658975+0000","last_fullsized":"2023-01-24T14:42:24.658975+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T14:42:23.493256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T14:42:23.493256+0000","last_clean_scrub_stamp":"2023-01-24T14:42:23.493256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-01-24T14:42:24.704164+0000","last_change":"2023-01-24T14:42:24.705611+0000","last_active":"2023-01-24T14:42:24.704164+0000","last_peered":"2023-01-24T14:42:24.704164+0000","last_clean":"2023-01-24T14:42:23.493256+0000","last_became_active":"2023-01-24T14:42:24.697628+0000","last_became_peered":"2023-01-24T14:42:24.697628+0000","last_unstale":"2023-01-24T14:42:24.704164+0000","last_undegraded":"2023-01-24T14:42:24.704164+0000","last_fullsized":"2023-01-24T14:42:24.704164+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T14:42:23.493256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T14:42:23.493256+0000","last_clean_scrub_stamp":"2023-01-24T14:42:23.493256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"12'50","reported_seq":61,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-01-24T14:42:25.709280+0000","last_change":"2023-01-24T14:42:23.597018+0000","last_active":"2023-01-24T14:42:25.709280+0000","last_peered":"2023-01-24T14:42:25.709280+0000","last_clean":"2023-01-24T14:42:25.709280+0000","last_became_active":"2023-01-24T14:42:23.584003+0000","last_became_peered":"2023-01-24T14:42:23.584003+0000","last_unstale":"2023-01-24T14:42:25.709280+0000","last_undegraded":"2023-01-24T14:42:25.709280+0000","last_fullsized":"2023-01-24T14:42:25.709280+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T14:42:22.489089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T14:42:22.489089+0000","last_clean_scrub_stamp":"2023-01-24T14:42:22.489089+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":25,"num_read_kb":18,"num_write":83,"num_write_kb":848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-01-24T14:42:24.641821+0000","last_change":"2023-01-24T14:42:24.641821+0000","last_active":"2023-01-24T14:42:24.641821+0000","last_peered":"2023-01-24T14:42:24.641821+0000","last_clean":"2023-01-24T14:42:24.641821+0000","last_became_active":"2023-01-24T14:42:24.637489+0000","last_became_peered":"2023-01-24T14:42:24.637489+0000","last_unstale":"2023-01-24T14:42:24.641821+0000","last_undegraded":"2023-01-24T14:42:24.641821+0000","last_fullsized":"2023-01-24T14:42:24.641821+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T14:42:23.493256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T14:42:23.493256+0000","last_clean_scrub_stamp":"2023-01-24T14:42:23.493256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-01-24T14:42:24.677226+0000","last_change":"2023-01-24T14:42:24.677952+0000","last_active":"2023-01-24T14:42:24.677226+0000","last_peered":"2023-01-24T14:42:24.677226+0000","last_clean":"2023-01-24T14:42:23.493256+0000","last_became_active":"2023-01-24T14:42:24.673637+0000","last_became_peered":"2023-01-24T14:42:24.673637+0000","last_unstale":"2023-01-24T14:42:24.677226+0000","last_undegraded":"2023-01-24T14:42:24.677226+0000","last_fullsized":"2023-01-24T14:42:24.677226+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T14:42:23.493256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T14:42:23.493256+0000","last_clean_scrub_stamp":"2023-01-24T14:42:23.493256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-01-24T14:42:24.686289+0000","last_change":"2023-01-24T14:42:24.686669+0000","last_active":"2023-01-24T14:42:24.686289+0000","last_peered":"2023-01-24T14:42:24.686289+0000","last_clean":"2023-01-24T14:42:23.493256+0000","last_became_active":"2023-01-24T14:42:24.683545+0000","last_became_peered":"2023-01-24T14:42:24.683545+0000","last_unstale":"2023-01-24T14:42:24.686289+0000","last_undegraded":"2023-01-24T14:42:24.686289+0000","last_fullsized":"2023-01-24T14:42:24.686289+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T14:42:23.493256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T14:42:23.493256+0000","last_clean_scrub_stamp":"2023-01-24T14:42:23.493256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-01-24T14:42:24.657421+0000","last_change":"2023-01-24T14:42:24.657421+0000","last_active":"2023-01-24T14:42:24.657421+0000","last_peered":"2023-01-24T14:42:24.657421+0000","last_clean":"2023-01-24T14:42:24.657421+0000","last_became_active":"2023-01-24T14:42:24.653175+0000","last_became_peered":"2023-01-24T14:42:24.653175+0000","last_unstale":"2023-01-24T14:42:24.657421+0000","last_undegraded":"2023-01-24T14:42:24.657421+0000","last_fullsized":"2023-01-24T14:42:24.657421+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T14:42:23.493256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T14:42:23.493256+0000","last_clean_scrub_stamp":"2023-01-24T14:42:23.493256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":25,"num_read_kb":18,"num_write":83,"num_write_kb":848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":50,"ondisk_log_size":50,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":2,"up_from":7,"seq":30064771078,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":4920,"kb_used_data":48,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":104852680,"statfs":{"total":107374182400,"available":107369144320,"internally_reserved":0,"allocated":49152,"data_stored":15018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":8,"seq":34359738373,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":5624,"kb_used_data":372,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":104851976,"statfs":{"total":107374182400,"available":107368423424,"internally_reserved":0,"allocated":380928,"data_stored":343226,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":8,"seq":34359738373,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":5624,"kb_used_data":372,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":104851976,"statfs":{"total":107374182400,"available":107368423424,"internally_reserved":0,"allocated":380928,"data_stored":343226,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2023-01-24T14:42:26.969 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-01-24T14:42:26.969 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2023-01-24T14:42:26.970 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2023-01-24T14:42:26.970 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-01-24T14:42:27.337 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-01-24T14:42:27.337 INFO:teuthology.orchestra.run.smithi179.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-01-24T14:42:27.351 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2023-01-24T14:42:27.351 INFO:teuthology.run_tasks:Running task workunit... 2023-01-24T14:42:27.364 INFO:tasks.workunit:Pulling workunits from ref 510284b66513490445619d1430aa869868c71a09 2023-01-24T14:42:27.365 INFO:tasks.workunit:Making a separate scratch dir for every client... 2023-01-24T14:42:27.365 DEBUG:teuthology.orchestra.run.smithi179:> stat -- /home/ubuntu/cephtest/mnt.0 2023-01-24T14:42:27.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-01-24T14:42:27.384 INFO:teuthology.orchestra.run.smithi179.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2023-01-24T14:42:27.384 DEBUG:teuthology.orchestra.run.smithi179:> mkdir -- /home/ubuntu/cephtest/mnt.0 2023-01-24T14:42:27.443 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2023-01-24T14:42:27.443 DEBUG:teuthology.orchestra.run.smithi179:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2023-01-24T14:42:27.501 INFO:tasks.workunit:timeout=3h 2023-01-24T14:42:27.502 INFO:tasks.workunit:cleanup=True 2023-01-24T14:42:27.502 DEBUG:teuthology.orchestra.run.smithi179:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 510284b66513490445619d1430aa869868c71a09 2023-01-24T14:42:27.560 INFO:tasks.workunit.client.0.smithi179.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2023-01-24T14:43:15.180 INFO:tasks.workunit.client.0.smithi179.stderr:Note: switching to '510284b66513490445619d1430aa869868c71a09'. 2023-01-24T14:43:15.180 INFO:tasks.workunit.client.0.smithi179.stderr: 2023-01-24T14:43:15.180 INFO:tasks.workunit.client.0.smithi179.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2023-01-24T14:43:15.181 INFO:tasks.workunit.client.0.smithi179.stderr:changes and commit them, and you can discard any commits you make in this 2023-01-24T14:43:15.181 INFO:tasks.workunit.client.0.smithi179.stderr:state without impacting any branches by switching back to a branch. 2023-01-24T14:43:15.181 INFO:tasks.workunit.client.0.smithi179.stderr: 2023-01-24T14:43:15.181 INFO:tasks.workunit.client.0.smithi179.stderr:If you want to create a new branch to retain commits you create, you may 2023-01-24T14:43:15.181 INFO:tasks.workunit.client.0.smithi179.stderr:do so (now or later) by using -c with the switch command. Example: 2023-01-24T14:43:15.182 INFO:tasks.workunit.client.0.smithi179.stderr: 2023-01-24T14:43:15.182 INFO:tasks.workunit.client.0.smithi179.stderr: git switch -c 2023-01-24T14:43:15.182 INFO:tasks.workunit.client.0.smithi179.stderr: 2023-01-24T14:43:15.182 INFO:tasks.workunit.client.0.smithi179.stderr:Or undo this operation with: 2023-01-24T14:43:15.183 INFO:tasks.workunit.client.0.smithi179.stderr: 2023-01-24T14:43:15.183 INFO:tasks.workunit.client.0.smithi179.stderr: git switch - 2023-01-24T14:43:15.183 INFO:tasks.workunit.client.0.smithi179.stderr: 2023-01-24T14:43:15.183 INFO:tasks.workunit.client.0.smithi179.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2023-01-24T14:43:15.183 INFO:tasks.workunit.client.0.smithi179.stderr: 2023-01-24T14:43:15.184 INFO:tasks.workunit.client.0.smithi179.stderr:HEAD is now at 510284b6651 Merge pull request #49791 from idryomov/wip-doc-exclusive-lock-transitions 2023-01-24T14:43:15.185 DEBUG:teuthology.orchestra.run.smithi179:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2023-01-24T14:43:15.249 INFO:tasks.workunit.client.0.smithi179.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2023-01-24T14:43:15.251 INFO:tasks.workunit.client.0.smithi179.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2023-01-24T14:43:15.251 INFO:tasks.workunit.client.0.smithi179.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2023-01-24T14:43:15.802 INFO:tasks.workunit.client.0.smithi179.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2023-01-24T14:43:15.867 INFO:tasks.workunit.client.0.smithi179.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2023-01-24T14:43:15.904 INFO:tasks.workunit.client.0.smithi179.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2023-01-24T14:43:15.906 INFO:tasks.workunit.client.0.smithi179.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2023-01-24T14:43:15.906 INFO:tasks.workunit.client.0.smithi179.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2023-01-24T14:43:15.942 INFO:tasks.workunit.client.0.smithi179.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2023-01-24T14:43:15.945 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2023-01-24T14:43:15.945 DEBUG:teuthology.orchestra.run.smithi179:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2023-01-24T14:43:16.005 INFO:tasks.workunit:Running workunits matching rbd/test_lock_fence.sh on client.0... 2023-01-24T14:43:16.006 INFO:tasks.workunit:Running workunit rbd/test_lock_fence.sh... 2023-01-24T14:43:16.006 DEBUG:teuthology.orchestra.run.smithi179:workunit test rbd/test_lock_fence.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=510284b66513490445619d1430aa869868c71a09 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 CRIMSON_COMPAT=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/test_lock_fence.sh 2023-01-24T14:43:16.067 INFO:tasks.workunit.client.0.smithi179.stderr:+ IMAGE=rbdrw-image 2023-01-24T14:43:16.067 INFO:tasks.workunit.client.0.smithi179.stderr:+ LOCKID=rbdrw 2023-01-24T14:43:16.068 INFO:tasks.workunit.client.0.smithi179.stderr:++ dirname /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/test_lock_fence.sh 2023-01-24T14:43:16.068 INFO:tasks.workunit.client.0.smithi179.stderr:+ RELPATH=/home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/../../../src/test/librbd 2023-01-24T14:43:16.068 INFO:tasks.workunit.client.0.smithi179.stderr:+ RBDRW=/home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/../../../src/test/librbd/rbdrw.py 2023-01-24T14:43:16.069 INFO:tasks.workunit.client.0.smithi179.stderr:+ rbd create rbdrw-image --size 10 --image-format 2 --image-shared 2023-01-24T14:43:17.798 INFO:tasks.workunit.client.0.smithi179.stderr:+ iochild=106167 2023-01-24T14:43:17.799 INFO:tasks.workunit.client.0.smithi179.stderr:+ LOCKS='[]' 2023-01-24T14:43:17.799 INFO:tasks.workunit.client.0.smithi179.stderr:+ '[' '[]' == '[]' ']' 2023-01-24T14:43:17.799 INFO:tasks.workunit.client.0.smithi179.stderr:+ python3 /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/../../../src/test/librbd/rbdrw.py rbdrw-image rbdrw 2023-01-24T14:43:17.800 INFO:tasks.workunit.client.0.smithi179.stderr:++ rbd lock list rbdrw-image --format json 2023-01-24T14:43:17.892 INFO:tasks.workunit.client.0.smithi179.stderr:+ LOCKS='[]' 2023-01-24T14:43:17.892 INFO:tasks.workunit.client.0.smithi179.stderr:+ sleep 1 2023-01-24T14:43:18.893 INFO:tasks.workunit.client.0.smithi179.stderr:+ '[' '[]' == '[]' ']' 2023-01-24T14:43:18.893 INFO:tasks.workunit.client.0.smithi179.stderr:++ rbd lock list rbdrw-image --format json 2023-01-24T14:43:19.022 INFO:tasks.workunit.client.0.smithi179.stderr:+ LOCKS='[{"id":"rbdrw","locker":"client.4188","address":"172.21.15.179:0/2559857169"}]' 2023-01-24T14:43:19.022 INFO:tasks.workunit.client.0.smithi179.stderr:+ sleep 1 2023-01-24T14:43:20.023 INFO:tasks.workunit.client.0.smithi179.stderr:+ '[' '[{"id":"rbdrw","locker":"client.4188","address":"172.21.15.179:0/2559857169"}]' == '[]' ']' 2023-01-24T14:43:20.024 INFO:tasks.workunit.client.0.smithi179.stderr:++ rbd lock list rbdrw-image 2023-01-24T14:43:20.024 INFO:tasks.workunit.client.0.smithi179.stderr:++ tail -1 2023-01-24T14:43:20.024 INFO:tasks.workunit.client.0.smithi179.stderr:++ awk '{print $NF;}' 2023-01-24T14:43:20.175 INFO:tasks.workunit.client.0.smithi179.stderr:+ clientaddr=172.21.15.179:0/2559857169 2023-01-24T14:43:20.176 INFO:tasks.workunit.client.0.smithi179.stderr:++ rbd lock list rbdrw-image 2023-01-24T14:43:20.176 INFO:tasks.workunit.client.0.smithi179.stderr:++ tail -1 2023-01-24T14:43:20.176 INFO:tasks.workunit.client.0.smithi179.stderr:++ awk '{print $1;}' 2023-01-24T14:43:20.258 INFO:tasks.workunit.client.0.smithi179.stderr:+ clientid=client.4188 2023-01-24T14:43:20.259 INFO:tasks.workunit.client.0.smithi179.stdout:clientaddr: 172.21.15.179:0/2559857169 2023-01-24T14:43:20.259 INFO:tasks.workunit.client.0.smithi179.stdout:clientid: client.4188 2023-01-24T14:43:20.260 INFO:tasks.workunit.client.0.smithi179.stderr:+ echo 'clientaddr: 172.21.15.179:0/2559857169' 2023-01-24T14:43:20.260 INFO:tasks.workunit.client.0.smithi179.stderr:+ echo 'clientid: client.4188' 2023-01-24T14:43:20.260 INFO:tasks.workunit.client.0.smithi179.stderr:+ ceph osd blocklist add 172.21.15.179:0/2559857169 2023-01-24T14:43:21.597 INFO:tasks.workunit.client.0.smithi179.stderr:blocklisting 172.21.15.179:0/2559857169 until 2023-01-24T15:43:20.720266+0000 (3600 sec) 2023-01-24T14:43:21.607 INFO:tasks.workunit.client.0.smithi179.stderr:+ wait 106167 2023-01-24T14:56:25.001 INFO:tasks.ceph.osd.0.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x2acc1 0x2b4c3 0x2b832 0x28adf 0xb4614 0x445f3527 0x445f875d 0x36aba437 0x3618baa7 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T14:56:25.002 INFO:tasks.ceph.osd.0.smithi179.stderr:kernel callstack: 2023-01-24T14:56:25.002 INFO:tasks.ceph.osd.0.smithi179.stderr:kernel callstack: 2023-01-24T14:58:01.434 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x3b712bd2 0x3b71318c 0x39842454 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T14:58:01.434 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 0xffffffffffffff80 0xffffffffb111f69e 0xffffffffb1121b06 0xffffffffb19f08e9 0xffffffffb19f0ed5 0xffffffffb112604a 0xffffffffb10043ab 0xffffffffb1a000a9 2023-01-24T14:58:01.435 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T14:58:01.435 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T14:58:01.435 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T14:58:01.435 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T14:58:01.436 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T14:58:01.436 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T14:58:01.436 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T14:58:01.436 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T14:58:01.436 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T14:58:01.437 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T14:58:16.989 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 34 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x2b3df 0x2b832 0x28adf 0xb732c 0x375a2908 0x3a34b133 0x383e45e5 0x36638f66 0x362de4b7 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T14:58:16.989 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T14:58:32.809 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x2c5f1 0x2857c 0xb62c3 0x35f61d7f 0x35f646f1 0x3b712963 0x3b71318c 0x39842454 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T14:58:32.810 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 0xffffffffffffff80 0xffffffffb12dfe21 0xffffffffb107c83a 0xffffffffb107cb47 0xffffffffb1a0114e 2023-01-24T14:59:19.745 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x37713fa7 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T14:59:19.746 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:01:22.523 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x3b712bd2 0x3b71318c 0x39842454 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:01:22.524 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:01:45.528 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x3b712d45 0x3b71318c 0x39842454 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:01:45.528 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:02:09.039 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 35 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x37714187 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:02:09.040 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:02:47.798 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x377141c8 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:02:47.799 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:02:47.799 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:02:47.799 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:02:50.160 INFO:tasks.ceph.osd.0.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x39761 0x943d7 0x94ce6 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36afd000 0x36b607c8 0x36b65635 0x36b679b8 0x36b684ad 0x36b688a6 0x36b69106 0x36b69fd5 0x36fea258 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:02:50.160 INFO:tasks.ceph.osd.0.smithi179.stderr:kernel callstack: 2023-01-24T15:03:53.045 INFO:tasks.ceph.osd.0.smithi179.stderr:Reactor stalled for 34 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x9443b 0x94ce6 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36afd000 0x36b607c8 0x36b65635 0x36b679b8 0x36b684ad 0x36b688a6 0x36b69106 0x36b69fd5 0x36fea258 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:03:53.046 INFO:tasks.ceph.osd.0.smithi179.stderr:kernel callstack: 2023-01-24T15:05:37.496 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x37713f85 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:05:37.497 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:05:37.497 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:05:37.497 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:05:37.497 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:05:45.162 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 34 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x2b45b 0x2b832 0x28adf 0xb4614 0x39d00246 0x39ca706b 0x3c254721 0x3c254925 0x3c254925 0x3c254925 0x44bee5ec 0x44beee79 0x362c6842 0x362dcdef 0x362dda11 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:05:45.163 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:06:39.760 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x377142d2 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:06:39.760 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:08:58.468 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 34 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x94379 0x94ce6 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:08:58.469 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:08:58.469 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:08:58.469 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:08:58.469 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:08:58.469 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:09:05.926 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x9443b 0x94ce6 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:09:05.926 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:09:21.340 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x37714292 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:09:21.340 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:09:21.341 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:09:29.393 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x39842203 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:09:29.393 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:09:37.159 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 34 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x3b713082 0x3b71318c 0x39842454 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:09:37.159 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:09:52.919 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x37714065 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:09:52.919 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:09:52.919 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:10:00.574 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0xc9774 0x94409 0x94ce6 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:10:00.574 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:10:17.982 INFO:tasks.ceph.osd.1.smithi179.stderr:Rate-limit: suppressed 2 backtraces on shard 0 2023-01-24T15:10:23.923 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x37714187 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:10:23.923 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:10:23.923 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:10:23.924 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:10:39.595 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x39761 0x943d7 0x94ce6 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:10:39.595 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:10:47.348 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x43c93420 0x43c93a0b 0x3d727f5b 0x3677c1a9 0x36bb1bd1 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:10:47.348 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:13:37.278 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x39761 0x943d7 0x94ce6 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:13:37.279 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:13:37.279 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 0xffffffffffffff80 0xffffffffb100436e 0xffffffffb1a000a9 2023-01-24T15:13:37.279 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:13:37.279 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 0xffffffffffffff80 0xffffffffb100436e 0xffffffffb1a000a9 2023-01-24T15:13:37.280 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:13:37.280 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:14:08.147 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x377140d7 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:14:08.147 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:14:09.889 INFO:tasks.ceph.osd.0.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x37714313 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36afd000 0x36b607c8 0x36b65635 0x36b679b8 0x36b684ad 0x36b688a6 0x36b69106 0x36b69fd5 0x36fea258 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:14:09.889 INFO:tasks.ceph.osd.0.smithi179.stderr:kernel callstack: 2023-01-24T15:14:09.889 INFO:tasks.ceph.osd.0.smithi179.stderr:kernel callstack: 2023-01-24T15:14:23.836 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x3771409c 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:14:23.836 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:16:05.661 INFO:tasks.ceph.osd.0.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0xc95af 0x94409 0x94ce6 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36afd000 0x36b607c8 0x36b65635 0x36b679b8 0x36b684ad 0x36b688a6 0x36b69106 0x36b69fd5 0x36fea258 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:16:05.661 INFO:tasks.ceph.osd.0.smithi179.stderr:kernel callstack: 2023-01-24T15:16:05.661 INFO:tasks.ceph.osd.0.smithi179.stderr:kernel callstack: 2023-01-24T15:16:42.330 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x37714a35 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:16:42.331 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 0xffffffffffffff80 0xffffffffb107c919 0xffffffffb107cb47 0xffffffffb1a0114e 2023-01-24T15:16:42.331 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:16:42.331 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:16:42.331 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:16:48.066 INFO:tasks.ceph.osd.0.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x3d17f97e 0x3d199cc6 0x3d0ca56e 0x3d0cdcba 0x3d0d7087 0x3d0dd724 0x3d0ddbf9 0x3d0e1631 0x3d0e1ab1 0x3d08c33c 0x3d0915e9 0x375eb7ca 0x375a4d03 0x375aaad1 0x3755da37 0x372b9694 0x36b001fe 0x36b607c8 0x36b65635 0x36b679b8 0x36b684ad 0x36b688a6 0x36b69106 0x36b69fd5 0x36fea258 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:16:48.066 INFO:tasks.ceph.osd.0.smithi179.stderr:kernel callstack: 2023-01-24T15:16:50.338 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x2c697 0x2857c 0xb62c3 0x35f61d7f 0x35f646f1 0x3b712963 0x3b71318c 0x39842454 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:16:50.338 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:16:55.230 INFO:tasks.ceph.osd.0.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x39ca89db 0x3c249e52 0x3c252879 0x3d0cdcec 0x3d0d7087 0x3d0dd724 0x3d0ddbf9 0x3d0e1631 0x3d0e1ab1 0x3d08c33c 0x3d0915e9 0x375eb7ca 0x375a4d03 0x375aaad1 0x3755da37 0x372b9694 0x36b001fe 0x36b607c8 0x36b65635 0x36b679b8 0x36b684ad 0x36b688a6 0x36b69106 0x36b69fd5 0x36fea258 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:16:55.230 INFO:tasks.ceph.osd.0.smithi179.stderr:kernel callstack: 2023-01-24T15:16:57.715 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x2b3db 0x2b832 0x28adf 0xb732c 0x398421a1 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:16:57.715 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:17:05.524 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x3977c 0x943d7 0x94ce6 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:17:05.524 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:17:13.442 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x3771407b 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:17:13.442 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:17:29.241 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x37714ed2 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:17:29.241 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:17:29.241 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:17:52.781 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x3973f 0x943d7 0x94ce6 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:17:52.781 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:18:00.436 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x94cae 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:18:00.436 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:18:08.373 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x94ccf 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:18:08.373 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:18:15.866 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x3b712bdb 0x3b71318c 0x39842454 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:18:15.866 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:18:54.613 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x3771417d 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:18:54.613 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:18:58.198 INFO:tasks.ceph.osd.0.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x94cab 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36afd000 0x36b607c8 0x36b65635 0x36b679b8 0x36b684ad 0x36b688a6 0x36b69106 0x36b69fd5 0x36fea258 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:18:58.198 INFO:tasks.ceph.osd.0.smithi179.stderr:kernel callstack: 2023-01-24T15:19:02.121 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 34 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x11a16 0x3ff1b 0xb8b25 0xf67fd 0xf6c7b 0xf496c 0x11ada1 0x4518b128 0x3627c645 0x36bb1977 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:19:02.121 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 0xffffffffffffff80 0xffffffffc0597550 0xffffffffc074f236 0xffffffffb1395291 0xffffffffb137e06a 0xffffffffb137e5b5 0xffffffffb129b95c 0xffffffffc075b8e5 0xffffffffb135d462 0xffffffffb1360c55 0xffffffffb1360edf 0xffffffffb10043ab 0xffffffffb1a000a9 2023-01-24T15:19:57.812 INFO:tasks.ceph.osd.0.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0xc37f8 0xc0f00 0xb7393 0x398421a1 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36afd000 0x36b607c8 0x36b65635 0x36b679b8 0x36b684ad 0x36b688a6 0x36b69106 0x36b69fd5 0x36fea258 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:19:57.812 INFO:tasks.ceph.osd.0.smithi179.stderr:kernel callstack: 2023-01-24T15:20:50.637 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x377141c8 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:20:50.637 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:20:50.637 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:20:50.637 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:20:50.638 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:20:50.638 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:20:58.443 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x2acf0 0x2b4c3 0x2b832 0x28adf 0xb732c 0x398421a1 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:20:58.443 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:21:06.016 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0xd5985 0xd548b 0x287b7 0xb732c 0x398421a1 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:21:06.016 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:21:13.910 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x2b3df 0x2b832 0x28adf 0xb732c 0x398421a1 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:21:13.910 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:21:21.697 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0xf7cd 0xc37ec 0xc0f00 0xb7393 0x398421a1 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:21:21.698 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:21:25.510 INFO:tasks.ceph.osd.0.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x37713fe5 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36afd000 0x36b607c8 0x36b65635 0x36b679b8 0x36b684ad 0x36b688a6 0x36b69106 0x36b69fd5 0x36fea258 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:21:25.510 INFO:tasks.ceph.osd.0.smithi179.stderr:kernel callstack: 2023-01-24T15:21:25.511 INFO:tasks.ceph.osd.0.smithi179.stderr:kernel callstack: 2023-01-24T15:21:29.585 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x398421f9 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:21:29.585 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:21:37.333 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x377140c4 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:21:37.333 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:22:00.660 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x3b712bd2 0x3b71318c 0x39842454 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:22:00.660 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:22:15.953 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x37714183 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:22:15.953 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:22:56.712 INFO:tasks.ceph.osd.0.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x2b350 0x2b832 0x28adf 0xb732c 0x398421a1 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36afd000 0x36b607c8 0x36b65635 0x36b679b8 0x36b684ad 0x36b688a6 0x36b69106 0x36b69fd5 0x36fea258 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:22:56.712 INFO:tasks.ceph.osd.0.smithi179.stderr:kernel callstack: 2023-01-24T15:22:56.712 INFO:tasks.ceph.osd.0.smithi179.stderr:kernel callstack: 2023-01-24T15:22:56.713 INFO:tasks.ceph.osd.0.smithi179.stderr:kernel callstack: 2023-01-24T15:22:56.713 INFO:tasks.ceph.osd.0.smithi179.stderr:kernel callstack: 2023-01-24T15:22:56.713 INFO:tasks.ceph.osd.0.smithi179.stderr:kernel callstack: 2023-01-24T15:23:01.989 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x94ceb 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:23:01.989 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:23:01.989 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:23:25.015 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0xba6df 0xc37ec 0xc15bc 0x2881c 0xb732c 0x398421a1 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:23:25.015 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:23:40.341 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x39761 0x943d7 0x94ce6 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:23:40.341 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:23:40.342 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:23:41.859 INFO:tasks.ceph.osd.0.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0xd5872 0xd5a00 0xd548b 0x287b7 0xb732c 0x398421a1 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36afd000 0x36b607c8 0x36b65635 0x36b679b8 0x36b684ad 0x36b688a6 0x36b69106 0x36b69fd5 0x36fea258 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:23:41.859 INFO:tasks.ceph.osd.0.smithi179.stderr:kernel callstack: 2023-01-24T15:24:10.817 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0xc18cf 0x3b64de43 0x3b64e399 0x3b6513c1 0x35f19188 0x3602c6e5 0x3602ef36 0x3602f68b 0x45189fad 0x4518acc4 0x3627c645 0x36bb1977 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:24:10.817 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:24:18.631 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x94c80 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:24:18.631 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:24:23.691 INFO:tasks.ceph.osd.0.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x37714371 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36afd000 0x36b607c8 0x36b65635 0x36b679b8 0x36b684ad 0x36b688a6 0x36b69106 0x36b69fd5 0x36fea258 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:24:23.691 INFO:tasks.ceph.osd.0.smithi179.stderr:kernel callstack: 2023-01-24T15:24:23.691 INFO:tasks.ceph.osd.0.smithi179.stderr:kernel callstack: 2023-01-24T15:24:26.255 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x37714371 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:24:26.255 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:24:34.098 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0xb7e11 0x2b490 0x2b832 0x28adf 0xb732c 0x398421a1 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:24:34.098 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:24:41.613 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x35f64947 0x3b712963 0x3b71318c 0x39842454 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:24:41.613 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:24:44.607 INFO:tasks.ceph.osd.0.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x11a16 0x3ff1b 0xb8b25 0xf67fd 0xf6c7b 0xf496c 0x11ada1 0x4518b128 0x372b80f1 0x36b001fe 0x36b607c8 0x36b65635 0x36b679b8 0x36b684ad 0x36b688a6 0x36b69106 0x36b69fd5 0x36fea258 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:24:44.607 INFO:tasks.ceph.osd.0.smithi179.stderr:kernel callstack: 0xffffffffffffff80 0xffffffffc05969a2 0xffffffffc0597811 0xffffffffc074f236 0xffffffffc0770f81 0xffffffffb129671e 0xffffffffb129b9c2 0xffffffffc075b8e5 0xffffffffb135d462 0xffffffffb1360c55 0xffffffffb1360edf 0xffffffffb10043ab 0xffffffffb1a000a9 2023-01-24T15:24:48.138 INFO:tasks.ceph.osd.0.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0xb7f77 0x2b490 0x2b832 0x28adf 0xb732c 0x3623ca23 0x3d1852e5 0x3d199cc6 0x3d0ca56e 0x3d0cdcba 0x3d0d7087 0x3d0dd724 0x3d0ddbf9 0x3d0e1631 0x3d0e1ab1 0x3d08c33c 0x3d0915e9 0x375eb7ca 0x375a4d03 0x375aaad1 0x3755da37 0x372b9694 0x36b001fe 0x36b607c8 0x36b65635 0x36b679b8 0x36b684ad 0x36b688a6 0x36b69106 0x36b69fd5 0x36fea258 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:24:48.139 INFO:tasks.ceph.osd.0.smithi179.stderr:kernel callstack: 2023-01-24T15:25:09.063 INFO:tasks.ceph.osd.0.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0xd0534 0x2b490 0x2b832 0x28adf 0xb732c 0x398421a1 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36afd000 0x36b607c8 0x36b65635 0x36b679b8 0x36b684ad 0x36b688a6 0x36b69106 0x36b69fd5 0x36fea258 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:25:09.063 INFO:tasks.ceph.osd.0.smithi179.stderr:kernel callstack: 2023-01-24T15:25:15.866 INFO:tasks.ceph.osd.0.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x2b45b 0x2b832 0x28adf 0xb732c 0x398421a1 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36afd000 0x36b607c8 0x36b65635 0x36b679b8 0x36b684ad 0x36b688a6 0x36b69106 0x36b69fd5 0x36fea258 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:25:15.866 INFO:tasks.ceph.osd.0.smithi179.stderr:kernel callstack: 2023-01-24T15:25:20.550 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x398421f9 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:25:20.550 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:25:20.550 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:25:28.310 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0xc96c7 0x94409 0x94ce6 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:25:28.311 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:25:43.616 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0xc97ec 0x94409 0x94ce6 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:25:43.616 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:25:51.452 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x39761 0x943d7 0x94ce6 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:25:51.452 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:26:37.721 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x3b712fab 0x3b71318c 0x39842454 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:26:37.722 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:26:37.722 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:26:53.682 INFO:tasks.ceph.osd.0.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x3771429d 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36afd000 0x36b607c8 0x36b65635 0x36b679b8 0x36b684ad 0x36b688a6 0x36b69106 0x36b69fd5 0x36fea258 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:26:53.682 INFO:tasks.ceph.osd.0.smithi179.stderr:kernel callstack: 2023-01-24T15:27:14.895 INFO:tasks.ceph.osd.0.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0xd5872 0xd5a00 0xd548b 0x287b7 0xb732c 0x398421a1 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36afd000 0x36b607c8 0x36b65635 0x36b679b8 0x36b684ad 0x36b688a6 0x36b69106 0x36b69fd5 0x36fea258 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:27:14.895 INFO:tasks.ceph.osd.0.smithi179.stderr:kernel callstack: 2023-01-24T15:28:10.147 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x35f64947 0x3b712963 0x3b71318c 0x39842454 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:28:10.147 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:28:17.818 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x36bb1dfe 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:28:17.818 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:28:25.548 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 34 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x2b45b 0x2b832 0x28adf 0xb4614 0x445f3527 0x445f875d 0x36aba437 0x3618baa7 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:28:25.549 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:28:33.493 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x2cd7b 0xb732c 0x398421a1 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:28:33.493 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:29:00.168 INFO:tasks.ceph.osd.0.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x37714187 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36afd000 0x36b607c8 0x36b65635 0x36b679b8 0x36b684ad 0x36b688a6 0x36b69106 0x36b69fd5 0x36fea258 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:29:00.168 INFO:tasks.ceph.osd.0.smithi179.stderr:kernel callstack: 2023-01-24T15:29:04.896 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x39761 0x943d7 0x94ce6 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:29:04.896 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:29:04.897 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:29:13.045 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 36 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x2cd7b 0xb732c 0x398421a1 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:29:13.045 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:29:28.315 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x3771409c 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:29:28.316 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:29:44.079 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x9438c 0x94ce6 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:29:44.079 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:29:59.437 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x3d6385d7 0x3d7266e8 0x3677c1a9 0x36bb1bd1 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:29:59.437 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:29:59.437 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:31:16.642 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x37714172 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:31:16.643 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:31:55.495 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x3771417d 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:31:55.495 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:31:55.495 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:31:55.496 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:32:03.425 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x39842203 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:32:03.425 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:32:10.854 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x37713ff9 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:32:10.855 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:32:18.772 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x3b712bd2 0x3b71318c 0x39842454 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:32:18.772 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:32:34.033 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0xb7f5f 0x2b490 0x2b832 0x28adf 0xb732c 0x398421a1 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:32:34.034 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:32:41.957 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x94c85 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:32:41.957 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:32:53.041 INFO:tasks.ceph.osd.0.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x37714704 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36afd000 0x36b607c8 0x36b65635 0x36b679b8 0x36b684ad 0x36b688a6 0x36b69106 0x36b69fd5 0x36fea258 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:32:53.041 INFO:tasks.ceph.osd.0.smithi179.stderr:kernel callstack: 2023-01-24T15:32:57.513 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x2c6f7 0x2857c 0xb62c3 0x35f61d7f 0x35f646f1 0x3b712963 0x3b71318c 0x39842454 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:32:57.513 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:33:05.448 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x39842636 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:33:05.449 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:33:21.053 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 37 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0xf78f 0xc37ec 0xc15bc 0x2b495 0x2b832 0x28adf 0xb4614 0x445f3527 0x445f875d 0x36aba437 0x3618baa7 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:33:21.053 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:33:36.551 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x3b712bd2 0x3b71318c 0x39842454 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:33:36.551 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:33:36.552 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:34:00.003 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x39842203 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:34:00.003 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:34:00.003 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:34:52.186 INFO:tasks.ceph.osd.0.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x3b712b97 0x3b71318c 0x39842454 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36afd000 0x36b607c8 0x36b65635 0x36b679b8 0x36b684ad 0x36b688a6 0x36b69106 0x36b69fd5 0x36fea258 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:34:52.186 INFO:tasks.ceph.osd.0.smithi179.stderr:kernel callstack: 2023-01-24T15:35:05.542 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x37714187 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:35:05.542 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:35:05.542 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:35:32.544 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x39761 0x943d7 0x94ce6 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:35:32.544 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:36:03.651 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x39790 0x943f6 0x94ce6 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:36:03.651 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:36:11.519 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x97daf 0x94e59 0x10d443 0xff717 0xffba3 0x3b64d6ee 0x3b64e399 0x3b6513c1 0x35f19188 0x3602c6e5 0x3602ef36 0x3602f68b 0x45189fad 0x4518acc4 0x3627c645 0x36bb1977 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:36:11.520 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:36:26.946 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x392a0f3c 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:36:26.946 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:36:34.618 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x2b350 0x2b832 0x28adf 0xb732c 0x398421a1 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:36:34.618 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:36:50.204 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x3b712bd2 0x3b71318c 0x39842454 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:36:50.204 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:36:58.052 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x377142a3 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:36:58.052 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:37:05.682 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 34 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x39742 0x943d7 0x94ce6 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:37:05.682 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:37:18.611 INFO:tasks.ceph.osd.1.smithi179.stderr:Rate-limit: suppressed 1 backtrace on shard 0 2023-01-24T15:37:21.283 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x2b3df 0x2b832 0x28adf 0xb4614 0x445f3527 0x445f875d 0x36aba437 0x3618baa7 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:37:21.283 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:37:21.283 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:37:29.269 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x943cf 0x94ce6 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:37:29.269 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:37:44.773 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x2b45b 0x2b832 0x28adf 0xb732c 0x398421a1 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:37:44.773 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:37:52.698 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x3b712f39 0x3b71318c 0x39842454 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:37:52.699 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:38:00.484 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0xc97c7 0x94409 0x94ce6 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:38:00.484 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:38:18.629 INFO:tasks.ceph.osd.1.smithi179.stderr:Rate-limit: suppressed 1 backtrace on shard 0 2023-01-24T15:38:19.645 INFO:tasks.ceph.osd.0.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0xc96c7 0x94409 0x94ce6 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36afd000 0x36b607c8 0x36b65635 0x36b679b8 0x36b684ad 0x36b688a6 0x36b69106 0x36b69fd5 0x36fea258 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:38:19.645 INFO:tasks.ceph.osd.0.smithi179.stderr:kernel callstack: 2023-01-24T15:38:23.542 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x2c6fc 0x2857c 0xb62c3 0x35f61d7f 0x35f646f1 0x3b712963 0x3b71318c 0x39842454 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:38:23.542 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:38:23.542 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:38:31.385 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x377140f6 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:38:31.385 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:38:38.945 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0xc96c7 0x94409 0x94ce6 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:38:38.945 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:39:10.127 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x37714062 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:39:10.127 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:39:10.127 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:39:47.228 INFO:tasks.ceph.osd.0.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x3b712bd2 0x3b71318c 0x39842454 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36afd000 0x36b607c8 0x36b65635 0x36b679b8 0x36b684ad 0x36b688a6 0x36b69106 0x36b69fd5 0x36fea258 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:39:47.228 INFO:tasks.ceph.osd.0.smithi179.stderr:kernel callstack: 2023-01-24T15:39:50.849 INFO:tasks.ceph.osd.0.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x377140ba 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36afd000 0x36b607c8 0x36b65635 0x36b679b8 0x36b684ad 0x36b688a6 0x36b69106 0x36b69fd5 0x36fea258 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:39:50.849 INFO:tasks.ceph.osd.0.smithi179.stderr:kernel callstack: 2023-01-24T15:40:32.654 INFO:tasks.ceph.osd.0.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0xb7e66 0x2b490 0x2b832 0x28adf 0xb4614 0x39d00246 0x39ca706b 0x3c254721 0x3c254925 0x3c254925 0x3c254925 0x3c254925 0x3c254925 0x3c254925 0x3c254925 0x3c254925 0x3c254925 0x3c254925 0x3c254925 0x3c254925 0x3c254925 0x3c254925 0x3c254925 0x3c254925 0x3c254925 0x3c254925 0x3c254925 0x3c254925 0x3c254925 0x44bee5ec 0x44beee79 0x362c6842 0x362dcdef 0x362dda11 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:40:32.655 INFO:tasks.ceph.osd.0.smithi179.stderr:kernel callstack: 2023-01-24T15:40:34.868 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x11b5a2 0x11bc8b 0x4518affa 0x3627c645 0x36bb1977 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:40:34.868 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:40:34.868 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:40:50.130 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x377141aa 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:40:50.130 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:40:57.962 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 34 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x94383 0x94ce6 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:40:57.962 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:41:29.102 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0xb7ea9 0x2b490 0x2b832 0x28adf 0xb732c 0x398421a1 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:41:29.103 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:41:29.103 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:41:44.866 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x3b712dcf 0x3b71318c 0x39842454 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:41:44.866 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:41:52.618 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0xb6254 0x35f61d7f 0x35f646f1 0x3b712963 0x3b71318c 0x39842454 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:41:52.618 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:42:00.292 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0xd04d3 0x2b490 0x2b832 0x28adf 0xb732c 0x398421a1 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:42:00.292 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:42:08.116 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x39842203 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:42:08.116 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:42:18.725 INFO:tasks.ceph.osd.1.smithi179.stderr:Rate-limit: suppressed 1 backtrace on shard 0 2023-01-24T15:42:23.392 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x3b7131a3 0x39842454 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:42:23.392 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:42:23.392 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:42:30.980 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x94cdb 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:42:30.980 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:42:52.708 INFO:tasks.ceph.osd.0.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x2ac8d 0x2b4c3 0x2b832 0x28adf 0xb732c 0x3c254a65 0x3c254925 0x3c254925 0x3c254925 0x3c254925 0x3c254925 0x3c254925 0x44bee5ec 0x44beee79 0x362c6842 0x362dcdef 0x362dda11 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:42:52.708 INFO:tasks.ceph.osd.0.smithi179.stderr:kernel callstack: 2023-01-24T15:43:39.415 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x3b712bdb 0x3b71318c 0x39842454 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:43:39.415 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:43:39.415 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:43:44.680 INFO:tasks.ceph.osd.0.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x39842203 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36afd000 0x36b607c8 0x36b65635 0x36b679b8 0x36b684ad 0x36b688a6 0x36b69106 0x36b69fd5 0x36fea258 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:43:44.680 INFO:tasks.ceph.osd.0.smithi179.stderr:kernel callstack: 2023-01-24T15:44:09.984 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x37714101 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:44:09.985 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:44:09.985 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:44:25.565 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x9435f 0x94421 0x94ce6 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:44:25.566 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:44:25.566 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:44:33.469 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x2c5c1 0x2857c 0xb62c3 0x35f61d7f 0x35f646f1 0x3b712963 0x3b71318c 0x39842454 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:44:33.469 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:44:41.300 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x39842203 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:44:41.300 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:44:49.040 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x2c642 0x2857c 0xb62c3 0x35f61d7f 0x35f646f1 0x3b712963 0x3b71318c 0x39842454 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:44:49.040 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:44:57.054 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x3771417d 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:44:57.054 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:45:12.263 INFO:tasks.ceph.osd.0.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x3978d 0x943f6 0x94ce6 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36afd000 0x36b607c8 0x36b65635 0x36b679b8 0x36b684ad 0x36b688a6 0x36b69106 0x36b69fd5 0x36fea258 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:45:12.263 INFO:tasks.ceph.osd.0.smithi179.stderr:kernel callstack: 2023-01-24T15:45:12.264 INFO:tasks.ceph.osd.0.smithi179.stderr:kernel callstack: 2023-01-24T15:45:18.754 INFO:tasks.ceph.osd.1.smithi179.stderr:Rate-limit: suppressed 1 backtrace on shard 0 2023-01-24T15:45:19.810 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x3b713145 0x39842454 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:45:19.810 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:45:19.810 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:45:27.676 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x398421f9 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:45:27.676 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:45:35.238 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x3977c 0x943d7 0x94ce6 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:45:35.238 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:45:58.270 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x11a16 0x3ff1b 0xb8b25 0xf67fd 0xf6c7b 0xf496c 0x11ada1 0x4518b128 0x3627c645 0x36bb1977 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:45:58.270 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 0xffffffffffffff80 0xffffffffb13a3c90 0xffffffffc0770f65 0xffffffffb129671e 0xffffffffb129b9c2 0xffffffffc075b8e5 0xffffffffb135d462 0xffffffffb1360c55 0xffffffffb1360edf 0xffffffffb10043ab 0xffffffffb1a000a9 2023-01-24T15:46:01.181 INFO:tasks.ceph.osd.0.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x3b712d2d 0x3b71318c 0x39842454 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36afd000 0x36b607c8 0x36b65635 0x36b679b8 0x36b684ad 0x36b688a6 0x36b69106 0x36b69fd5 0x36fea258 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:46:01.181 INFO:tasks.ceph.osd.0.smithi179.stderr:kernel callstack: 2023-01-24T15:46:01.674 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x3771422e 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:46:01.674 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:46:18.764 INFO:tasks.ceph.osd.1.smithi179.stderr:Rate-limit: suppressed 1 backtrace on shard 0 2023-01-24T15:47:04.263 INFO:tasks.ceph.osd.0.smithi179.stderr:Reactor stalled for 34 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x3771417d 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36afd000 0x36b607c8 0x36b65635 0x36b679b8 0x36b684ad 0x36b688a6 0x36b69106 0x36b69fd5 0x36fea258 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:47:04.264 INFO:tasks.ceph.osd.0.smithi179.stderr:kernel callstack: 2023-01-24T15:47:11.095 INFO:tasks.ceph.osd.0.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x94409 0x94ce6 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36afd000 0x36b607c8 0x36b65635 0x36b679b8 0x36b684ad 0x36b688a6 0x36b69106 0x36b69fd5 0x36fea258 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:47:11.095 INFO:tasks.ceph.osd.0.smithi179.stderr:kernel callstack: 2023-01-24T15:47:23.232 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0xc97e8 0x94409 0x94ce6 0x37714a1d 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:47:23.233 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:47:23.233 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:47:23.233 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:47:23.233 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:47:23.234 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:47:38.330 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0xd5da3 0xc0f8c 0xb7393 0x398421a1 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:47:38.330 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:47:45.806 INFO:tasks.ceph.osd.0.smithi179.stderr:Reactor stalled for 34 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x94409 0x94ce6 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36afd000 0x36b607c8 0x36b65635 0x36b679b8 0x36b684ad 0x36b688a6 0x36b69106 0x36b69fd5 0x36fea258 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:47:45.806 INFO:tasks.ceph.osd.0.smithi179.stderr:kernel callstack: 2023-01-24T15:47:46.017 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0xb7e3d 0x2b490 0x2b832 0x28adf 0xb732c 0x398421a1 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:47:46.017 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:48:00.184 INFO:tasks.ceph.osd.0.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x37714130 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36afd000 0x36b607c8 0x36b65635 0x36b679b8 0x36b684ad 0x36b688a6 0x36b69106 0x36b69fd5 0x36fea258 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:48:00.184 INFO:tasks.ceph.osd.0.smithi179.stderr:kernel callstack: 2023-01-24T15:48:07.066 INFO:tasks.ceph.osd.0.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x37714949 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36afd000 0x36b607c8 0x36b65635 0x36b679b8 0x36b684ad 0x36b688a6 0x36b69106 0x36b69fd5 0x36fea258 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:48:07.066 INFO:tasks.ceph.osd.0.smithi179.stderr:kernel callstack: 2023-01-24T15:48:32.752 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x3b712d2d 0x3b71318c 0x39842454 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:48:32.753 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:48:40.823 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x37713f9c 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:48:40.823 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:48:48.281 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0xd0534 0x2b490 0x2b832 0x28adf 0xb732c 0x398421a1 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:48:48.281 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:48:56.101 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x37714030 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:48:56.101 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:49:03.943 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x37714187 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:49:03.944 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:49:18.835 INFO:tasks.ceph.osd.1.smithi179.stderr:Rate-limit: suppressed 1 backtrace on shard 0 2023-01-24T15:49:19.279 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x3b712bd2 0x3b71318c 0x39842454 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:49:19.279 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:49:19.279 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:49:26.518 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x39761 0x943d7 0x94ce6 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:49:26.518 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:49:34.244 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x36df98e0 0x36f48c45 0x392a0fec 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:49:34.244 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:49:45.596 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x3b712d0f 0x3b71318c 0x39842454 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:49:45.596 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:49:49.619 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0xb7e78 0x2b490 0x2b832 0x28adf 0xb732c 0x3626723a 0x36b0719e 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:49:49.619 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:50:05.681 INFO:tasks.ceph.osd.0.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x2b45b 0x2b832 0x28adf 0xb732c 0x398421a1 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36afd000 0x36b607c8 0x36b65635 0x36b679b8 0x36b684ad 0x36b688a6 0x36b69106 0x36b69fd5 0x36fea258 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:50:05.682 INFO:tasks.ceph.osd.0.smithi179.stderr:kernel callstack: 2023-01-24T15:50:18.853 INFO:tasks.ceph.osd.1.smithi179.stderr:Rate-limit: suppressed 1 backtrace on shard 0 2023-01-24T15:50:28.034 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0xd5872 0xd5a00 0xd548b 0x2c64a 0x2857c 0xb62c3 0x36bb2465 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:50:28.034 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:50:28.035 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:50:43.706 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x9437b 0x94ce6 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:50:43.706 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:50:51.376 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x37714253 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:50:51.376 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:51:06.969 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0xcdf7a 0xd0f4e 0x2b4c3 0x2b832 0x28adf 0xb732c 0x398421a1 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:51:06.969 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 0xffffffffffffff80 0xffffffffb12e0a0a 0xffffffffb12e1b31 0xffffffffb12e1d0f 0xffffffffb12e2158 0xffffffffb12e21e7 0xffffffffb10043ab 0xffffffffb1a000a9 2023-01-24T15:51:14.560 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x3677c961 0x36bb1bd1 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:51:14.560 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:51:22.479 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x377143b2 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:51:22.479 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:51:26.376 INFO:tasks.ceph.osd.0.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0xc15af 0x2b495 0x2b832 0x28adf 0xb732c 0x398421a1 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36afd000 0x36b607c8 0x36b65635 0x36b679b8 0x36b684ad 0x36b688a6 0x36b69106 0x36b69fd5 0x36fea258 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:51:26.376 INFO:tasks.ceph.osd.0.smithi179.stderr:kernel callstack: 2023-01-24T15:51:26.377 INFO:tasks.ceph.osd.0.smithi179.stderr:kernel callstack: 2023-01-24T15:51:26.377 INFO:tasks.ceph.osd.0.smithi179.stderr:kernel callstack: 2023-01-24T15:51:30.002 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0xc97c7 0x94409 0x94ce6 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:51:30.002 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:51:37.794 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 34 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0xc96c3 0x94409 0x94ce6 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:51:37.795 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:51:45.780 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x3b712bd2 0x3b71318c 0x39842454 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:51:45.780 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:52:01.396 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x2acc1 0x2b4c3 0x2b832 0x28adf 0xb4614 0x445f3527 0x445f875d 0x36aba437 0x3618baa7 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:52:01.396 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:52:01.396 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:52:24.800 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0xb7eb0 0x2b490 0x2b832 0x28adf 0xb732c 0x398421a1 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:52:24.800 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:52:32.392 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x377147fc 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:52:32.392 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:52:40.252 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x2a841 0x2ab8e 0x27c98 0x2af57 0x2b4c3 0x2b832 0x28adf 0xb732c 0x398421a1 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:52:40.252 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:52:47.978 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x3977c 0x943d7 0x94ce6 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:52:47.978 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:52:55.899 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x39761 0x943d7 0x94ce6 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:52:55.899 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:53:18.551 INFO:tasks.ceph.osd.0.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0xc206f 0xc0f67 0xb632f 0x35f61d7f 0x35f646f1 0x3b712963 0x3b71318c 0x39842454 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36afd000 0x36b607c8 0x36b65635 0x36b679b8 0x36b684ad 0x36b688a6 0x36b69106 0x36b69fd5 0x36fea258 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:53:18.551 INFO:tasks.ceph.osd.0.smithi179.stderr:kernel callstack: 2023-01-24T15:53:18.551 INFO:tasks.ceph.osd.0.smithi179.stderr:kernel callstack: 2023-01-24T15:53:18.930 INFO:tasks.ceph.osd.1.smithi179.stderr:Rate-limit: suppressed 1 backtrace on shard 0 2023-01-24T15:53:21.883 INFO:tasks.ceph.osd.0.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x3b712dc1 0x3b71318c 0x39842454 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36afd000 0x36b607c8 0x36b65635 0x36b679b8 0x36b684ad 0x36b688a6 0x36b69106 0x36b69fd5 0x36fea258 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:53:21.883 INFO:tasks.ceph.osd.0.smithi179.stderr:kernel callstack: 2023-01-24T15:53:25.574 INFO:tasks.ceph.osd.0.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x2b45b 0x2b832 0x28adf 0xb732c 0x398421a1 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36afd000 0x36b607c8 0x36b65635 0x36b679b8 0x36b684ad 0x36b688a6 0x36b69106 0x36b69fd5 0x36fea258 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:53:25.575 INFO:tasks.ceph.osd.0.smithi179.stderr:kernel callstack: 2023-01-24T15:54:07.709 INFO:tasks.ceph.osd.0.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x2b350 0x2b832 0x28adf 0xb732c 0x398421a1 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36afd000 0x36b607c8 0x36b65635 0x36b679b8 0x36b684ad 0x36b688a6 0x36b69106 0x36b69fd5 0x36fea258 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:54:07.709 INFO:tasks.ceph.osd.0.smithi179.stderr:kernel callstack: 2023-01-24T15:54:09.158 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x37713f91 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:54:09.158 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:54:09.158 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:54:09.159 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:54:28.314 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0xc97e8 0x94409 0x94ce6 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:54:28.314 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:54:35.576 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x39761 0x943d7 0x94ce6 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:54:35.576 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:54:43.575 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x37714062 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:54:43.576 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:54:51.170 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x39842203 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:54:51.170 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:54:53.103 INFO:tasks.ceph.osd.0.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x94445 0x94ce6 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36afd000 0x36b607c8 0x36b65635 0x36b679b8 0x36b684ad 0x36b688a6 0x36b69106 0x36b69fd5 0x36fea258 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:54:53.103 INFO:tasks.ceph.osd.0.smithi179.stderr:kernel callstack: 2023-01-24T15:55:06.681 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x3771417d 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:55:06.681 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:55:22.116 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0xc97e8 0x94409 0x94ce6 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:55:22.116 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:55:29.761 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x3771404d 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:55:29.762 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:55:42.133 INFO:tasks.ceph.osd.0.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x94409 0x94ce6 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36afd000 0x36b607c8 0x36b65635 0x36b679b8 0x36b684ad 0x36b688a6 0x36b69106 0x36b69fd5 0x36fea258 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:55:42.134 INFO:tasks.ceph.osd.0.smithi179.stderr:kernel callstack: 2023-01-24T15:55:42.134 INFO:tasks.ceph.osd.0.smithi179.stderr:kernel callstack: 2023-01-24T15:55:45.265 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0xc97c7 0x94409 0x94ce6 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:55:45.265 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:55:52.655 INFO:tasks.ceph.osd.0.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x37714743 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36afd000 0x36b607c8 0x36b65635 0x36b679b8 0x36b684ad 0x36b688a6 0x36b69106 0x36b69fd5 0x36fea258 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:55:52.655 INFO:tasks.ceph.osd.0.smithi179.stderr:kernel callstack: 2023-01-24T15:56:01.028 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x3b712dc1 0x3b71318c 0x39842454 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:56:01.028 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:56:08.778 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 36 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0xc2077 0xc0f67 0xb7393 0x398421a1 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:56:08.778 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:56:18.988 INFO:tasks.ceph.osd.1.smithi179.stderr:Rate-limit: suppressed 1 backtrace on shard 0 2023-01-24T15:56:24.319 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x39761 0x943d7 0x94ce6 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:56:24.319 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:56:24.319 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:56:31.063 INFO:tasks.ceph.osd.0.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x3b712db2 0x3b71318c 0x39842454 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36afd000 0x36b607c8 0x36b65635 0x36b679b8 0x36b684ad 0x36b688a6 0x36b69106 0x36b69fd5 0x36fea258 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:56:31.063 INFO:tasks.ceph.osd.0.smithi179.stderr:kernel callstack: 2023-01-24T15:56:31.923 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x2cd27 0xb732c 0x398421a1 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:56:31.924 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:56:38.194 INFO:tasks.ceph.osd.0.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x3b713145 0x39842454 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36afd000 0x36b607c8 0x36b65635 0x36b679b8 0x36b684ad 0x36b688a6 0x36b69106 0x36b69fd5 0x36fea258 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:56:38.194 INFO:tasks.ceph.osd.0.smithi179.stderr:kernel callstack: 2023-01-24T15:56:39.887 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x943cf 0x94ce6 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:56:39.888 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:56:47.238 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x3b712f7e 0x3b71318c 0x39842454 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:56:47.239 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:57:02.827 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x36de5e0c 0x392a0574 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:57:02.828 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:57:19.017 INFO:tasks.ceph.osd.1.smithi179.stderr:Rate-limit: suppressed 2 backtraces on shard 0 2023-01-24T15:57:26.936 INFO:tasks.ceph.osd.0.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x2b3db 0x2b832 0x28adf 0xb732c 0x398421a1 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36afd000 0x36b607c8 0x36b65635 0x36b679b8 0x36b684ad 0x36b688a6 0x36b69106 0x36b69fd5 0x36fea258 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:57:26.937 INFO:tasks.ceph.osd.0.smithi179.stderr:kernel callstack: 2023-01-24T15:57:26.937 INFO:tasks.ceph.osd.0.smithi179.stderr:kernel callstack: 2023-01-24T15:57:33.887 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x3602ec43 0x3602f68b 0x4518aad9 0x3627c645 0x36bb1977 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:57:33.887 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:57:33.888 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:57:33.888 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:57:33.888 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:57:37.934 INFO:tasks.ceph.osd.0.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x2b350 0x2b832 0x28adf 0xb732c 0x398421a1 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36afd000 0x36b607c8 0x36b65635 0x36b679b8 0x36b684ad 0x36b688a6 0x36b69106 0x36b69fd5 0x36fea258 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:57:37.935 INFO:tasks.ceph.osd.0.smithi179.stderr:kernel callstack: 2023-01-24T15:57:41.587 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x37714027 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:57:41.587 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:58:09.494 INFO:tasks.ceph.osd.0.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x39842203 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36afd000 0x36b607c8 0x36b65635 0x36b679b8 0x36b684ad 0x36b688a6 0x36b69106 0x36b69fd5 0x36fea258 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:58:09.494 INFO:tasks.ceph.osd.0.smithi179.stderr:kernel callstack: 2023-01-24T15:58:32.635 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x36a505bc 0x36a510fe 0x36f56262 0x36b05938 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:58:32.635 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:58:32.635 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:58:32.635 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:58:32.636 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:58:32.636 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:58:41.021 INFO:tasks.ceph.osd.0.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x2b3db 0x2b832 0x28adf 0xb732c 0x398421a1 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36afd000 0x36b607c8 0x36b65635 0x36b679b8 0x36b684ad 0x36b688a6 0x36b69106 0x36b69fd5 0x36fea258 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:58:41.022 INFO:tasks.ceph.osd.0.smithi179.stderr:kernel callstack: 2023-01-24T15:58:44.569 INFO:tasks.ceph.osd.0.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x3771417d 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36afd000 0x36b607c8 0x36b65635 0x36b679b8 0x36b684ad 0x36b688a6 0x36b69106 0x36b69fd5 0x36fea258 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:58:44.569 INFO:tasks.ceph.osd.0.smithi179.stderr:kernel callstack: 2023-01-24T15:58:52.044 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0xa88ff 0xa7bed 0xffb94 0x3b64d6ee 0x3b64e399 0x3b6513c1 0x35f19188 0x3602c6e5 0x3602ef36 0x3602f68b 0x45189fad 0x4518acc4 0x3627c645 0x36bb1977 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:58:52.044 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:58:52.044 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:58:59.544 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x248ef 0x94421 0x94ce6 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:58:59.544 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:59:07.490 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x94c8e 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:59:07.490 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:59:15.217 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x397a9 0x943d7 0x94ce6 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:59:15.217 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:59:16.242 INFO:tasks.ceph.osd.0.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x2ac9b 0x2b4c3 0x2b832 0x28adf 0xb732c 0x398421a1 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36afd000 0x36b607c8 0x36b65635 0x36b679b8 0x36b684ad 0x36b688a6 0x36b69106 0x36b69fd5 0x36fea258 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:59:16.242 INFO:tasks.ceph.osd.0.smithi179.stderr:kernel callstack: 2023-01-24T15:59:16.243 INFO:tasks.ceph.osd.0.smithi179.stderr:kernel callstack: 2023-01-24T15:59:22.992 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x3771428f 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:59:22.992 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:59:30.662 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x11a16 0x3ff1b 0xb8b25 0xf67fd 0xf6c7b 0xf496c 0x11ada1 0x4518b128 0x3627c645 0x36bb1977 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:59:30.663 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 0xffffffffffffff80 0xffffffffc074f231 0xffffffffb1395291 0xffffffffb137e06a 0xffffffffb137e5b5 0xffffffffb129b95c 0xffffffffc075b8e5 0xffffffffb135d462 0xffffffffb1360c55 0xffffffffb1360edf 0xffffffffb10043ab 0xffffffffb1a000a9 2023-01-24T15:59:38.328 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x35f64517 0x3b712963 0x3b71318c 0x39842454 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:59:38.328 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T15:59:53.812 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x37714f2a 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T15:59:53.812 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:00:01.737 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x37714027 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:00:01.737 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:00:19.039 INFO:tasks.ceph.osd.1.smithi179.stderr:Rate-limit: suppressed 2 backtraces on shard 0 2023-01-24T16:00:23.312 INFO:tasks.ceph.osd.0.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0xc0f1a 0xb7393 0x398421a1 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36afd000 0x36b607c8 0x36b65635 0x36b679b8 0x36b684ad 0x36b688a6 0x36b69106 0x36b69fd5 0x36fea258 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:00:23.313 INFO:tasks.ceph.osd.0.smithi179.stderr:kernel callstack: 2023-01-24T16:00:25.404 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x94442 0x94ce6 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:00:25.404 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:00:25.404 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:00:25.404 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:00:32.686 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x3b712de3 0x3b71318c 0x39842454 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:00:32.686 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:00:40.739 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x28829 0xb732c 0x398421a1 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:00:40.740 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:01:11.498 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0xc9774 0x94409 0x94ce6 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:01:11.498 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:01:19.219 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x3973f 0x943f6 0x94ce6 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:01:19.219 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:01:26.943 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x37714269 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:01:26.943 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:01:34.707 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x39842203 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:01:34.707 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:01:42.167 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x3771498b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:01:42.167 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:01:45.941 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 34 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x2b3db 0x2b832 0x28adf 0xb4614 0x445f3527 0x445f875d 0x36aba437 0x3618baa7 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:01:45.941 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:02:15.586 INFO:tasks.ceph.osd.0.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x2cb19 0x2857c 0xb62c3 0x35f61d7f 0x35f646f1 0x3b712963 0x3b71318c 0x39842454 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36afd000 0x36b607c8 0x36b65635 0x36b679b8 0x36b684ad 0x36b688a6 0x36b69106 0x36b69fd5 0x36fea258 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:02:15.587 INFO:tasks.ceph.osd.0.smithi179.stderr:kernel callstack: 2023-01-24T16:02:19.066 INFO:tasks.ceph.osd.1.smithi179.stderr:Rate-limit: suppressed 2 backtraces on shard 0 2023-01-24T16:02:21.270 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x39761 0x943d7 0x94ce6 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:02:21.271 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:02:21.271 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:02:21.271 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:02:21.272 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:02:21.272 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:02:28.601 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x94401 0x94ce6 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:02:28.602 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:02:36.604 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x37714114 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:02:36.604 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:02:44.273 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x36a5130f 0x36f56262 0x36b05938 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:02:44.273 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:02:51.957 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x29114 0x2c49c 0x2857c 0xb62c3 0x35f61d7f 0x35f646f1 0x3b712963 0x3b71318c 0x39842454 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:02:51.958 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:03:19.081 INFO:tasks.ceph.osd.1.smithi179.stderr:Rate-limit: suppressed 3 backtraces on shard 0 2023-01-24T16:03:22.801 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0xd5898 0xd5a00 0xd548b 0x287b7 0xb732c 0x398421a1 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:03:22.802 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:03:22.802 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:03:22.802 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:03:22.802 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:03:30.478 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x2b5c2 0x2b832 0x28adf 0xb732c 0x398421a1 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:03:30.478 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:03:39.569 INFO:tasks.ceph.osd.0.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x94ccf 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36afd000 0x36b607c8 0x36b65635 0x36b679b8 0x36b684ad 0x36b688a6 0x36b69106 0x36b69fd5 0x36fea258 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:03:39.570 INFO:tasks.ceph.osd.0.smithi179.stderr:kernel callstack: 2023-01-24T16:03:46.123 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x39842203 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:03:46.124 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:03:46.124 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:03:53.954 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x37714062 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:03:53.955 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:04:01.802 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x94393 0x94ce6 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:04:01.802 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:04:11.478 INFO:tasks.ceph.osd.0.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x3771417d 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36afd000 0x36b607c8 0x36b65635 0x36b679b8 0x36b684ad 0x36b688a6 0x36b69106 0x36b69fd5 0x36fea258 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:04:11.479 INFO:tasks.ceph.osd.0.smithi179.stderr:kernel callstack: 2023-01-24T16:04:25.466 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x3b712bd2 0x3b71318c 0x39842454 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:04:25.467 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:04:25.467 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:04:25.467 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:04:40.803 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x3771417d 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:04:40.803 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:04:48.721 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x94cab 0x37714a1d 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:04:48.721 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:04:56.727 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x377146d3 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:04:56.727 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:05:04.473 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0xb7e78 0x2b490 0x2b832 0x28adf 0xb732c 0x398421a1 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:05:04.474 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:05:19.122 INFO:tasks.ceph.osd.1.smithi179.stderr:Rate-limit: suppressed 1 backtrace on shard 0 2023-01-24T16:05:20.015 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 34 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x3771429d 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:05:20.015 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:05:20.016 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:05:21.759 INFO:tasks.ceph.osd.0.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0xf7b2 0xc37ec 0xc15bc 0x2b495 0x2b832 0x28adf 0xb732c 0x398421a1 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36afd000 0x36b607c8 0x36b65635 0x36b679b8 0x36b684ad 0x36b688a6 0x36b69106 0x36b69fd5 0x36fea258 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:05:21.760 INFO:tasks.ceph.osd.0.smithi179.stderr:kernel callstack: 2023-01-24T16:05:27.839 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x94c85 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:05:27.840 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:06:09.831 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0xc96c3 0x94409 0x94ce6 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:06:09.831 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 0xffffffffffffff80 0xffffffffb13259fd 0xffffffffb13b452c 0xffffffffb13b8f5e 0xffffffffb13b4910 0xffffffffb13b6524 0xffffffffb13b6e4e 0xffffffffb10043ab 0xffffffffb1a000a9 2023-01-24T16:06:09.831 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:06:17.655 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x3b712fab 0x3b71318c 0x39842454 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:06:17.656 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:06:25.656 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x35f6453d 0x3b712963 0x3b71318c 0x39842454 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:06:25.656 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:06:28.403 INFO:tasks.ceph.osd.0.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x39793 0x943d7 0x94ce6 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36afd000 0x36b607c8 0x36b65635 0x36b679b8 0x36b684ad 0x36b688a6 0x36b69106 0x36b69fd5 0x36fea258 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:06:28.404 INFO:tasks.ceph.osd.0.smithi179.stderr:kernel callstack: 2023-01-24T16:06:35.421 INFO:tasks.ceph.osd.0.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x37714223 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36afd000 0x36b607c8 0x36b65635 0x36b679b8 0x36b684ad 0x36b688a6 0x36b69106 0x36b69fd5 0x36fea258 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:06:35.422 INFO:tasks.ceph.osd.0.smithi179.stderr:kernel callstack: 2023-01-24T16:07:10.621 INFO:tasks.ceph.osd.0.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0xc96c7 0x94409 0x94ce6 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36afd000 0x36b607c8 0x36b65635 0x36b679b8 0x36b684ad 0x36b688a6 0x36b69106 0x36b69fd5 0x36fea258 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:07:10.621 INFO:tasks.ceph.osd.0.smithi179.stderr:kernel callstack: 2023-01-24T16:07:13.864 INFO:tasks.ceph.osd.0.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x3b712bd2 0x3b71318c 0x39842454 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36afd000 0x36b607c8 0x36b65635 0x36b679b8 0x36b684ad 0x36b688a6 0x36b69106 0x36b69fd5 0x36fea258 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:07:13.864 INFO:tasks.ceph.osd.0.smithi179.stderr:kernel callstack: 2023-01-24T16:07:15.738 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x37714292 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:07:15.739 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:07:15.739 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:07:15.739 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:07:15.740 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:07:23.378 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x35f64517 0x3b712963 0x3b71318c 0x39842454 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:07:23.378 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:07:28.094 INFO:tasks.ceph.osd.0.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x2b350 0x2b832 0x28adf 0xb732c 0x398421a1 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36afd000 0x36b607c8 0x36b65635 0x36b679b8 0x36b684ad 0x36b688a6 0x36b69106 0x36b69fd5 0x36fea258 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:07:28.095 INFO:tasks.ceph.osd.0.smithi179.stderr:kernel callstack: 2023-01-24T16:07:31.464 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0xc96c7 0x94409 0x94ce6 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:07:31.465 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:07:38.887 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x3b712bd2 0x3b71318c 0x39842454 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:07:38.888 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:07:46.562 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x398421f9 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:07:46.562 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:07:54.328 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x37714187 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:07:54.328 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:08:10.274 INFO:tasks.ceph.osd.0.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x3b712bd2 0x3b71318c 0x39842454 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36afd000 0x36b607c8 0x36b65635 0x36b679b8 0x36b684ad 0x36b688a6 0x36b69106 0x36b69fd5 0x36fea258 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:08:10.275 INFO:tasks.ceph.osd.0.smithi179.stderr:kernel callstack: 2023-01-24T16:08:19.184 INFO:tasks.ceph.osd.1.smithi179.stderr:Rate-limit: suppressed 2 backtraces on shard 0 2023-01-24T16:08:25.593 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x3b712ddd 0x3b71318c 0x39842454 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:08:25.593 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:08:25.593 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:08:25.593 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:08:25.594 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:08:33.490 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0xc1fff 0xc0f72 0xb632f 0x35f61d7f 0x35f646f1 0x3b712963 0x3b71318c 0x39842454 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:08:33.490 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:08:41.156 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 36 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x37714193 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:08:41.157 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:08:48.911 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x377140da 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:08:48.911 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:08:56.594 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x2c5c1 0x2857c 0xb62c3 0x35f61d7f 0x35f646f1 0x3b712963 0x3b71318c 0x39842454 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:08:56.594 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:09:19.217 INFO:tasks.ceph.osd.1.smithi179.stderr:Rate-limit: suppressed 2 backtraces on shard 0 2023-01-24T16:09:20.029 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 35 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x943b7 0x94ce6 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:09:20.029 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:09:20.029 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:09:20.029 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:09:27.749 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x37714371 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:09:27.749 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:09:35.291 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x39761 0x943d7 0x94ce6 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:09:35.291 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:10:10.107 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x35f6456a 0x3b712963 0x3b71318c 0x39842454 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:10:10.107 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:10:18.052 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x248ef 0x94421 0x94ce6 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:10:18.053 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:10:41.670 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x3771417d 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:10:41.670 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:10:41.670 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:10:41.671 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:10:45.155 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0xc96c3 0x94409 0x94ce6 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:10:45.155 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:10:49.178 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 37 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x2b3db 0x2b832 0x28adf 0xb4614 0x445f3527 0x445f875d 0x36aba437 0x3618baa7 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:10:49.178 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:10:53.172 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0xc97ec 0x94409 0x94ce6 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:10:53.172 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:11:16.607 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x94c92 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:11:16.607 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 0xffffffffffffff80 0xffffffffb112622c 0xffffffffb11a442d 0xffffffffb11a6e16 0xffffffffb11a7705 0xffffffffb10043ab 0xffffffffb1a000a9 2023-01-24T16:11:16.607 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:11:23.500 INFO:tasks.ceph.osd.0.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x3b712bd2 0x3b71318c 0x39842454 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36afd000 0x36b607c8 0x36b65635 0x36b679b8 0x36b684ad 0x36b688a6 0x36b69106 0x36b69fd5 0x36fea258 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:11:23.500 INFO:tasks.ceph.osd.0.smithi179.stderr:kernel callstack: 2023-01-24T16:11:24.033 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x3771417d 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:11:24.033 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:11:39.625 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x9437b 0x94ce6 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:11:39.625 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:11:46.990 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0xb7f77 0x2c8ec 0x2857c 0xb62c3 0x35f61d7f 0x35f646f1 0x3b712963 0x3b71318c 0x39842454 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:11:46.990 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:11:51.706 INFO:tasks.ceph.osd.0.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x3771423f 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36afd000 0x36b607c8 0x36b65635 0x36b679b8 0x36b684ad 0x36b688a6 0x36b69106 0x36b69fd5 0x36fea258 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:11:51.706 INFO:tasks.ceph.osd.0.smithi179.stderr:kernel callstack: 2023-01-24T16:11:54.993 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x3b712de0 0x3b71318c 0x39842454 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:11:54.993 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:12:02.428 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0xd5874 0xd5a00 0xd548b 0x287b7 0xb732c 0x398421a1 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:12:02.428 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:12:19.276 INFO:tasks.ceph.osd.1.smithi179.stderr:Rate-limit: suppressed 1 backtrace on shard 0 2023-01-24T16:12:25.963 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x398423b8 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:12:25.963 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:12:25.964 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:12:25.964 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:12:49.382 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x3973f 0x943f6 0x94ce6 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:12:49.382 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:12:49.382 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:12:57.158 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 34 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0xd6585 0xc0f8c 0xb632f 0x35f61d7f 0x35f646f1 0x3b712963 0x3b71318c 0x39842454 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:12:57.159 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:13:08.951 INFO:tasks.ceph.osd.0.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x2b343 0x2b832 0x28adf 0xb732c 0x398421a1 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36afd000 0x36b607c8 0x36b65635 0x36b679b8 0x36b684ad 0x36b688a6 0x36b69106 0x36b69fd5 0x36fea258 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:13:08.951 INFO:tasks.ceph.osd.0.smithi179.stderr:kernel callstack: 2023-01-24T16:13:08.951 INFO:tasks.ceph.osd.0.smithi179.stderr:kernel callstack: 2023-01-24T16:13:20.099 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x94c80 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:13:20.100 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:13:28.174 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x3b712bd2 0x3b71318c 0x39842454 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:13:28.174 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:13:35.517 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x3b712bdb 0x3b71318c 0x39842454 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:13:35.517 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:13:43.367 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x39761 0x943d7 0x94ce6 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:13:43.367 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:13:58.545 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 35 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0xd592f 0xd548b 0x287b7 0xb732c 0x398421a1 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:13:58.545 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:14:11.807 INFO:tasks.ceph.osd.0.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0xc97ec 0x94409 0x94ce6 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36afd000 0x36b607c8 0x36b65635 0x36b679b8 0x36b684ad 0x36b688a6 0x36b69106 0x36b69fd5 0x36fea258 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:14:11.808 INFO:tasks.ceph.osd.0.smithi179.stderr:kernel callstack: 2023-01-24T16:14:53.931 INFO:tasks.ceph.osd.0.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x29114 0x2c4ab 0x2857c 0xb62c3 0x35f61d7f 0x35f646f1 0x3b712963 0x3b71318c 0x39842454 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36afd000 0x36b607c8 0x36b65635 0x36b679b8 0x36b684ad 0x36b688a6 0x36b69106 0x36b69fd5 0x36fea258 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:14:53.931 INFO:tasks.ceph.osd.0.smithi179.stderr:kernel callstack: 2023-01-24T16:14:56.680 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0xc97e8 0x94409 0x94ce6 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:14:56.680 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:15:00.294 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x3771417d 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:15:00.294 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:15:07.870 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x3677a09b 0x3677c6e4 0x36bb1bd1 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:15:07.870 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:15:15.542 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0xcdf7a 0xd0f4e 0x2b4c3 0x2b832 0x28adf 0xb732c 0x398421a1 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:15:15.542 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 0xffffffffffffff80 0xffffffffb12ec1f1 0xffffffffb12d75c9 0xffffffffb12d8140 0xffffffffb12dfc28 0xffffffffb12e1e6d 0xffffffffb12e2158 0xffffffffb12e21e7 0xffffffffb10043ab 0xffffffffb1a000a9 2023-01-24T16:15:18.632 INFO:tasks.ceph.osd.0.smithi179.stderr:Reactor stalled for 35 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x3771417d 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36afd000 0x36b607c8 0x36b65635 0x36b679b8 0x36b684ad 0x36b688a6 0x36b69106 0x36b69fd5 0x36fea258 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:15:18.632 INFO:tasks.ceph.osd.0.smithi179.stderr:kernel callstack: 2023-01-24T16:15:23.380 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x39761 0x943d7 0x94ce6 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:15:23.380 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:15:30.888 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x3977c 0x943d7 0x94ce6 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:15:30.888 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:15:38.727 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x39842203 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:15:38.727 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:15:46.398 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x35f64565 0x3b712963 0x3b71318c 0x39842454 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:15:46.398 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:15:54.033 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x3b712daf 0x3b71318c 0x39842454 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:15:54.033 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:16:00.738 INFO:tasks.ceph.osd.0.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x2ac8d 0x2b4c3 0x2b832 0x28adf 0xb732c 0x398421a1 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36afd000 0x36b607c8 0x36b65635 0x36b679b8 0x36b684ad 0x36b688a6 0x36b69106 0x36b69fd5 0x36fea258 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:16:00.738 INFO:tasks.ceph.osd.0.smithi179.stderr:kernel callstack: 2023-01-24T16:16:19.360 INFO:tasks.ceph.osd.1.smithi179.stderr:Rate-limit: suppressed 2 backtraces on shard 0 2023-01-24T16:16:25.649 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 51 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x3771429d 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:16:25.649 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:16:25.650 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:16:25.650 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:16:41.160 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x3771409c 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:16:41.160 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:16:48.894 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x37714728 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:16:48.895 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:17:04.539 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x398421f9 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:17:04.539 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:17:04.539 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:17:20.072 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 35 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0xc37f8 0xc0f00 0xb7393 0x398421a1 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:17:20.072 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:17:20.072 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:17:25.185 INFO:tasks.ceph.osd.0.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0xd59d6 0xd548b 0x287b7 0xb732c 0x398421a1 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36afd000 0x36b607c8 0x36b65635 0x36b679b8 0x36b684ad 0x36b688a6 0x36b69106 0x36b69fd5 0x36fea258 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:17:25.185 INFO:tasks.ceph.osd.0.smithi179.stderr:kernel callstack: 2023-01-24T16:17:27.688 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x3771423c 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:17:27.688 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:17:32.382 INFO:tasks.ceph.osd.0.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x9435f 0x94421 0x94ce6 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36afd000 0x36b607c8 0x36b65635 0x36b679b8 0x36b684ad 0x36b688a6 0x36b69106 0x36b69fd5 0x36fea258 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:17:32.383 INFO:tasks.ceph.osd.0.smithi179.stderr:kernel callstack: 2023-01-24T16:17:35.624 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x39761 0x943d7 0x94ce6 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:17:35.624 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:17:43.544 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x39ca4efd 0x39cbb07a 0x39cd0b25 0x36f56c5a 0x36b05938 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:17:43.544 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:17:51.001 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x3b712de7 0x3b71318c 0x39842454 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:17:51.001 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:18:19.411 INFO:tasks.ceph.osd.1.smithi179.stderr:Rate-limit: suppressed 1 backtrace on shard 0 2023-01-24T16:18:49.162 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x36e736ab 0x36f58741 0x36b05938 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:18:49.162 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:18:49.163 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:18:49.163 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:18:52.615 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x377148ae 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:18:52.616 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:18:56.265 INFO:tasks.ceph.osd.0.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x2b3f4 0x2b832 0x28adf 0xb732c 0x398421a1 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36afd000 0x36b607c8 0x36b65635 0x36b679b8 0x36b684ad 0x36b688a6 0x36b69106 0x36b69fd5 0x36fea258 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:18:56.265 INFO:tasks.ceph.osd.0.smithi179.stderr:kernel callstack: 2023-01-24T16:19:07.789 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0xc37df 0xc15bc 0x2b495 0x2b832 0x28adf 0xb732c 0x36267631 0x36b0719e 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:19:07.789 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:19:07.790 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:19:15.460 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x3771404d 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:19:15.460 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:19:19.504 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x3b712b97 0x3b71318c 0x39842454 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:19:19.504 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:19:23.164 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x2b350 0x2b832 0x28adf 0xb732c 0x398421a1 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:19:23.164 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:19:31.078 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x943a7 0x94ce6 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:19:31.078 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:19:38.506 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x39742 0x943d7 0x94ce6 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:19:38.506 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:19:46.240 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0xd0551 0x2b490 0x2b832 0x28adf 0xb732c 0x398421a1 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:19:46.240 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:20:19.435 INFO:tasks.ceph.osd.1.smithi179.stderr:Rate-limit: suppressed 2 backtraces on shard 0 2023-01-24T16:20:25.416 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x398421f9 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:20:25.416 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:20:25.417 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:20:25.417 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:20:41.082 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x3b712bd2 0x3b71318c 0x39842454 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:20:41.082 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:20:45.190 INFO:tasks.ceph.osd.0.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x377141e9 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36afd000 0x36b607c8 0x36b65635 0x36b679b8 0x36b684ad 0x36b688a6 0x36b69106 0x36b69fd5 0x36fea258 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:20:45.190 INFO:tasks.ceph.osd.0.smithi179.stderr:kernel callstack: 2023-01-24T16:20:56.497 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x2cd25 0xb732c 0x398421a1 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:20:56.497 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:21:04.255 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0xc37e7 0x287a7 0xb732c 0x398421a1 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:21:04.255 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:21:12.009 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x3771417d 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:21:12.009 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:21:27.352 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0xc0ee4 0xb632f 0x35f61d7f 0x35f646f1 0x3b712963 0x3b71318c 0x39842454 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:21:27.352 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:21:34.700 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x398421f9 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:21:34.700 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:21:51.657 INFO:tasks.ceph.osd.0.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0xd0457 0x2b490 0x2b832 0x28adf 0xb732c 0x398421a1 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36afd000 0x36b607c8 0x36b65635 0x36b679b8 0x36b684ad 0x36b688a6 0x36b69106 0x36b69fd5 0x36fea258 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:21:51.657 INFO:tasks.ceph.osd.0.smithi179.stderr:kernel callstack: 2023-01-24T16:21:51.657 INFO:tasks.ceph.osd.0.smithi179.stderr:kernel callstack: 2023-01-24T16:22:05.481 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x37714187 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:22:05.481 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:22:05.481 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:22:05.482 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:22:12.539 INFO:tasks.ceph.osd.0.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x39842203 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36afd000 0x36b607c8 0x36b65635 0x36b679b8 0x36b684ad 0x36b688a6 0x36b69106 0x36b69fd5 0x36fea258 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:22:12.539 INFO:tasks.ceph.osd.0.smithi179.stderr:kernel callstack: 2023-01-24T16:22:13.237 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0xc97ec 0x94409 0x94ce6 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:22:13.238 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:22:13.238 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:22:36.186 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x3979e 0x943f6 0x94ce6 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:22:36.186 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:22:36.186 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:22:43.910 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0xc96cd 0x94409 0x94ce6 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:22:43.910 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:22:47.563 INFO:tasks.ceph.osd.0.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x2b46d 0x2b832 0x28adf 0xb732c 0x398421a1 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36afd000 0x36b607c8 0x36b65635 0x36b679b8 0x36b684ad 0x36b688a6 0x36b69106 0x36b69fd5 0x36fea258 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:22:47.563 INFO:tasks.ceph.osd.0.smithi179.stderr:kernel callstack: 2023-01-24T16:22:51.426 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x37714027 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:22:51.426 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:23:06.794 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x37714371 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:23:06.794 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:23:14.530 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0xc0ec9 0xb7393 0x398421a1 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:23:14.530 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:23:30.191 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x39842203 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:23:30.191 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:23:30.191 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:23:45.802 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x39761 0x943d7 0x94ce6 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:23:45.802 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:23:53.634 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x39761 0x943d7 0x94ce6 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:23:53.634 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:24:01.373 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x39761 0x943d7 0x94ce6 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:24:01.373 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:24:08.960 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x2b343 0x2b832 0x28adf 0xb732c 0x398421a1 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:24:08.960 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:24:18.514 INFO:tasks.ceph.osd.0.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x3b712bdb 0x3b71318c 0x39842454 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36afd000 0x36b607c8 0x36b65635 0x36b679b8 0x36b684ad 0x36b688a6 0x36b69106 0x36b69fd5 0x36fea258 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:24:18.514 INFO:tasks.ceph.osd.0.smithi179.stderr:kernel callstack: 2023-01-24T16:24:18.514 INFO:tasks.ceph.osd.0.smithi179.stderr:kernel callstack: 2023-01-24T16:24:19.506 INFO:tasks.ceph.osd.1.smithi179.stderr:Rate-limit: suppressed 1 backtrace on shard 0 2023-01-24T16:24:32.302 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x28829 0xb732c 0x398421a1 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:24:32.303 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:24:32.303 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:24:40.203 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x37713ff3 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:24:40.203 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:24:40.204 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:24:48.068 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x3974d 0x943d7 0x94ce6 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:24:48.068 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:24:55.720 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0xc97e8 0x94409 0x94ce6 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:24:55.720 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:25:03.330 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 35 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x377140b1 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:25:03.330 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:25:14.603 INFO:tasks.ceph.osd.0.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0xf7c3 0xc37ec 0xc15bc 0x2b495 0x2b832 0x28adf 0xb4614 0x39d00246 0x39ca706b 0x3c254721 0x3c254925 0x3c254925 0x3c254925 0x3c254925 0x3c254925 0x3c254925 0x3c254925 0x3c254925 0x3c254925 0x3c254925 0x3c254925 0x3c254925 0x3c254925 0x3c254925 0x3c254925 0x3c254925 0x3c254925 0x3c254925 0x3c254925 0x3c254925 0x3c254925 0x3c254925 0x44bee5ec 0x44beee79 0x362c6842 0x362dcdef 0x362dda11 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:25:14.604 INFO:tasks.ceph.osd.0.smithi179.stderr:kernel callstack: 2023-01-24T16:25:19.546 INFO:tasks.ceph.osd.1.smithi179.stderr:Rate-limit: suppressed 1 backtrace on shard 0 2023-01-24T16:25:22.291 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0xb7e1a 0x2b490 0x2b832 0x28adf 0xb732c 0x398421a1 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:25:22.292 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 0xffffffffffffff80 0xffffffffb13b5ec7 0xffffffffb10043ab 0xffffffffb1a000a9 2023-01-24T16:25:22.292 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:25:22.292 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:25:26.531 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0xb729f 0x398421a1 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:25:26.531 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:25:38.042 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0xc97d6 0x94409 0x94ce6 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:25:38.042 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:25:41.776 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x37714187 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:25:41.776 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:25:57.639 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x3b712bdb 0x3b71318c 0x39842454 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:25:57.639 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:26:19.548 INFO:tasks.ceph.osd.1.smithi179.stderr:Rate-limit: suppressed 2 backtraces on shard 0 2023-01-24T16:26:20.483 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x3b712bdb 0x3b71318c 0x39842454 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:26:20.483 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:26:20.483 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:26:20.484 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:26:20.484 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:26:24.340 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x37714187 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:26:24.342 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:26:27.941 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x2c753 0x2857c 0xb62c3 0x35f61d7f 0x35f646f1 0x3b712963 0x3b71318c 0x39842454 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:26:27.942 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:26:43.223 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0xc96c7 0x94409 0x94ce6 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:26:43.224 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:26:43.224 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:26:51.041 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x3b712bd2 0x3b71318c 0x39842454 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:26:51.041 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:27:19.563 INFO:tasks.ceph.osd.1.smithi179.stderr:Rate-limit: suppressed 2 backtraces on shard 0 2023-01-24T16:27:22.076 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x39761 0x943d7 0x94ce6 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:27:22.076 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:27:22.077 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:27:22.077 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:27:29.909 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0xc97e8 0x94409 0x94ce6 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:27:29.909 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:28:00.720 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x94c8e 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:28:00.720 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:28:00.721 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:28:08.635 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x94c92 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:28:08.635 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:28:15.977 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x37713fa7 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:28:15.977 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:28:20.494 INFO:tasks.ceph.osd.0.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x9437b 0x94ce6 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36afd000 0x36b607c8 0x36b65635 0x36b679b8 0x36b684ad 0x36b688a6 0x36b69106 0x36b69fd5 0x36fea258 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:28:20.494 INFO:tasks.ceph.osd.0.smithi179.stderr:kernel callstack: 2023-01-24T16:28:20.494 INFO:tasks.ceph.osd.0.smithi179.stderr:kernel callstack: 2023-01-24T16:28:24.144 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x397a9 0x943d7 0x94ce6 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:28:24.145 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:28:34.303 INFO:tasks.ceph.osd.0.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x2ac65 0x2b4c3 0x2b832 0x28adf 0xb732c 0x398421a1 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36afd000 0x36b607c8 0x36b65635 0x36b679b8 0x36b684ad 0x36b688a6 0x36b69106 0x36b69fd5 0x36fea258 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:28:34.303 INFO:tasks.ceph.osd.0.smithi179.stderr:kernel callstack: 2023-01-24T16:28:39.322 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x37714130 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:28:39.322 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:28:47.078 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x3771416f 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:28:47.078 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:28:54.763 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x39761 0x943d7 0x94ce6 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:28:54.763 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:29:02.270 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x35f64881 0x3b712963 0x3b71318c 0x39842454 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:29:02.270 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:29:19.623 INFO:tasks.ceph.osd.1.smithi179.stderr:Rate-limit: suppressed 2 backtraces on shard 0 2023-01-24T16:29:25.448 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x2c29c 0x2c5a7 0x2857c 0xb62c3 0x3d95f724 0x3d7c2f69 0x3d7c6ae5 0x3d725604 0x3d725e62 0x3d727707 0x3677c1a9 0x36bb1bd1 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:29:25.448 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 0xffffffffffffff80 0xffffffffb19f2fb0 0xffffffffb13829a6 0xffffffffb1360eba 0xffffffffb10043ab 0xffffffffb1a000a9 2023-01-24T16:29:25.448 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:29:25.449 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 0xffffffffffffff80 0xffffffffc065e2c5 0xffffffffc0768219 0xffffffffc0768435 0xffffffffc0768ad2 0xffffffffc076c972 0xffffffffc076ccd7 0xffffffffc0772099 0xffffffffb1395291 0xffffffffb137e06a 0xffffffffb137e5b5 0xffffffffb129b95c 0xffffffffc075b8e5 0xffffffffb135d462 0xffffffffb1360c55 0xffffffffb1360edf 0xffffffffb10043ab 0xffffffffb1a000a9 2023-01-24T16:29:25.449 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:29:33.111 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x943ba 0x94ce6 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:29:33.111 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:29:44.863 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0xb729f 0x398421a1 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:29:44.864 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:29:44.864 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:29:48.703 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x39761 0x943d7 0x94ce6 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:29:48.703 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:29:52.662 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0xb7f5f 0x2b490 0x2b832 0x28adf 0xb732c 0x398421a1 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:29:52.662 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:30:08.987 INFO:tasks.ceph.osd.0.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x377141fb 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36afd000 0x36b607c8 0x36b65635 0x36b679b8 0x36b684ad 0x36b688a6 0x36b69106 0x36b69fd5 0x36fea258 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:30:08.988 INFO:tasks.ceph.osd.0.smithi179.stderr:kernel callstack: 2023-01-24T16:30:19.627 INFO:tasks.ceph.osd.1.smithi179.stderr:Rate-limit: suppressed 3 backtraces on shard 0 2023-01-24T16:30:26.907 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x377144a3 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:30:26.907 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:30:26.908 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:30:26.908 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:30:26.908 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:30:33.758 INFO:tasks.ceph.osd.0.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0xc97c7 0x94409 0x94ce6 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36afd000 0x36b607c8 0x36b65635 0x36b679b8 0x36b684ad 0x36b688a6 0x36b69106 0x36b69fd5 0x36fea258 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:30:33.758 INFO:tasks.ceph.osd.0.smithi179.stderr:kernel callstack: 2023-01-24T16:30:34.741 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x37714187 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:30:34.741 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:30:42.443 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x94c92 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:30:42.443 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:30:50.282 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x3973f 0x943f6 0x94ce6 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:30:50.282 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:30:58.037 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x2b343 0x2b832 0x28adf 0xb732c 0x398421a1 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:30:58.037 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:31:01.822 INFO:tasks.ceph.osd.0.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x2b350 0x2b832 0x28adf 0xb732c 0x398421a1 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36afd000 0x36b607c8 0x36b65635 0x36b679b8 0x36b684ad 0x36b688a6 0x36b69106 0x36b69fd5 0x36fea258 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:31:01.823 INFO:tasks.ceph.osd.0.smithi179.stderr:kernel callstack: 2023-01-24T16:31:19.647 INFO:tasks.ceph.osd.1.smithi179.stderr:Rate-limit: suppressed 1 backtrace on shard 0 2023-01-24T16:31:29.065 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x37714187 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:31:29.065 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:31:29.066 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:31:37.041 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0xc0f72 0xb7393 0x398421a1 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:31:37.041 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:31:44.645 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x3771417d 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:31:44.645 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:31:52.565 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x37714187 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:31:52.565 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:32:00.084 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x39842203 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:32:00.085 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:32:08.639 INFO:tasks.ceph.osd.0.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x2b3db 0x2b832 0x28adf 0xb732c 0x398421a1 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36afd000 0x36b607c8 0x36b65635 0x36b679b8 0x36b684ad 0x36b688a6 0x36b69106 0x36b69fd5 0x36fea258 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:32:08.639 INFO:tasks.ceph.osd.0.smithi179.stderr:kernel callstack: 2023-01-24T16:32:19.650 INFO:tasks.ceph.osd.1.smithi179.stderr:Rate-limit: suppressed 2 backtraces on shard 0 2023-01-24T16:32:23.338 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x2b45b 0x2b832 0x28adf 0xb732c 0x398421a1 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:32:23.338 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:32:23.339 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:32:23.339 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:32:27.147 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x36038d61 0x4518a0ba 0x4518acc4 0x3627c645 0x36bb1977 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:32:27.147 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:32:31.169 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0xc96d0 0x94409 0x94ce6 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:32:31.169 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:32:38.954 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x2c2b2 0x2c5a7 0x2857c 0xb62c3 0x35f61d7f 0x35f646f1 0x3b712963 0x3b71318c 0x39842454 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:32:38.954 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:32:46.354 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x94ceb 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:32:46.354 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:32:46.354 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:32:57.761 INFO:tasks.ceph.osd.0.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x39761 0x943d7 0x94ce6 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36afd000 0x36b607c8 0x36b65635 0x36b679b8 0x36b684ad 0x36b688a6 0x36b69106 0x36b69fd5 0x36fea258 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:32:57.762 INFO:tasks.ceph.osd.0.smithi179.stderr:kernel callstack: 2023-01-24T16:33:19.687 INFO:tasks.ceph.osd.1.smithi179.stderr:Rate-limit: suppressed 4 backtraces on shard 0 2023-01-24T16:33:24.930 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x3b712fab 0x3b71318c 0x39842454 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:33:24.930 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:33:24.931 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:33:24.931 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:33:24.931 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:33:24.931 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:33:24.931 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:33:39.413 INFO:tasks.ceph.osd.0.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x2accf 0x2b4c3 0x2b832 0x28adf 0xb4614 0x39d00246 0x39ca706b 0x3c254721 0x3c254925 0x3c254925 0x3c254925 0x3c254925 0x3c254925 0x3c254925 0x3c254925 0x3c254925 0x3c254925 0x3c254925 0x3c254925 0x3c254925 0x3c254925 0x3c254925 0x3c254925 0x3c254925 0x3c254925 0x3c254925 0x3c254925 0x3c254925 0x3c254925 0x44bee5ec 0x44beee79 0x362c6842 0x362dcdef 0x362dda11 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:33:39.413 INFO:tasks.ceph.osd.0.smithi179.stderr:kernel callstack: 2023-01-24T16:33:39.414 INFO:tasks.ceph.osd.0.smithi179.stderr:kernel callstack: 2023-01-24T16:33:40.193 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0xc97cb 0x94409 0x94ce6 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:33:40.194 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:33:48.109 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0xd59c0 0xd548b 0x287b7 0xb732c 0x398421a1 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:33:48.109 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:33:56.028 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0xc97c7 0x94409 0x94ce6 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:33:56.028 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:34:03.540 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 35 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x37714187 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:34:03.540 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:34:19.698 INFO:tasks.ceph.osd.1.smithi179.stderr:Rate-limit: suppressed 1 backtrace on shard 0 2023-01-24T16:34:26.805 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0xc96c7 0x94409 0x94ce6 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:34:26.806 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:34:26.806 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:34:35.593 INFO:tasks.ceph.osd.0.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x2b45b 0x2b832 0x28adf 0xb732c 0x398421a1 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36afd000 0x36b607c8 0x36b65635 0x36b679b8 0x36b684ad 0x36b688a6 0x36b69106 0x36b69fd5 0x36fea258 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:34:35.594 INFO:tasks.ceph.osd.0.smithi179.stderr:kernel callstack: 2023-01-24T16:34:50.321 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x3771423f 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:34:50.321 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:34:58.096 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0xb7e3d 0x2b490 0x2b832 0x28adf 0xb4614 0x445f3527 0x445f875d 0x36aba437 0x3618baa7 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:34:58.096 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:35:21.420 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0xc97c7 0x94409 0x94ce6 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:35:21.421 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:35:37.010 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x9443b 0x94ce6 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:35:37.010 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:35:52.559 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0xc97cb 0x94409 0x94ce6 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:35:52.559 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:36:00.236 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0xd59c5 0xd548b 0x287b7 0xb732c 0x398421a1 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:36:00.236 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:36:23.404 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x3b712bdb 0x3b71318c 0x39842454 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:36:23.405 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:36:23.405 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:36:30.690 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 35 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x35f64958 0x3b712963 0x3b71318c 0x39842454 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:36:30.690 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:36:30.691 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:36:38.508 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x94cab 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:36:38.508 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 0xffffffffffffff80 0xffffffffb13173dd 0xffffffffb12d96c4 0xffffffffb12dc2f3 0xffffffffb12dc62a 0xffffffffb107c8b0 0xffffffffb107cb47 0xffffffffb1a0114e 2023-01-24T16:36:38.508 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:36:50.081 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x37714187 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:36:50.081 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:36:53.979 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x94ceb 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:36:53.979 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:37:19.785 INFO:tasks.ceph.osd.1.smithi179.stderr:Rate-limit: suppressed 4 backtraces on shard 0 2023-01-24T16:37:32.453 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x398421f9 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:37:32.454 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:37:32.454 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:37:32.454 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:37:32.455 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:37:32.455 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:37:32.455 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:37:40.179 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x398421a1 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:37:40.179 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:37:48.166 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 32 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0xd5872 0xd5a00 0xd548b 0x287b7 0xb732c 0x398421a1 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:37:48.166 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:37:55.967 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0xc37ec 0xc15bc 0x2b495 0x2b832 0x28adf 0xb732c 0x398421a1 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:37:55.967 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:38:11.248 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x39842203 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:38:11.248 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:38:19.790 INFO:tasks.ceph.osd.1.smithi179.stderr:Rate-limit: suppressed 1 backtrace on shard 0 2023-01-24T16:38:23.554 INFO:tasks.ceph.osd.0.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0xb7e78 0x2b490 0x2b832 0x28adf 0xb732c 0x398421a1 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36afd000 0x36b607c8 0x36b65635 0x36b679b8 0x36b684ad 0x36b688a6 0x36b69106 0x36b69fd5 0x36fea258 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:38:23.555 INFO:tasks.ceph.osd.0.smithi179.stderr:kernel callstack: 2023-01-24T16:38:23.555 INFO:tasks.ceph.osd.0.smithi179.stderr:kernel callstack: 2023-01-24T16:38:34.403 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 54 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x2a377 0x2ab8e 0x27c98 0x2af57 0x2b4c3 0x2b832 0x28adf 0xb4614 0x445f3527 0x445f875d 0x36aba437 0x3618baa7 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:38:34.403 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:38:34.403 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:38:34.403 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:38:42.379 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0xd5882 0xd5a00 0xd548b 0x287b7 0xb732c 0x398421a1 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:38:42.379 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:38:50.195 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x3b712bd2 0x3b71318c 0x39842454 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:38:50.195 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:38:58.029 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x3b712ddd 0x3b71318c 0x39842454 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:38:58.029 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:39:05.788 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x2c5c1 0x2857c 0xb62c3 0x35f61d7f 0x35f646f1 0x3b712963 0x3b71318c 0x39842454 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:39:05.788 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:39:21.219 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x3977c 0x943d7 0x94ce6 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:39:21.219 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:39:28.932 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 35 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x3771420d 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:39:28.933 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:39:36.673 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 35 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x3b712bd2 0x3b71318c 0x39842454 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:39:36.673 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:39:51.938 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x377141df 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:39:51.938 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:39:51.938 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:39:59.586 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x94ceb 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:39:59.587 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:40:16.209 INFO:tasks.ceph.osd.0.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x2b350 0x2b832 0x28adf 0xb732c 0x398421a1 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36afd000 0x36b607c8 0x36b65635 0x36b679b8 0x36b684ad 0x36b688a6 0x36b69106 0x36b69fd5 0x36fea258 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:40:16.209 INFO:tasks.ceph.osd.0.smithi179.stderr:kernel callstack: 2023-01-24T16:40:19.864 INFO:tasks.ceph.osd.1.smithi179.stderr:Rate-limit: suppressed 1 backtrace on shard 0 2023-01-24T16:40:22.889 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x94c6f 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:40:22.889 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:40:22.890 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:40:22.890 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:40:30.623 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0xd5da4 0xc0f8c 0xb632f 0x35f61d7f 0x35f646f1 0x3b712963 0x3b71318c 0x39842454 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:40:30.623 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 0xffffffffffffff80 0xffffffffc0598120 0xffffffffc074f4d6 0xffffffffc076ca6a 0xffffffffc076ccb1 0xffffffffc0772099 0xffffffffb1395291 0xffffffffb13a3d67 0xffffffffc0770f65 0xffffffffb129671e 0xffffffffb129b9c2 0xffffffffc075b8e5 0xffffffffb135d462 0xffffffffb1360c55 0xffffffffb1360edf 0xffffffffb10043ab 0xffffffffb1a000a9 2023-01-24T16:40:30.624 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:40:38.397 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0xd58aa 0xd5a00 0xd548b 0x287b7 0xb732c 0x398421a1 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:40:38.397 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:40:46.207 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0xd58aa 0xd5a00 0xd548b 0x287b7 0xb732c 0x398421a1 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:40:46.208 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:40:54.112 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0xd045a 0x2c6e3 0x2857c 0xb62c3 0x35f61d7f 0x35f646f1 0x3b712963 0x3b71318c 0x39842454 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:40:54.113 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:41:19.244 INFO:tasks.ceph.osd.0.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x398421cb 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36afd000 0x36b607c8 0x36b65635 0x36b679b8 0x36b684ad 0x36b688a6 0x36b69106 0x36b69fd5 0x36fea258 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:41:19.244 INFO:tasks.ceph.osd.0.smithi179.stderr:kernel callstack: 2023-01-24T16:41:19.908 INFO:tasks.ceph.osd.1.smithi179.stderr:Rate-limit: suppressed 3 backtraces on shard 0 2023-01-24T16:41:24.753 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x35f6492f 0x3b712963 0x3b71318c 0x39842454 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:41:24.754 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:41:24.754 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:41:24.754 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:41:24.755 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:41:24.755 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:41:24.755 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:41:29.019 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 35 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0xc97e8 0x94409 0x94ce6 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:41:29.020 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:41:32.379 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x2c5f5 0x2857c 0xb62c3 0x35f61d7f 0x35f646f1 0x3b712963 0x3b71318c 0x39842454 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:41:32.379 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:41:40.292 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x398421f9 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:41:40.292 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:41:44.497 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0xd59c7 0xd548b 0x287b7 0xb732c 0x398421a1 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:41:44.497 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:41:57.895 INFO:tasks.ceph.osd.0.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x27c6f 0x2af57 0x2b4c3 0x2b832 0x28adf 0xb732c 0x398421a1 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36afd000 0x36b607c8 0x36b65635 0x36b679b8 0x36b684ad 0x36b688a6 0x36b69106 0x36b69fd5 0x36fea258 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:41:57.896 INFO:tasks.ceph.osd.0.smithi179.stderr:kernel callstack: 2023-01-24T16:42:19.938 INFO:tasks.ceph.osd.1.smithi179.stderr:Rate-limit: suppressed 4 backtraces on shard 0 2023-01-24T16:42:26.420 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x37713f91 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:42:26.421 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:42:26.421 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:42:26.421 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:42:26.421 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:42:26.421 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:42:34.093 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x39771 0x943f6 0x94ce6 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:42:34.093 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:42:50.092 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x39842203 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:42:50.093 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:42:57.771 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x3b712bd2 0x3b71318c 0x39842454 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:42:57.771 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:43:05.520 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x3a4dfb3c 0x3b712fdb 0x3b71318c 0x39842454 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:43:05.521 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:43:14.628 INFO:tasks.ceph.osd.0.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x39842203 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36afd000 0x36b607c8 0x36b65635 0x36b679b8 0x36b684ad 0x36b688a6 0x36b69106 0x36b69fd5 0x36fea258 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:43:14.628 INFO:tasks.ceph.osd.0.smithi179.stderr:kernel callstack: 2023-01-24T16:43:19.966 INFO:tasks.ceph.osd.1.smithi179.stderr:Rate-limit: suppressed 1 backtrace on shard 0 2023-01-24T16:43:29.125 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x3771414a 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:43:29.126 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:43:29.126 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:43:44.664 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x3771417d 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:43:44.665 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:43:44.665 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:43:44.665 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:43:52.336 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0xd58a7 0xd5a00 0xd548b 0x287b7 0xb732c 0x398421a1 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:43:52.336 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:44:03.399 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x3b712bd2 0x3b71318c 0x39842454 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:44:03.400 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:44:07.845 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x94c85 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:44:07.845 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:44:19.998 INFO:tasks.ceph.osd.1.smithi179.stderr:Rate-limit: suppressed 1 backtrace on shard 0 2023-01-24T16:44:23.038 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0xb7e1e 0x2c8ec 0x2857c 0xb62c3 0x35f61d7f 0x35f646f1 0x3b712963 0x3b71318c 0x39842454 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:44:23.039 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:44:23.039 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:44:23.039 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:44:30.757 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x2c590 0x2857c 0xb62c3 0x35f61d7f 0x35f646f1 0x3b712963 0x3b71318c 0x39842454 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:44:30.758 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:44:31.946 INFO:tasks.ceph.osd.0.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x2b350 0x2b832 0x28adf 0xb732c 0x398421a1 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36afd000 0x36b607c8 0x36b65635 0x36b679b8 0x36b684ad 0x36b688a6 0x36b69106 0x36b69fd5 0x36fea258 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:44:31.946 INFO:tasks.ceph.osd.0.smithi179.stderr:kernel callstack: 2023-01-24T16:44:34.302 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x3b712b65 0x3b71318c 0x39842454 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:44:34.302 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:44:38.502 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0xc96cd 0x94409 0x94ce6 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:44:38.502 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:44:42.479 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x2cebc 0xb732c 0x398421a1 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:44:42.480 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:45:20.016 INFO:tasks.ceph.osd.1.smithi179.stderr:Rate-limit: suppressed 4 backtraces on shard 0 2023-01-24T16:45:24.712 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 35 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x39842636 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:45:24.712 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:45:24.713 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:45:24.713 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:45:24.713 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:45:24.713 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:45:28.729 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x3b712fab 0x3b71318c 0x39842454 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:45:28.729 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:45:31.272 INFO:tasks.ceph.osd.0.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x37714062 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36afd000 0x36b607c8 0x36b65635 0x36b679b8 0x36b684ad 0x36b688a6 0x36b69106 0x36b69fd5 0x36fea258 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:45:31.272 INFO:tasks.ceph.osd.0.smithi179.stderr:kernel callstack: 2023-01-24T16:45:32.051 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x39842203 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:45:32.052 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:45:47.841 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x37713ff9 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:45:47.842 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:45:55.407 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x3b712bd2 0x3b71318c 0x39842454 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:45:55.408 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:46:41.920 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x37714391 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:46:41.921 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:46:41.921 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:46:49.741 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x398421f9 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:46:49.742 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:46:57.581 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x943c3 0x94ce6 0x37714a1d 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:46:57.581 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:47:05.007 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x94c85 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:47:05.007 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:47:13.045 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 35 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x3771441e 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:47:13.045 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:47:20.351 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x2cebc 0xb732c 0x398421a1 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:47:20.351 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:47:28.117 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x2c5fa 0x2857c 0xb62c3 0x35f61d7f 0x35f646f1 0x3b712963 0x3b71318c 0x39842454 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:47:28.117 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:47:35.890 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x35f644ed 0x3b712963 0x3b71318c 0x39842454 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:47:35.890 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:47:46.863 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x3771487e 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:47:46.863 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:47:51.214 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x3b712bd2 0x3b71318c 0x39842454 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:47:51.214 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:48:20.071 INFO:tasks.ceph.osd.1.smithi179.stderr:Rate-limit: suppressed 2 backtraces on shard 0 2023-01-24T16:48:22.194 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0xc156a 0x3b64d53d 0x3b64e399 0x3b6513c1 0x35f19188 0x3602c6e5 0x3602ef36 0x3602f68b 0x45189fad 0x4518acc4 0x3627c645 0x36bb1977 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:48:22.195 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:48:22.195 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:48:22.195 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:48:22.195 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:48:22.195 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:48:23.384 INFO:tasks.ceph.osd.0.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x2ac8d 0x2b4c3 0x2b832 0x28adf 0xb732c 0x362de716 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:48:23.385 INFO:tasks.ceph.osd.0.smithi179.stderr:kernel callstack: 2023-01-24T16:48:29.972 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0xc97e4 0x94409 0x94ce6 0x37714a1d 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:48:29.973 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:48:29.973 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:48:45.539 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x377142e0 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:48:45.540 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:48:45.540 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:48:57.061 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 34 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x377141df 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:48:57.061 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:49:01.662 INFO:tasks.ceph.osd.0.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0xc97e8 0x94409 0x94ce6 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36afd000 0x36b607c8 0x36b65635 0x36b679b8 0x36b684ad 0x36b688a6 0x36b69106 0x36b69fd5 0x36fea258 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:49:01.663 INFO:tasks.ceph.osd.0.smithi179.stderr:kernel callstack: 2023-01-24T16:49:08.719 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x94cf6 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:49:08.719 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:49:20.095 INFO:tasks.ceph.osd.1.smithi179.stderr:Rate-limit: suppressed 1 backtrace on shard 0 2023-01-24T16:49:23.816 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0xf7c3 0xc37ec 0xc15bc 0x2b495 0x2b832 0x28adf 0xb732c 0x398421a1 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:49:23.817 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:49:23.817 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:49:31.568 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0xb729f 0x398421a1 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:49:31.569 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:49:39.323 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x39761 0x943d7 0x94ce6 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:49:39.323 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:49:47.078 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0xc96c7 0x94409 0x94ce6 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:49:47.079 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:49:54.914 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x2b3f7 0x2b832 0x28adf 0xb732c 0x398421a1 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:49:54.914 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:50:20.113 INFO:tasks.ceph.osd.1.smithi179.stderr:Rate-limit: suppressed 1 backtrace on shard 0 2023-01-24T16:50:26.191 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0xc97e8 0x94409 0x94ce6 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:50:26.192 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:50:26.192 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:50:26.192 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:50:26.192 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:50:33.948 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x37714187 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:50:33.949 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:50:41.782 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x2b45b 0x2b832 0x28adf 0xb732c 0x398421a1 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:50:41.783 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:50:49.380 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x3771417d 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:50:49.381 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:50:57.295 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 35 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x2b350 0x2b832 0x28adf 0xb4614 0x445f3527 0x445f875d 0x36aba437 0x3618baa7 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:50:57.295 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:51:20.129 INFO:tasks.ceph.osd.1.smithi179.stderr:Rate-limit: suppressed 2 backtraces on shard 0 2023-01-24T16:51:20.612 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x3b712bd2 0x3b71318c 0x39842454 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:51:20.613 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:51:20.613 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:51:20.613 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:51:36.006 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x39761 0x943d7 0x94ce6 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:51:36.006 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:51:43.924 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x943a7 0x94ce6 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:51:43.925 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:51:59.206 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x3771417d 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:51:59.207 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:52:06.876 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0xd5872 0xd5a00 0xd548b 0x2c64a 0x2857c 0xb62c3 0x35f61d7f 0x35f646f1 0x3b712963 0x3b71318c 0x39842454 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:52:06.876 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:52:18.155 INFO:tasks.ceph.osd.0.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0xb7f5f 0x2c8ec 0x2857c 0xb62c3 0x39cb6133 0x3757d8eb 0x37554bb7 0x372b9694 0x36b001fe 0x36b607c8 0x36b65635 0x36b679b8 0x36b684ad 0x36b688a6 0x36b69106 0x36b69fd5 0x36fea258 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:52:18.156 INFO:tasks.ceph.osd.0.smithi179.stderr:kernel callstack: 2023-01-24T16:52:18.156 INFO:tasks.ceph.osd.0.smithi179.stderr:kernel callstack: 2023-01-24T16:52:20.162 INFO:tasks.ceph.osd.1.smithi179.stderr:Rate-limit: suppressed 2 backtraces on shard 0 2023-01-24T16:52:22.120 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x2c3ef 0x2857c 0xb62c3 0x35f61d7f 0x35f646f1 0x3b712963 0x3b71318c 0x39842454 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:52:22.121 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:52:22.121 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:52:22.121 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:52:29.957 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x377142b5 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:52:29.958 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:52:37.473 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0xc97e8 0x94409 0x94ce6 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:52:37.474 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:52:45.422 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x3771400b 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:52:45.422 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:52:49.609 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x35f64459 0x3b712963 0x3b71318c 0x39842454 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:52:49.610 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:53:20.203 INFO:tasks.ceph.osd.1.smithi179.stderr:Rate-limit: suppressed 2 backtraces on shard 0 2023-01-24T16:53:21.196 INFO:tasks.ceph.osd.0.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0xd5872 0xd5a00 0xd548b 0x287b7 0xb732c 0x398421a1 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36afd000 0x36b607c8 0x36b65635 0x36b679b8 0x36b684ad 0x36b688a6 0x36b69106 0x36b69fd5 0x36fea258 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:53:21.196 INFO:tasks.ceph.osd.0.smithi179.stderr:kernel callstack: 2023-01-24T16:53:24.099 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x39761 0x943d7 0x94ce6 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:53:24.100 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:53:24.100 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:53:24.100 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:53:31.852 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x94ceb 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:53:31.853 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:53:39.656 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x3b712bd2 0x3b71318c 0x39842454 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:53:39.657 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:53:41.912 INFO:tasks.ceph.osd.0.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x39761 0x943d7 0x94ce6 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36afd000 0x36b607c8 0x36b65635 0x36b679b8 0x36b684ad 0x36b688a6 0x36b69106 0x36b69fd5 0x36fea258 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:53:41.913 INFO:tasks.ceph.osd.0.smithi179.stderr:kernel callstack: 2023-01-24T16:53:47.279 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0xc96c7 0x94409 0x94ce6 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:53:47.280 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:54:02.654 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x37714187 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:54:02.654 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:54:20.231 INFO:tasks.ceph.osd.1.smithi179.stderr:Rate-limit: suppressed 1 backtrace on shard 0 2023-01-24T16:54:33.493 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x2c590 0x2857c 0xb62c3 0x35f61d7f 0x35f646f1 0x3b712963 0x3b71318c 0x39842454 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:54:33.494 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:54:33.494 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:54:33.494 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:54:33.494 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:54:37.957 INFO:tasks.ceph.osd.0.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x3b712bd2 0x3b71318c 0x39842454 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36afd000 0x36b607c8 0x36b65635 0x36b679b8 0x36b684ad 0x36b688a6 0x36b69106 0x36b69fd5 0x36fea258 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:54:37.958 INFO:tasks.ceph.osd.0.smithi179.stderr:kernel callstack: 2023-01-24T16:54:41.157 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x35f61c2c 0x35f646f1 0x3b712963 0x3b71318c 0x39842454 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:54:41.158 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:54:56.875 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0xc0ecc 0xb7393 0x398421a1 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:54:56.875 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:55:04.554 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x3b71286e 0x3b71318c 0x39842454 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:55:04.555 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:55:12.114 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x3b712bd2 0x3b71318c 0x39842454 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:55:12.115 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:55:12.731 INFO:tasks.ceph.osd.0.smithi179.stderr:Reactor stalled for 35 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x37714292 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36afd000 0x36b607c8 0x36b65635 0x36b679b8 0x36b684ad 0x36b688a6 0x36b69106 0x36b69fd5 0x36fea258 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:55:12.732 INFO:tasks.ceph.osd.0.smithi179.stderr:kernel callstack: 2023-01-24T16:55:20.263 INFO:tasks.ceph.osd.1.smithi179.stderr:Rate-limit: suppressed 1 backtrace on shard 0 2023-01-24T16:55:23.088 INFO:tasks.ceph.osd.0.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x3771417d 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36afd000 0x36b607c8 0x36b65635 0x36b679b8 0x36b684ad 0x36b688a6 0x36b69106 0x36b69fd5 0x36fea258 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:55:23.089 INFO:tasks.ceph.osd.0.smithi179.stderr:kernel callstack: 2023-01-24T16:55:27.394 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x2c5c7 0x2857c 0xb62c3 0x35f61d7f 0x35f646f1 0x3b712963 0x3b71318c 0x39842454 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:55:27.395 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:55:27.395 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:55:35.231 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x3b712bd2 0x3b71318c 0x39842454 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:55:35.232 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:55:35.232 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:55:42.643 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x377141fb 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:55:42.644 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:55:46.666 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x943b7 0x94ce6 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:55:46.666 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:55:50.742 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x37713f91 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:55:50.743 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:56:19.149 INFO:tasks.ceph.osd.0.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0xd5872 0xd5a00 0xd548b 0x287b7 0xb732c 0x398421a1 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36afd000 0x36b607c8 0x36b65635 0x36b679b8 0x36b684ad 0x36b688a6 0x36b69106 0x36b69fd5 0x36fea258 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:56:19.150 INFO:tasks.ceph.osd.0.smithi179.stderr:kernel callstack: 2023-01-24T16:56:20.266 INFO:tasks.ceph.osd.1.smithi179.stderr:Rate-limit: suppressed 4 backtraces on shard 0 2023-01-24T16:56:21.817 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x3b712bd2 0x3b71318c 0x39842454 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:56:21.818 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:56:21.818 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:56:21.818 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:56:21.819 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:56:21.819 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:56:21.819 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:56:29.527 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x39842203 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:56:29.528 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:56:29.528 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:56:37.267 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x377142e0 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:56:37.268 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:56:41.349 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x3b64d544 0x3b64e399 0x3b6513c1 0x35f19188 0x3602c6e5 0x3602ef36 0x3602f68b 0x45189fad 0x4518acc4 0x3627c645 0x36bb1977 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:56:41.350 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:56:44.940 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x3b71308d 0x3b71318c 0x39842454 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:56:44.941 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:57:08.428 INFO:tasks.ceph.osd.0.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x3771417d 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36afd000 0x36b607c8 0x36b65635 0x36b679b8 0x36b684ad 0x36b688a6 0x36b69106 0x36b69fd5 0x36fea258 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:57:08.428 INFO:tasks.ceph.osd.0.smithi179.stderr:kernel callstack: 2023-01-24T16:57:20.299 INFO:tasks.ceph.osd.1.smithi179.stderr:Rate-limit: suppressed 3 backtraces on shard 0 2023-01-24T16:57:23.895 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x2b350 0x2b832 0x28adf 0xb732c 0x398421a1 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:57:23.896 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:57:23.896 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:57:23.896 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:57:23.897 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:57:23.897 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:57:25.919 INFO:tasks.ceph.osd.0.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x9435f 0x94421 0x94ce6 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36afd000 0x36b607c8 0x36b65635 0x36b679b8 0x36b684ad 0x36b688a6 0x36b69106 0x36b69fd5 0x36fea258 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:57:25.919 INFO:tasks.ceph.osd.0.smithi179.stderr:kernel callstack: 2023-01-24T16:57:31.243 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x3b712bd2 0x3b71318c 0x39842454 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:57:31.244 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:57:39.375 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0xd5d53 0xc0f8c 0xb7393 0x398421a1 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:57:39.376 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:57:43.393 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x3b712bd2 0x3b71318c 0x39842454 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:57:43.394 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:57:47.079 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x3771417d 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:57:47.080 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:58:20.333 INFO:tasks.ceph.osd.1.smithi179.stderr:Rate-limit: suppressed 2 backtraces on shard 0 2023-01-24T16:58:33.606 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x94c80 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:58:33.607 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:58:33.607 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:58:33.607 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:58:33.607 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:58:57.196 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0xc96bf 0x94409 0x94ce6 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:58:57.197 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:58:57.197 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:59:04.766 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x94413 0x94ce6 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:59:04.766 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:59:20.345 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0xc96cd 0x94409 0x94ce6 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:59:20.346 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:59:20.346 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:59:27.780 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x397a6 0x943f6 0x94ce6 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:59:27.781 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:59:35.532 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x3b712bd2 0x3b71318c 0x39842454 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:59:35.533 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:59:35.533 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:59:46.689 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x2ac4d 0x2b4c3 0x2b832 0x28adf 0xb732c 0x398421a1 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:59:46.730 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:59:46.731 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T16:59:51.039 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x3b713082 0x3b71318c 0x39842454 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T16:59:51.040 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:00:00.470 INFO:tasks.ceph.osd.0.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x94377 0x94ce6 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36afd000 0x36b607c8 0x36b65635 0x36b679b8 0x36b684ad 0x36b688a6 0x36b69106 0x36b69fd5 0x36fea258 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:00:00.471 INFO:tasks.ceph.osd.0.smithi179.stderr:kernel callstack: 2023-01-24T17:00:06.387 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x37714187 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:00:06.388 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:00:06.388 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:00:45.510 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x94c6f 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:00:45.511 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:00:45.511 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:00:45.511 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:00:53.215 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x37714130 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:00:53.216 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:01:01.024 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x3b712b85 0x3b71318c 0x39842454 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:01:01.025 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:01:08.729 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x3b712ceb 0x3b71318c 0x39842454 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:01:08.730 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:01:17.852 INFO:tasks.ceph.osd.0.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x39842203 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36afd000 0x36b607c8 0x36b65635 0x36b679b8 0x36b684ad 0x36b688a6 0x36b69106 0x36b69fd5 0x36fea258 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:01:17.852 INFO:tasks.ceph.osd.0.smithi179.stderr:kernel callstack: 2023-01-24T17:01:24.237 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0xb7f5f 0x28817 0xb732c 0x398421a1 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:01:24.238 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:01:24.238 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:01:31.828 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0xcdf9d 0x2adc3 0x2b4c3 0x2b832 0x28adf 0xb732c 0x398421a1 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:01:31.829 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 0xffffffffffffff80 0xffffffffb107fca9 0xffffffffb12e4ef9 0xffffffffb12e50ec 0xffffffffb12d827a 0xffffffffb12fd957 0xffffffffb12ff106 0xffffffffb10043ab 0xffffffffb1a000a9 2023-01-24T17:01:39.477 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x3b712bd2 0x3b71318c 0x39842454 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:01:39.477 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:01:46.983 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 34 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x2b3fe 0x2b832 0x28adf 0xb4614 0x445f3527 0x445f875d 0x36aba437 0x3618baa7 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:01:46.983 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:01:54.975 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 34 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0xb7e1a 0x2b490 0x2b832 0x28adf 0xb4614 0x445f3527 0x445f875d 0x36aba437 0x3618baa7 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:01:54.975 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:02:20.383 INFO:tasks.ceph.osd.1.smithi179.stderr:Rate-limit: suppressed 1 backtrace on shard 0 2023-01-24T17:02:26.337 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 34 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x2b45b 0x2b832 0x28adf 0xb732c 0x398421a1 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:02:26.337 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:02:26.338 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:02:26.338 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:02:34.097 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 36 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0xc6eef 0x2c74b 0x2857c 0xb62c3 0x35f61d7f 0x35f646f1 0x3b712963 0x3b71318c 0x39842454 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:02:34.097 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:02:41.929 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x37714223 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:02:41.930 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:02:45.838 INFO:tasks.ceph.osd.0.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x35f61c1a 0x35f646f1 0x3b712963 0x3b71318c 0x39842454 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36afd000 0x36b607c8 0x36b65635 0x36b679b8 0x36b684ad 0x36b688a6 0x36b69106 0x36b69fd5 0x36fea258 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:02:45.839 INFO:tasks.ceph.osd.0.smithi179.stderr:kernel callstack: 2023-01-24T17:02:49.385 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 35 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x37714046 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:02:49.385 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:03:05.030 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x3771418f 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:03:05.031 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:03:20.415 INFO:tasks.ceph.osd.1.smithi179.stderr:Rate-limit: suppressed 2 backtraces on shard 0 2023-01-24T17:03:28.120 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0xd5872 0xd5a00 0xd548b 0x2c64a 0x2857c 0xb62c3 0x35f61d7f 0x35f646f1 0x3b712963 0x3b71318c 0x39842454 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:03:28.120 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:03:28.121 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:03:28.121 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:03:35.558 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0xc97cb 0x94409 0x94ce6 0x37714a1d 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:03:35.559 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:03:43.223 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x377141c8 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:03:43.224 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:03:51.223 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x290db 0x2c4ab 0x2857c 0xb62c3 0x35f61d7f 0x35f646f1 0x3b712963 0x3b71318c 0x39842454 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:03:51.224 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:03:51.224 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:03:58.810 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x3771417d 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:03:58.810 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:04:20.458 INFO:tasks.ceph.osd.1.smithi179.stderr:Rate-limit: suppressed 2 backtraces on shard 0 2023-01-24T17:04:22.088 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x2c69f 0x2857c 0xb62c3 0x35f61d7f 0x35f646f1 0x3b712963 0x3b71318c 0x39842454 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:04:22.089 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:04:22.089 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:04:22.089 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:04:22.090 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:04:22.090 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:04:26.112 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x10d5be 0xff74d 0xffba3 0x3b64d6ee 0x3b64e399 0x3b6513c1 0x35f19188 0x3602c6e5 0x3602ef36 0x3602f68b 0x45189fad 0x4518acc4 0x3627c645 0x36bb1977 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:04:26.113 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:04:29.842 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x39769 0x943d7 0x94ce6 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:04:29.842 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:04:37.266 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x3771417d 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:04:37.267 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:04:37.267 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:04:37.553 INFO:tasks.ceph.osd.0.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x2b429 0x2b832 0x28adf 0xb732c 0x398421a1 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36afd000 0x36b607c8 0x36b65635 0x36b679b8 0x36b684ad 0x36b688a6 0x36b69106 0x36b69fd5 0x36fea258 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:04:37.554 INFO:tasks.ceph.osd.0.smithi179.stderr:kernel callstack: 2023-01-24T17:04:37.554 INFO:tasks.ceph.osd.0.smithi179.stderr:kernel callstack: 2023-01-24T17:04:45.038 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x39761 0x943d7 0x94ce6 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:04:45.039 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:05:20.478 INFO:tasks.ceph.osd.1.smithi179.stderr:Rate-limit: suppressed 4 backtraces on shard 0 2023-01-24T17:05:23.716 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0xc97c7 0x94409 0x94ce6 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:05:23.717 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 0xffffffffffffff80 0xffffffffb12dc592 0xffffffffb107c8b0 0xffffffffb107cb47 0xffffffffb1a0114e 2023-01-24T17:05:23.717 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:05:23.717 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:05:23.717 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:05:23.717 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:05:23.718 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:05:23.718 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:05:31.319 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x377140e4 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:05:31.320 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:05:39.084 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0xf7c3 0xc37ec 0xc15bc 0x2b495 0x2b832 0x28adf 0xb4614 0x445f3527 0x445f875d 0x36aba437 0x3618baa7 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:05:39.085 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:05:44.487 INFO:tasks.ceph.osd.0.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x3771417d 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36afd000 0x36b607c8 0x36b65635 0x36b679b8 0x36b684ad 0x36b688a6 0x36b69106 0x36b69fd5 0x36fea258 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:05:44.488 INFO:tasks.ceph.osd.0.smithi179.stderr:kernel callstack: 2023-01-24T17:05:54.586 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x3771417d 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:05:54.587 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:05:54.587 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:06:01.852 INFO:tasks.ceph.osd.0.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x2b44f 0x2b832 0x28adf 0xb732c 0x398421a1 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36afd000 0x36b607c8 0x36b65635 0x36b679b8 0x36b684ad 0x36b688a6 0x36b69106 0x36b69fd5 0x36fea258 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:06:01.853 INFO:tasks.ceph.osd.0.smithi179.stderr:kernel callstack: 2023-01-24T17:06:02.425 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x3771424d 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:06:02.425 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:06:20.495 INFO:tasks.ceph.osd.1.smithi179.stderr:Rate-limit: suppressed 1 backtrace on shard 0 2023-01-24T17:06:25.738 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x2b343 0x2b832 0x28adf 0xb732c 0x398421a1 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:06:25.739 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:06:25.739 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:06:25.739 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:06:49.170 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0xc2077 0xc0f67 0xb7393 0x398421a1 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:06:49.171 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:06:56.942 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x2b3c7 0x2b832 0x28adf 0xb732c 0x398421a1 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:06:56.943 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:07:04.569 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0xc97e8 0x94409 0x94ce6 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:07:04.570 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:07:08.285 INFO:tasks.ceph.osd.0.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x28892 0xb732c 0x398421a1 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36afd000 0x36b607c8 0x36b65635 0x36b679b8 0x36b684ad 0x36b688a6 0x36b69106 0x36b69fd5 0x36fea258 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:07:08.286 INFO:tasks.ceph.osd.0.smithi179.stderr:kernel callstack: 2023-01-24T17:07:12.510 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x3b712ade 0x3b71318c 0x39842454 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:07:12.511 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:07:20.519 INFO:tasks.ceph.osd.1.smithi179.stderr:Rate-limit: suppressed 2 backtraces on shard 0 2023-01-24T17:07:23.341 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x3771407b 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:07:23.341 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:07:23.341 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:07:23.341 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:07:25.759 INFO:tasks.ceph.osd.0.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x943eb 0x94ce6 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36afd000 0x36b607c8 0x36b65635 0x36b679b8 0x36b684ad 0x36b688a6 0x36b69106 0x36b69fd5 0x36fea258 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:07:25.759 INFO:tasks.ceph.osd.0.smithi179.stderr:kernel callstack: 2023-01-24T17:07:27.687 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x39842203 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:07:27.688 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:07:31.175 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0xc97e8 0x94409 0x94ce6 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:07:31.175 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:07:43.043 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x3a4dfb3c 0x3b712fdb 0x3b71318c 0x39842454 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:07:43.044 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:07:54.627 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x94379 0x94ce6 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:07:54.627 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:07:54.627 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:08:11.573 INFO:tasks.ceph.osd.0.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x3771480e 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36afd000 0x36b607c8 0x36b65635 0x36b679b8 0x36b684ad 0x36b688a6 0x36b69106 0x36b69fd5 0x36fea258 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:08:11.574 INFO:tasks.ceph.osd.0.smithi179.stderr:kernel callstack: 2023-01-24T17:08:20.535 INFO:tasks.ceph.osd.1.smithi179.stderr:Rate-limit: suppressed 3 backtraces on shard 0 2023-01-24T17:08:21.679 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0xc6f0e 0x2c74b 0x2857c 0xb62c3 0x35f61d7f 0x35f646f1 0x3b712963 0x3b71318c 0x39842454 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:08:21.680 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:08:21.680 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:08:21.680 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:08:21.680 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:08:21.681 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:08:29.501 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x3771417d 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:08:29.509 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 0xffffffffffffff80 0xffffffffb100436e 0xffffffffb1a000a9 2023-01-24T17:08:29.510 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:08:33.605 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x37713c83 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:08:33.606 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:08:44.802 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x37714187 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:08:44.803 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:08:44.803 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:08:52.469 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0xc96c7 0x94409 0x94ce6 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:08:52.470 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:08:53.823 INFO:tasks.ceph.osd.0.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x377146e8 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36afd000 0x36b607c8 0x36b65635 0x36b679b8 0x36b684ad 0x36b688a6 0x36b69106 0x36b69fd5 0x36fea258 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:08:53.823 INFO:tasks.ceph.osd.0.smithi179.stderr:kernel callstack: 2023-01-24T17:09:20.554 INFO:tasks.ceph.osd.1.smithi179.stderr:Rate-limit: suppressed 2 backtraces on shard 0 2023-01-24T17:09:23.702 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x94c85 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:09:23.702 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:09:23.702 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:09:23.702 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:09:32.317 INFO:tasks.ceph.osd.0.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0xf7c3 0xc37ec 0xc15bc 0x2b495 0x2b832 0x28adf 0xb732c 0x398421a1 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36afd000 0x36b607c8 0x36b65635 0x36b679b8 0x36b684ad 0x36b688a6 0x36b69106 0x36b69fd5 0x36fea258 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:09:32.317 INFO:tasks.ceph.osd.0.smithi179.stderr:kernel callstack: 2023-01-24T17:09:38.972 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x3771417d 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:09:38.972 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:09:54.575 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0xd59e8 0xd548b 0x2c64a 0x2857c 0xb62c3 0x35f61d7f 0x35f646f1 0x3b712963 0x3b71318c 0x39842454 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:09:54.576 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:09:54.576 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:10:49.151 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x398421eb 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:10:49.152 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:10:49.152 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:10:57.129 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x3771417d 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:10:57.130 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:11:04.661 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x3b712bd2 0x3b71318c 0x39842454 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:11:04.662 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:11:12.194 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x94ceb 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:11:12.195 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:11:20.260 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 34 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x37714371 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:11:20.261 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:11:27.753 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x3771417d 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:11:27.754 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:11:35.601 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0xc97c7 0x94409 0x94ce6 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:11:35.601 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:11:43.186 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0xc15af 0x2b495 0x2b832 0x28adf 0xb732c 0x398421a1 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:11:43.187 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:11:43.187 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:11:46.841 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x37714187 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:11:46.841 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:11:49.506 INFO:tasks.ceph.osd.0.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x94cab 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36afd000 0x36b607c8 0x36b65635 0x36b679b8 0x36b684ad 0x36b688a6 0x36b69106 0x36b69fd5 0x36fea258 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:11:49.507 INFO:tasks.ceph.osd.0.smithi179.stderr:kernel callstack: 2023-01-24T17:11:49.507 INFO:tasks.ceph.osd.0.smithi179.stderr:kernel callstack: 2023-01-24T17:11:50.942 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x37714223 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:11:50.943 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:12:20.614 INFO:tasks.ceph.osd.1.smithi179.stderr:Rate-limit: suppressed 5 backtraces on shard 0 2023-01-24T17:12:22.207 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x366a054f 0x3677bfc8 0x36bb1bd1 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:12:22.207 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:12:22.207 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:12:22.207 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:12:22.208 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:12:22.208 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:12:22.208 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:12:22.208 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:12:29.731 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:12:29.732 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:12:29.732 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:12:37.674 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x37714027 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:12:37.675 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:12:37.675 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:12:41.408 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x2b45b 0x2b832 0x28adf 0xb4614 0x445f3527 0x445f875d 0x36aba437 0x3618baa7 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:12:41.408 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:12:45.236 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x365f5ff1 0x3677c539 0x36bb1bd1 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:12:45.237 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:13:20.641 INFO:tasks.ceph.osd.1.smithi179.stderr:Rate-limit: suppressed 4 backtraces on shard 0 2023-01-24T17:13:43.461 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x2b350 0x2b832 0x28adf 0xb732c 0x398421a1 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:13:43.462 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:13:43.462 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:13:43.462 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 0xffffffffffffff80 0xffffffffb12f4bc6 0xffffffffb12f7a46 0xffffffffb12a74aa 0xffffffffb12e4e84 0xffffffffb12e50ec 0xffffffffb12d827a 0xffffffffb12fd957 0xffffffffb12ff106 0xffffffffb10043ab 0xffffffffb1a000a9 2023-01-24T17:13:43.462 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 0xffffffffffffff80 0xffffffffb134164d 0xffffffffb13206f1 0xffffffffb1326d06 0xffffffffb12dfeba 0xffffffffb12e1ea2 0xffffffffb12e35f9 0xffffffffb12e3edb 0xffffffffb12bbed4 0xffffffffb12e1369 0xffffffffb10043ab 0xffffffffb1a000a9 2023-01-24T17:13:43.463 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:13:43.463 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:13:45.226 INFO:tasks.ceph.osd.0.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x2ac65 0x2b4c3 0x2b832 0x28adf 0xb732c 0x398421a1 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36afd000 0x36b607c8 0x36b65635 0x36b679b8 0x36b684ad 0x36b688a6 0x36b69106 0x36b69fd5 0x36fea258 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:13:45.226 INFO:tasks.ceph.osd.0.smithi179.stderr:kernel callstack: 2023-01-24T17:13:50.915 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0xc97c7 0x94409 0x94ce6 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:13:50.915 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:13:50.916 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:14:02.581 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x3771420d 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:14:02.581 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 0xffffffffffffff80 0xffffffffb12e3dd2 0xffffffffb12bbed4 0xffffffffb12e1369 0xffffffffb10043ab 0xffffffffb1a000a9 2023-01-24T17:14:02.581 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:14:06.149 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x3b712bd2 0x3b71318c 0x39842454 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:14:06.149 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:14:10.352 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 37 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0xc97cb 0x94409 0x94ce6 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:14:10.352 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:14:16.852 INFO:tasks.ceph.osd.0.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0xc97c7 0x94409 0x94ce6 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36afd000 0x36b607c8 0x36b65635 0x36b679b8 0x36b684ad 0x36b688a6 0x36b69106 0x36b69fd5 0x36fea258 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:14:16.853 INFO:tasks.ceph.osd.0.smithi179.stderr:kernel callstack: 2023-01-24T17:14:21.737 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x2b35a 0x2b832 0x28adf 0xb732c 0x398421a1 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:14:21.738 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:14:21.738 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:14:25.849 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x3771417d 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:14:25.849 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:14:29.258 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x2b45b 0x2b832 0x28adf 0xb732c 0x398421a1 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:14:29.258 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:14:37.246 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x2b3db 0x2b832 0x28adf 0xb732c 0x398421a1 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:14:37.247 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:14:41.427 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x2b34d 0x2b832 0x28adf 0xb732c 0x398421a1 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:14:41.427 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:15:20.702 INFO:tasks.ceph.osd.1.smithi179.stderr:Rate-limit: suppressed 4 backtraces on shard 0 2023-01-24T17:15:23.215 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x2b350 0x2b832 0x28adf 0xb732c 0x398421a1 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:15:23.216 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:15:23.216 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:15:23.216 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:15:23.217 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 0xffffffffffffff80 0xffffffffb1003dfb 0xffffffffb100449d 0xffffffffb1a000a9 2023-01-24T17:15:23.217 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:15:23.217 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:15:38.869 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x2ac7b 0x2b4c3 0x2b832 0x28adf 0xb732c 0x398421a1 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:15:38.870 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:15:46.712 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0xd586f 0xd5a00 0xd548b 0x2c64a 0x2857c 0xb62c3 0x35f61d7f 0x35f646f1 0x3b712963 0x3b71318c 0x39842454 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:15:46.713 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:16:02.285 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0xc37df 0xc15bc 0x2b495 0x2b832 0x28adf 0xb732c 0x398421a1 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:16:02.286 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:16:25.460 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 34 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x2a3a1 0x2ab8e 0x27c98 0x2af57 0x2b4c3 0x2b832 0x28adf 0xb732c 0x398421a1 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:16:25.461 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 0xffffffffffffff80 0xffffffffb107c919 0xffffffffb107cb47 0xffffffffb1a0114e 2023-01-24T17:16:36.818 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x36dfa8ca 0x36f48c45 0x392a0fec 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:16:36.818 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:16:36.818 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:16:41.165 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0xd594d 0xd548b 0x2c64a 0x2857c 0xb62c3 0x35f61d7f 0x35f646f1 0x3b712963 0x3b71318c 0x39842454 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:16:41.165 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:16:48.998 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x3b712b85 0x3b71318c 0x39842454 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:16:48.999 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:16:54.676 INFO:tasks.ceph.osd.0.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x3771417d 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36afd000 0x36b607c8 0x36b65635 0x36b679b8 0x36b684ad 0x36b688a6 0x36b69106 0x36b69fd5 0x36fea258 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:16:54.677 INFO:tasks.ceph.osd.0.smithi179.stderr:kernel callstack: 2023-01-24T17:16:54.677 INFO:tasks.ceph.osd.0.smithi179.stderr:kernel callstack: 2023-01-24T17:16:56.933 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x2c5c1 0x2857c 0xb62c3 0x35f61d7f 0x35f646f1 0x3b712963 0x3b71318c 0x39842454 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:16:56.933 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:16:58.005 INFO:tasks.ceph.osd.0.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0xd5872 0xd5a00 0xd548b 0x2c64a 0x2857c 0xb62c3 0x35f61d7f 0x35f646f1 0x3b712963 0x3b71318c 0x39842454 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36afd000 0x36b607c8 0x36b65635 0x36b679b8 0x36b684ad 0x36b688a6 0x36b69106 0x36b69fd5 0x36fea258 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:16:58.006 INFO:tasks.ceph.osd.0.smithi179.stderr:kernel callstack: 2023-01-24T17:17:20.753 INFO:tasks.ceph.osd.1.smithi179.stderr:Rate-limit: suppressed 3 backtraces on shard 0 2023-01-24T17:17:23.589 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x37714353 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:17:23.589 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:17:23.589 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:17:23.589 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:17:23.590 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:17:23.590 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:17:23.590 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:17:27.936 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x2b3df 0x2b832 0x28adf 0xb732c 0x398421a1 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:17:27.936 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:17:35.500 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x94c8e 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:17:35.500 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:17:39.116 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x37714505 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:17:39.116 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:17:43.444 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x2cd2d 0xb732c 0x398421a1 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:17:43.444 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:18:15.010 INFO:tasks.ceph.osd.0.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0xc96c7 0x94409 0x94ce6 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36afd000 0x36b607c8 0x36b65635 0x36b679b8 0x36b684ad 0x36b688a6 0x36b69106 0x36b69fd5 0x36fea258 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:18:15.010 INFO:tasks.ceph.osd.0.smithi179.stderr:kernel callstack: 2023-01-24T17:18:20.788 INFO:tasks.ceph.osd.1.smithi179.stderr:Rate-limit: suppressed 8 backtraces on shard 0 2023-01-24T17:18:21.887 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0xd5872 0xd5a00 0xd548b 0x287b7 0xb732c 0x398421a1 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:18:21.888 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:18:21.888 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:18:21.888 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:18:21.889 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:18:21.889 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:18:21.889 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:18:21.889 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:18:21.889 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:18:21.890 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:18:25.883 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x35f03ffc 0x36a528e8 0x36a52f06 0x36f565e9 0x36b05938 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:18:25.884 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:18:29.536 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x3771417d 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:18:29.536 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:18:36.987 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x3a4dfd25 0x3b712fdb 0x3b71318c 0x39842454 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:18:36.987 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:18:36.987 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:18:45.121 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x94c85 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:18:45.121 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:18:45.121 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:19:20.795 INFO:tasks.ceph.osd.1.smithi179.stderr:Rate-limit: suppressed 8 backtraces on shard 0 2023-01-24T17:19:23.439 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x37714193 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:19:23.440 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:19:23.440 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:19:23.440 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:19:23.441 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:19:23.441 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:19:23.441 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:19:23.441 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:19:23.441 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:19:23.442 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:19:23.442 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:19:31.237 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 34 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x39761 0x943d7 0x94ce6 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:19:31.237 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:19:35.461 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x37714187 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:19:35.462 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:19:39.141 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x2b45b 0x2b832 0x28adf 0xb732c 0x398421a1 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:19:39.141 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:19:47.054 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0xc96d0 0x94409 0x94ce6 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:19:47.054 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:20:20.819 INFO:tasks.ceph.osd.1.smithi179.stderr:Rate-limit: suppressed 3 backtraces on shard 0 2023-01-24T17:20:25.780 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x2b3fe 0x2b832 0x28adf 0xb732c 0x398421a1 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:20:25.780 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:20:25.780 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:20:25.780 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:20:25.781 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:20:25.781 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:20:33.778 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x37714187 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:20:33.779 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:20:33.779 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:20:49.139 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x2b45b 0x2b832 0x28adf 0xb732c 0x398421a1 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:20:49.139 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:20:56.792 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x3b712b44 0x3b71318c 0x39842454 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:20:56.793 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:21:12.055 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x3b712dc1 0x3b71318c 0x39842454 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:21:12.056 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:21:20.829 INFO:tasks.ceph.osd.1.smithi179.stderr:Rate-limit: suppressed 2 backtraces on shard 0 2023-01-24T17:21:24.251 INFO:tasks.ceph.osd.0.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x3977c 0x943d7 0x94ce6 0x37714a1d 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36afd000 0x36b607c8 0x36b65635 0x36b679b8 0x36b684ad 0x36b688a6 0x36b69106 0x36b69fd5 0x36fea258 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:21:24.252 INFO:tasks.ceph.osd.0.smithi179.stderr:kernel callstack: 2023-01-24T17:21:27.656 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0xd5cef 0xc0f8c 0xb632f 0x35f61d7f 0x35f646f1 0x3b712963 0x3b71318c 0x39842454 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:21:27.657 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:21:27.657 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:21:27.657 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:21:35.078 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x2ac8d 0x2b4c3 0x2b832 0x28adf 0xb732c 0x398421a1 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:21:35.078 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:21:35.078 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:21:46.849 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x94c80 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:21:46.849 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:21:58.182 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x37713ff3 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:21:58.182 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:21:59.700 INFO:tasks.ceph.osd.0.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0xc96c3 0x94409 0x94ce6 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36afd000 0x36b607c8 0x36b65635 0x36b679b8 0x36b684ad 0x36b688a6 0x36b69106 0x36b69fd5 0x36fea258 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:21:59.701 INFO:tasks.ceph.osd.0.smithi179.stderr:kernel callstack: 2023-01-24T17:22:05.778 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x3b712bdb 0x3b71318c 0x39842454 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:22:05.778 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:22:20.852 INFO:tasks.ceph.osd.1.smithi179.stderr:Rate-limit: suppressed 1 backtrace on shard 0 2023-01-24T17:22:21.132 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x37714187 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:22:21.132 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:22:21.132 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:22:25.402 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 37 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x2b4b4 0x2b832 0x28adf 0xb4614 0x445f3527 0x445f875d 0x36aba437 0x3618baa7 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:22:25.403 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:22:28.767 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0xb7f77 0x2b490 0x2b832 0x28adf 0xb732c 0x398421a1 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:22:28.768 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:22:33.198 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x37714292 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:22:33.198 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:22:36.396 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x39761 0x943d7 0x94ce6 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:22:36.396 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:23:20.860 INFO:tasks.ceph.osd.1.smithi179.stderr:Rate-limit: suppressed 5 backtraces on shard 0 2023-01-24T17:23:30.473 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x37713cb1 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:23:30.473 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:23:30.473 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:23:30.474 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:23:30.474 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:23:30.474 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:23:30.474 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:23:30.474 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:23:38.313 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x37714187 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:23:38.313 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:23:41.974 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x2b45b 0x2b832 0x28adf 0xb732c 0x398421a1 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:23:41.974 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:23:45.836 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x2acb5 0x2b4c3 0x2b832 0x28adf 0xb732c 0x398421a1 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:23:45.836 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:23:53.899 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x2b350 0x2b832 0x28adf 0xb732c 0x398421a1 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:23:53.899 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:24:20.867 INFO:tasks.ceph.osd.1.smithi179.stderr:Rate-limit: suppressed 4 backtraces on shard 0 2023-01-24T17:24:24.635 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x2b45b 0x2b832 0x28adf 0xb732c 0x398421a1 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:24:24.635 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:24:24.635 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:24:24.635 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:24:24.636 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:24:24.636 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:24:28.004 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x398422b2 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:24:28.004 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:24:32.351 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x94ccf 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:24:32.351 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:24:35.673 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x3771417d 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:24:35.673 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:24:40.020 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x2b350 0x2b832 0x28adf 0xb732c 0x398421a1 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:24:40.020 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:25:20.873 INFO:tasks.ceph.osd.1.smithi179.stderr:Rate-limit: suppressed 5 backtraces on shard 0 2023-01-24T17:25:22.264 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0xb62e6 0x35f61d7f 0x35f646f1 0x3b712963 0x3b71318c 0x39842454 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:25:22.264 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:25:22.264 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:25:22.265 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:25:22.265 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:25:22.265 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:25:22.265 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:25:22.265 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:25:22.265 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:25:26.317 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x2b350 0x2b832 0x28adf 0xb732c 0x398421a1 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:25:26.318 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:25:34.000 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x943b0 0x94ce6 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:25:34.000 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:25:37.855 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x943b0 0x94ce6 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:25:37.856 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:25:41.832 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0xd58aa 0xd5a00 0xd548b 0x287b7 0xb732c 0x398421a1 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:25:41.832 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:25:46.916 INFO:tasks.ceph.osd.0.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x2b350 0x2b832 0x28adf 0xb732c 0x398421a1 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36afd000 0x36b607c8 0x36b65635 0x36b679b8 0x36b684ad 0x36b688a6 0x36b69106 0x36b69fd5 0x36fea258 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:25:46.917 INFO:tasks.ceph.osd.0.smithi179.stderr:kernel callstack: 2023-01-24T17:26:20.903 INFO:tasks.ceph.osd.1.smithi179.stderr:Rate-limit: suppressed 7 backtraces on shard 0 2023-01-24T17:26:24.587 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x3973f 0x943d7 0x94ce6 0x37714a1d 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:26:24.587 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:26:24.587 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:26:24.588 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:26:24.588 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:26:24.588 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:26:24.589 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:26:24.589 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:26:24.589 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:26:28.043 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x2b36c 0x2b832 0x28adf 0xb732c 0x398421a1 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:26:28.043 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:26:35.749 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x3771414a 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:26:35.749 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:26:43.556 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x9443b 0x94ce6 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:26:43.556 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:26:58.951 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x2ac9b 0x2b4c3 0x2b832 0x28adf 0xb732c 0x398421a1 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:26:58.951 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:27:20.927 INFO:tasks.ceph.osd.1.smithi179.stderr:Rate-limit: suppressed 2 backtraces on shard 0 2023-01-24T17:27:30.151 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x37713fa0 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:27:30.151 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:27:30.151 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:27:30.151 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:27:32.191 INFO:tasks.ceph.osd.0.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x3b7130ad 0x3b71318c 0x39842454 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36afd000 0x36b607c8 0x36b65635 0x36b679b8 0x36b684ad 0x36b688a6 0x36b69106 0x36b69fd5 0x36fea258 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:27:32.191 INFO:tasks.ceph.osd.0.smithi179.stderr:kernel callstack: 2023-01-24T17:27:38.057 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x2b2d6 0x2b832 0x28adf 0xb732c 0x398421a1 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:27:38.057 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:27:45.821 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x2b4a6 0x2b832 0x28adf 0xb732c 0x398421a1 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:27:45.821 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:28:04.914 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x39761 0x943d7 0x94ce6 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:28:04.914 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:28:04.914 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:28:09.180 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x2c3f7 0x2857c 0xb62c3 0x35f61d7f 0x35f646f1 0x3b712963 0x3b71318c 0x39842454 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:28:09.181 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:28:10.933 INFO:tasks.ceph.osd.0.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x377141e9 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36afd000 0x36b607c8 0x36b65635 0x36b679b8 0x36b684ad 0x36b688a6 0x36b69106 0x36b69fd5 0x36fea258 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:28:10.933 INFO:tasks.ceph.osd.0.smithi179.stderr:kernel callstack: 2023-01-24T17:28:32.786 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x2b45b 0x2b832 0x28adf 0xb732c 0x398421a1 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:28:32.786 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:28:32.786 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:28:39.473 INFO:tasks.ceph.osd.0.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x3771436c 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36afd000 0x36b607c8 0x36b65635 0x36b679b8 0x36b684ad 0x36b688a6 0x36b69106 0x36b69fd5 0x36fea258 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:28:39.474 INFO:tasks.ceph.osd.0.smithi179.stderr:kernel callstack: 2023-01-24T17:28:40.499 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 34 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x2b45b 0x2b832 0x28adf 0xb732c 0x398421a1 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:28:40.500 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:28:48.170 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0xf79e 0xc37ec 0xc15bc 0x2b495 0x2b832 0x28adf 0xb732c 0x398421a1 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:28:48.170 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:28:48.170 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:28:56.203 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0xd59c5 0xd548b 0x287b7 0xb732c 0x398421a1 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:28:56.203 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:29:03.439 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x2b3fe 0x2b832 0x28adf 0xb732c 0x398421a1 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:29:03.440 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:29:20.973 INFO:tasks.ceph.osd.1.smithi179.stderr:Rate-limit: suppressed 2 backtraces on shard 0 2023-01-24T17:29:26.826 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x2ac9b 0x2b4c3 0x2b832 0x28adf 0xb732c 0x398421a1 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:29:26.826 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:29:26.826 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:29:26.826 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:29:30.929 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x3677a6be 0x3677c6e4 0x36bb1bd1 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:29:30.929 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:29:34.805 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0xc97c7 0x94409 0x94ce6 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:29:34.805 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:29:50.080 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x377147b5 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:29:50.080 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:30:01.941 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x39ca6b3a 0x3626722d 0x36b0719e 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:30:01.941 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:30:01.941 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:30:20.976 INFO:tasks.ceph.osd.1.smithi179.stderr:Rate-limit: suppressed 2 backtraces on shard 0 2023-01-24T17:30:21.010 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0xcdf7a 0xd0f4e 0x2b4c3 0x2b832 0x28adf 0xb732c 0x398421a1 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:30:21.010 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:30:21.010 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:30:21.010 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:30:21.010 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 0xffffffffffffff80 0xffffffffb19f6e91 0xffffffffb12a74e4 0xffffffffb12e4e84 0xffffffffb12e50ec 0xffffffffb12dfc51 0xffffffffb12e1e6d 0xffffffffb12e2158 0xffffffffb12e21e7 0xffffffffb10043ab 0xffffffffb1a000a9 2023-01-24T17:30:28.930 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0xd586f 0xd5a00 0xd548b 0x2c64a 0x2857c 0xb62c3 0x35f61d7f 0x35f646f1 0x3b712963 0x3b71318c 0x39842454 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:30:28.930 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:30:28.930 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:30:36.634 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0xc0f81 0xb632f 0x35f61d7f 0x35f646f1 0x3b712963 0x3b71318c 0x39842454 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:30:36.634 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:30:52.228 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0xc37df 0xc15bc 0x2b495 0x2b832 0x28adf 0xb732c 0x398421a1 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:30:52.229 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:31:04.086 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x2b3d7 0x2b832 0x28adf 0xb732c 0x398421a1 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:31:04.086 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:31:10.267 INFO:tasks.ceph.osd.0.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0xd054c 0x2b490 0x2b832 0x28adf 0xb732c 0x398421a1 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36afd000 0x36b607c8 0x36b65635 0x36b679b8 0x36b684ad 0x36b688a6 0x36b69106 0x36b69fd5 0x36fea258 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:31:10.267 INFO:tasks.ceph.osd.0.smithi179.stderr:kernel callstack: 2023-01-24T17:31:20.987 INFO:tasks.ceph.osd.1.smithi179.stderr:Rate-limit: suppressed 1 backtrace on shard 0 2023-01-24T17:31:23.033 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x2b3fe 0x2b832 0x28adf 0xb732c 0x398421a1 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:31:23.034 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:31:23.034 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:31:23.034 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:31:30.954 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x2b350 0x2b832 0x28adf 0xb732c 0x398421a1 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:31:30.954 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:31:31.474 INFO:tasks.ceph.osd.0.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0xb7f5f 0x2b490 0x2b832 0x28adf 0xb732c 0x398421a1 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36afd000 0x36b607c8 0x36b65635 0x36b679b8 0x36b684ad 0x36b688a6 0x36b69106 0x36b69fd5 0x36fea258 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:31:31.474 INFO:tasks.ceph.osd.0.smithi179.stderr:kernel callstack: 2023-01-24T17:31:38.667 INFO:tasks.ceph.osd.0.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0xb7e9f 0x2b490 0x2b832 0x28adf 0xb732c 0x398421a1 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36afd000 0x36b607c8 0x36b65635 0x36b679b8 0x36b684ad 0x36b688a6 0x36b69106 0x36b69fd5 0x36fea258 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:31:38.667 INFO:tasks.ceph.osd.0.smithi179.stderr:kernel callstack: 2023-01-24T17:31:46.581 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0xd054c 0x2b490 0x2b832 0x28adf 0xb4614 0x445f3527 0x445f875d 0x36aba437 0x3618baa7 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:31:46.581 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:31:54.374 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x2b350 0x2b832 0x28adf 0xb732c 0x398421a1 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:31:54.374 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:32:02.131 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x2ace3 0x2b4c3 0x2b832 0x28adf 0xb732c 0x398421a1 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:32:02.133 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:32:21.000 INFO:tasks.ceph.osd.1.smithi179.stderr:Rate-limit: suppressed 3 backtraces on shard 0 2023-01-24T17:32:25.338 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0xb7f77 0x2b490 0x2b832 0x28adf 0xb732c 0x398421a1 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:32:25.339 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:32:25.339 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:32:25.339 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:32:25.340 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:32:32.895 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x39769 0x943f6 0x94ce6 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:32:32.895 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:32:38.088 INFO:tasks.ceph.osd.0.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x3b712d1f 0x3b71318c 0x39842454 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36afd000 0x36b607c8 0x36b65635 0x36b679b8 0x36b684ad 0x36b688a6 0x36b69106 0x36b69fd5 0x36fea258 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:32:38.088 INFO:tasks.ceph.osd.0.smithi179.stderr:kernel callstack: 2023-01-24T17:32:40.753 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x9443b 0x94ce6 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:32:40.753 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:32:40.753 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:32:44.051 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x37714302 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:32:44.051 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:32:48.494 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x3771417d 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:32:48.494 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:33:21.009 INFO:tasks.ceph.osd.1.smithi179.stderr:Rate-limit: suppressed 4 backtraces on shard 0 2023-01-24T17:33:23.034 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0xc155f 0x3b64d53d 0x3b64e399 0x3b6513c1 0x35f19188 0x3602c6e5 0x3602ef36 0x3602f68b 0x45189fad 0x4518acc4 0x3627c645 0x36bb1977 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:33:23.034 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:33:23.034 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:33:23.035 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:33:23.035 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:33:23.035 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:33:23.035 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:33:23.035 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:33:27.219 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0xb7e52 0x2b490 0x2b832 0x28adf 0xb732c 0x398421a1 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:33:27.219 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:33:38.745 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x3b712bd2 0x3b71318c 0x39842454 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:33:38.745 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:33:38.745 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:33:41.362 INFO:tasks.ceph.osd.0.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0xf78f 0xc37ec 0x287a7 0xb732c 0x398421a1 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36afd000 0x36b607c8 0x36b65635 0x36b679b8 0x36b684ad 0x36b688a6 0x36b69106 0x36b69fd5 0x36fea258 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:33:41.362 INFO:tasks.ceph.osd.0.smithi179.stderr:kernel callstack: 2023-01-24T17:33:42.716 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x39775 0x943f6 0x94ce6 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:33:42.716 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:33:50.292 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x2b350 0x2b832 0x28adf 0xb732c 0x398421a1 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:33:50.292 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 0xffffffffffffff80 0xffffffffb107c919 0xffffffffb107cb47 0xffffffffb1a0114e 2023-01-24T17:33:50.292 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:33:52.137 INFO:tasks.ceph.osd.0.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x2b45b 0x2b832 0x28adf 0xb732c 0x398421a1 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36afd000 0x36b607c8 0x36b65635 0x36b679b8 0x36b684ad 0x36b688a6 0x36b69106 0x36b69fd5 0x36fea258 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:33:52.138 INFO:tasks.ceph.osd.0.smithi179.stderr:kernel callstack: 2023-01-24T17:34:21.015 INFO:tasks.ceph.osd.1.smithi179.stderr:Rate-limit: suppressed 5 backtraces on shard 0 2023-01-24T17:34:21.333 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0xc97cb 0x94409 0x94ce6 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:34:21.333 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:34:21.333 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:34:21.333 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:34:21.334 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:34:21.334 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 0xffffffffffffff80 0xffffffffc079b14f 0xffffffffc0768f05 0xffffffffc076ca22 0xffffffffc076ccb1 0xffffffffc0772099 0xffffffffb1395291 0xffffffffb137e06a 0xffffffffb137e5b5 0xffffffffb129b95c 0xffffffffc075b8e5 0xffffffffb135d462 0xffffffffb1360c55 0xffffffffb1360edf 0xffffffffb10043ab 0xffffffffb1a000a9 2023-01-24T17:34:21.334 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:34:21.334 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:34:21.334 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:34:29.270 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x2b45b 0x2b832 0x28adf 0xb732c 0x398421a1 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:34:29.271 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:34:33.617 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0xc97e8 0x94409 0x94ce6 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:34:33.617 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:34:37.042 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x3b712d1f 0x3b71318c 0x39842454 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:34:37.043 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:34:44.878 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x3771417d 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:34:44.878 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:34:44.878 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:35:20.189 INFO:tasks.ceph.osd.0.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x2c746 0x2857c 0xb62c3 0x35f61d7f 0x35f646f1 0x3b712963 0x3b71318c 0x39842454 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36afd000 0x36b607c8 0x36b65635 0x36b679b8 0x36b684ad 0x36b688a6 0x36b69106 0x36b69fd5 0x36fea258 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:35:20.189 INFO:tasks.ceph.osd.0.smithi179.stderr:kernel callstack: 2023-01-24T17:35:21.016 INFO:tasks.ceph.osd.1.smithi179.stderr:Rate-limit: suppressed 5 backtraces on shard 0 2023-01-24T17:35:23.354 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0xb7e3d 0x2b490 0x2b832 0x28adf 0xb732c 0x398421a1 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:35:23.354 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:35:23.355 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:35:23.355 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:35:23.355 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:35:23.355 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:35:23.355 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:35:34.212 INFO:tasks.ceph.osd.0.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0xcdf9d 0x2a8e8 0x2ab8e 0x27c98 0x2af57 0x2b4c3 0x2b832 0x28adf 0xb732c 0x398421a1 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36afd000 0x36b607c8 0x36b65635 0x36b679b8 0x36b684ad 0x36b688a6 0x36b69106 0x36b69fd5 0x36fea258 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:35:34.212 INFO:tasks.ceph.osd.0.smithi179.stderr:kernel callstack: 2023-01-24T17:36:02.318 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0xc37ec 0xc15bc 0x2b495 0x2b832 0x28adf 0xb732c 0x398421a1 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:36:02.318 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:36:02.318 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:36:02.318 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:36:25.445 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 36 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x2acb5 0x2b4c3 0x2b832 0x28adf 0xb4614 0x445f3527 0x445f875d 0x36aba437 0x3618baa7 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:36:25.446 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:36:25.446 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:36:37.100 INFO:tasks.ceph.osd.0.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x2b45b 0x2b832 0x28adf 0xb732c 0x380ae579 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:36:37.100 INFO:tasks.ceph.osd.0.smithi179.stderr:kernel callstack: 2023-01-24T17:36:40.919 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 35 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x943fe 0x94ce6 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:36:40.920 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:36:40.920 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:36:48.764 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x2b350 0x2b832 0x28adf 0xb732c 0x398421a1 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:36:48.765 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:36:48.765 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:36:52.158 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x398421f9 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:36:52.158 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:36:56.510 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x377142e6 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:36:56.510 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:36:58.364 INFO:tasks.ceph.osd.0.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0xb7e11 0x2b490 0x2b832 0x28adf 0xb732c 0x3618bd06 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:36:58.364 INFO:tasks.ceph.osd.0.smithi179.stderr:kernel callstack: 2023-01-24T17:37:05.291 INFO:tasks.ceph.osd.0.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0xd5872 0xd5a00 0xd548b 0x2c64a 0x2857c 0xb62c3 0x35f61d7f 0x35f646f1 0x3b712963 0x3b71318c 0x39842454 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36afd000 0x36b607c8 0x36b65635 0x36b679b8 0x36b684ad 0x36b688a6 0x36b69106 0x36b69fd5 0x36fea258 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:37:05.291 INFO:tasks.ceph.osd.0.smithi179.stderr:kernel callstack: 2023-01-24T17:37:15.636 INFO:tasks.ceph.osd.0.smithi179.stderr:Reactor stalled for 32 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x94ceb 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36afd000 0x36b607c8 0x36b65635 0x36b679b8 0x36b684ad 0x36b688a6 0x36b69106 0x36b69fd5 0x36fea258 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:37:15.637 INFO:tasks.ceph.osd.0.smithi179.stderr:kernel callstack: 2023-01-24T17:37:21.032 INFO:tasks.ceph.osd.1.smithi179.stderr:Rate-limit: suppressed 1 backtrace on shard 0 2023-01-24T17:37:42.760 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0xd5872 0xd5a00 0xd548b 0x287b7 0xb732c 0x398421a1 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:37:42.760 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:37:42.760 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:37:42.760 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:37:42.761 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:37:42.761 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:37:42.761 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:37:42.761 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:37:42.761 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:37:50.628 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0xd5872 0xd5a00 0xd548b 0x2c64a 0x2857c 0xb62c3 0x35f61d7f 0x35f646f1 0x3b712963 0x3b71318c 0x39842454 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:37:50.629 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:37:50.629 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:37:58.143 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x94409 0x94ce6 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:37:58.143 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:37:58.143 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:38:05.827 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x2b45b 0x2b832 0x28adf 0xb732c 0x398421a1 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:38:05.827 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:38:05.827 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:38:09.931 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x2c5c1 0x2857c 0xb62c3 0x35f61d7f 0x35f646f1 0x3b712963 0x3b71318c 0x39842454 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:38:09.931 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:38:21.037 INFO:tasks.ceph.osd.1.smithi179.stderr:Rate-limit: suppressed 2 backtraces on shard 0 2023-01-24T17:38:21.234 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x2c6b1 0x2857c 0xb62c3 0x35f61d7f 0x35f646f1 0x3b712963 0x3b71318c 0x39842454 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:38:21.234 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:38:21.234 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:38:21.234 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:38:29.166 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x2c278 0x2c5a7 0x2857c 0xb62c3 0x35f61d7f 0x35f646f1 0x3b712963 0x3b71318c 0x39842454 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:38:29.167 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:38:44.431 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x3977c 0x943f6 0x94ce6 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:38:44.431 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:38:52.091 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x2b350 0x2b832 0x28adf 0xb732c 0x398421a1 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:38:52.091 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 0xffffffffffffff80 0xffffffffb1192d60 0xffffffffb18e468a 0xffffffffb18e5952 0xffffffffb18d1000 0xffffffffb18d1167 0xffffffffb17fa642 0xffffffffb17fa9ba 0xffffffffb17fcd1c 0xffffffffb17fcde7 0xffffffffb10043ab 0xffffffffb1a000a9 2023-01-24T17:38:52.091 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:39:07.532 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x2acc1 0x2b4c3 0x2b832 0x28adf 0xb732c 0x398421a1 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:39:07.532 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:39:34.856 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x37713f91 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:39:34.856 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:39:34.856 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:39:34.857 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:39:54.256 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0xd054c 0x2b490 0x2b832 0x28adf 0xb732c 0x398421a1 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:39:54.256 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:39:54.256 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:40:02.368 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x2c96c 0x2857c 0xb62c3 0x35f61d7f 0x35f646f1 0x3b712963 0x3b71318c 0x39842454 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:40:02.368 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:40:09.763 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x2b389 0x2b832 0x28adf 0xb732c 0x398421a1 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:40:09.764 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:40:09.764 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:40:17.940 INFO:tasks.ceph.osd.0.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0xc15af 0x2b495 0x2b832 0x28adf 0xb732c 0x398421a1 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36afd000 0x36b607c8 0x36b65635 0x36b679b8 0x36b684ad 0x36b688a6 0x36b69106 0x36b69fd5 0x36fea258 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:40:17.940 INFO:tasks.ceph.osd.0.smithi179.stderr:kernel callstack: 2023-01-24T17:40:17.940 INFO:tasks.ceph.osd.0.smithi179.stderr:kernel callstack: 2023-01-24T17:40:25.017 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 34 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x2ac56 0x2b4c3 0x2b832 0x28adf 0xb732c 0x398421a1 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:40:25.018 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:40:32.697 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x377148f4 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:40:32.697 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:40:36.198 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x3a4dfa17 0x3b712fdb 0x3b71318c 0x39842454 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:40:36.199 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:40:38.621 INFO:tasks.ceph.osd.0.smithi179.stderr:Reactor stalled for 34 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x2b45b 0x2b832 0x28adf 0xb732c 0x398421a1 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36afd000 0x36b607c8 0x36b65635 0x36b679b8 0x36b684ad 0x36b688a6 0x36b69106 0x36b69fd5 0x36fea258 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:40:38.621 INFO:tasks.ceph.osd.0.smithi179.stderr:kernel callstack: 2023-01-24T17:40:47.626 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0xf79e 0xc37ec 0xc15bc 0x2b495 0x2b832 0x28adf 0xb732c 0x398421a1 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:40:47.626 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:40:47.626 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:40:51.366 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x3979e 0x943f6 0x94ce6 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:40:51.366 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:41:21.089 INFO:tasks.ceph.osd.1.smithi179.stderr:Rate-limit: suppressed 4 backtraces on shard 0 2023-01-24T17:41:22.724 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 35 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x3a4dfa34 0x3b712fdb 0x3b71318c 0x39842454 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:41:22.725 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:41:22.725 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:41:22.725 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:41:22.726 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:41:22.726 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:41:22.726 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:41:22.727 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:41:26.074 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0xc15cd 0x2c68d 0x2857c 0xb62c3 0x35f61d7f 0x35f646f1 0x3b712963 0x3b71318c 0x39842454 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:41:26.074 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:41:30.891 INFO:tasks.ceph.osd.0.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x37714253 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36afd000 0x36b607c8 0x36b65635 0x36b679b8 0x36b684ad 0x36b688a6 0x36b69106 0x36b69fd5 0x36fea258 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:41:30.891 INFO:tasks.ceph.osd.0.smithi179.stderr:kernel callstack: 2023-01-24T17:41:34.068 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x39793 0x943d7 0x94ce6 0x37714318 0x3771462b 0x398425c3 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:41:34.068 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:41:38.056 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x39842203 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:41:38.056 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:41:41.344 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x3a4dfb13 0x3b712fdb 0x3b71318c 0x39842454 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:41:41.344 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:42:21.106 INFO:tasks.ceph.osd.1.smithi179.stderr:Rate-limit: suppressed 6 backtraces on shard 0 2023-01-24T17:42:23.351 INFO:tasks.ceph.osd.0.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x3b712bd2 0x3b71318c 0x39842454 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36afd000 0x36b607c8 0x36b65635 0x36b679b8 0x36b684ad 0x36b688a6 0x36b69106 0x36b69fd5 0x36fea258 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:42:23.352 INFO:tasks.ceph.osd.0.smithi179.stderr:kernel callstack: 2023-01-24T17:42:47.018 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0xb733f 0x398421a1 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:42:47.018 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:42:47.018 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:42:47.019 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:42:47.019 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:42:47.019 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:42:47.019 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:42:47.019 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:42:47.020 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:42:47.020 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:42:50.779 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 36 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0xb734f 0x398421a1 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:42:50.779 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:43:06.466 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0xc2023 0xc0f72 0xb632f 0x35f61d7f 0x35f646f1 0x3b712963 0x3b71318c 0x39842454 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:43:06.466 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:43:13.963 INFO:tasks.ceph.osd.1.smithi179.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d4d 0x442cc6e9 0x43fe48bb 0x43ffefad 0x43fff454 0x43fff6e8 0x43fff9b7 0x12cef 0x2b45b 0x2b832 0x28adf 0xb732c 0x398421a1 0x398a26d5 0x36b05372 0x392a04b0 0x392f5fc4 0x36bb160c 0x383d40fd 0x383d8823 0x383d9e2b 0x383ddc8b 0x366d93bf 0x366dacec 0x43fc2c02 0x4401fd85 0x44219ea7 0x4421bebe 0x43b5c2cf 0x43b60bc1 0x361d2ca4 0x3ad84 0x35e7229d 2023-01-24T17:43:13.963 INFO:tasks.ceph.osd.1.smithi179.stderr:kernel callstack: 2023-01-24T17:43:16.066 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-01-24T17:43:16.126 INFO:tasks.workunit:Stopping ['rbd/test_lock_fence.sh'] on client.0... 2023-01-24T17:43:16.126 DEBUG:teuthology.orchestra.run.smithi179:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2023-01-24T17:43:16.391 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_8e1bba12e584b7ae912e92e94afe9c075ab51fa3/teuthology/run_tasks.py", line 103, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_8e1bba12e584b7ae912e92e94afe9c075ab51fa3/teuthology/run_tasks.py", line 82, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_510284b66513490445619d1430aa869868c71a09/qa/tasks/workunit.py", line 129, in task p.spawn(_run_tests, ctx, refspec, role, tests, File "/home/teuthworker/src/git.ceph.com_teuthology_8e1bba12e584b7ae912e92e94afe9c075ab51fa3/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_8e1bba12e584b7ae912e92e94afe9c075ab51fa3/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_8e1bba12e584b7ae912e92e94afe9c075ab51fa3/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_8e1bba12e584b7ae912e92e94afe9c075ab51fa3/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_510284b66513490445619d1430aa869868c71a09/qa/tasks/workunit.py", line 424, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_8e1bba12e584b7ae912e92e94afe9c075ab51fa3/teuthology/orchestra/remote.py", line 525, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_8e1bba12e584b7ae912e92e94afe9c075ab51fa3/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_8e1bba12e584b7ae912e92e94afe9c075ab51fa3/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_8e1bba12e584b7ae912e92e94afe9c075ab51fa3/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test rbd/test_lock_fence.sh) on smithi179 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=510284b66513490445619d1430aa869868c71a09 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 CRIMSON_COMPAT=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/test_lock_fence.sh' 2023-01-24T17:43:16.494 ERROR:teuthology.run_tasks: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=64f0e46c9d19412e99510a38c7142f7e Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_8e1bba12e584b7ae912e92e94afe9c075ab51fa3/teuthology/run_tasks.py", line 103, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_8e1bba12e584b7ae912e92e94afe9c075ab51fa3/teuthology/run_tasks.py", line 82, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_510284b66513490445619d1430aa869868c71a09/qa/tasks/workunit.py", line 129, in task p.spawn(_run_tests, ctx, refspec, role, tests, File "/home/teuthworker/src/git.ceph.com_teuthology_8e1bba12e584b7ae912e92e94afe9c075ab51fa3/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_8e1bba12e584b7ae912e92e94afe9c075ab51fa3/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_8e1bba12e584b7ae912e92e94afe9c075ab51fa3/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_8e1bba12e584b7ae912e92e94afe9c075ab51fa3/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_510284b66513490445619d1430aa869868c71a09/qa/tasks/workunit.py", line 424, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_8e1bba12e584b7ae912e92e94afe9c075ab51fa3/teuthology/orchestra/remote.py", line 525, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_8e1bba12e584b7ae912e92e94afe9c075ab51fa3/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_8e1bba12e584b7ae912e92e94afe9c075ab51fa3/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_8e1bba12e584b7ae912e92e94afe9c075ab51fa3/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test rbd/test_lock_fence.sh) on smithi179 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=510284b66513490445619d1430aa869868c71a09 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 CRIMSON_COMPAT=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/test_lock_fence.sh' 2023-01-24T17:43:16.498 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-01-24T17:43:16.513 INFO:tasks.ceph.ceph_manager.ceph:Canceling any pending splits or merges... 2023-01-24T17:43:16.538 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-01-24T17:43:16.865 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-01-24T17:43:16.866 INFO:teuthology.orchestra.run.smithi179.stdout:{"epoch":17,"fsid":"91764032-a053-4a42-a007-59362bb9c17e","created":"2023-01-24T14:42:13.552129+0000","modified":"2023-01-24T15:43:24.120444+0000","last_up_change":"2023-01-24T14:42:20.478271+0000","last_in_change":"2023-01-24T14:42:14.554200+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-01-24T14:42:21.544008+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-01-24T14:42:22.569448+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":14,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"3abca7f8-4a61-4e04-b5d4-315fb643d792","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6803","nonce":103809}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":103809}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6809","nonce":103809}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6811","nonce":103809}]},"public_addr":"172.21.15.179:6803/103809","cluster_addr":"172.21.15.179:6802/103809","heartbeat_back_addr":"172.21.15.179:6809/103809","heartbeat_front_addr":"172.21.15.179:6811/103809","state":["exists","up"]},{"osd":1,"uuid":"a0e7d95f-3409-475d-a8d0-a6d00dbd9f99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6805","nonce":103808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":103808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":103808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6810","nonce":103808}]},"public_addr":"172.21.15.179:6805/103808","cluster_addr":"172.21.15.179:6804/103808","heartbeat_back_addr":"172.21.15.179:6808/103808","heartbeat_front_addr":"172.21.15.179:6810/103808","state":["exists","up"]},{"osd":2,"uuid":"79d1ce53-95fc-4407-96a5-cea3d9595b13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":7,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6801","nonce":103807}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":103807}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6806","nonce":103807}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6807","nonce":103807}]},"public_addr":"172.21.15.179:6801/103807","cluster_addr":"172.21.15.179:6800/103807","heartbeat_back_addr":"172.21.15.179:6806/103807","heartbeat_front_addr":"172.21.15.179:6807/103807","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-01-24T17:43:16.880 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-01-24T17:43:16.881 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-01-24T17:43:17.197 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-01-24T17:43:17.199 INFO:teuthology.orchestra.run.smithi179.stderr:dumped all 2023-01-24T17:43:17.211 INFO:teuthology.orchestra.run.smithi179.stdout:{"pg_ready":true,"pg_map":{"version":5437,"stamp":"2023-01-24T17:43:15.358491+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":594499,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":398,"num_read_kb":29,"num_write":132606,"num_write_kb":530689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":12623,"ondisk_log_size":12623,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":314572800,"kb_used":357840,"kb_used_data":1400,"kb_used_omap":0,"kb_used_meta":356416,"kb_avail":314214960,"statfs":{"total":322122547200,"available":321756119040,"internally_reserved":0,"allocated":1433600,"data_stored":1275932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":364969984},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":125,"num_write_kb":494,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001917"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":14,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-01-24T14:43:21.597173+0000","last_change":"2023-01-24T14:42:24.664412+0000","last_active":"2023-01-24T14:43:21.597173+0000","last_peered":"2023-01-24T14:43:21.597173+0000","last_clean":"2023-01-24T14:43:21.597173+0000","last_became_active":"2023-01-24T14:42:24.661922+0000","last_became_peered":"2023-01-24T14:42:24.661922+0000","last_unstale":"2023-01-24T14:43:21.597173+0000","last_undegraded":"2023-01-24T14:43:21.597173+0000","last_fullsized":"2023-01-24T14:43:21.597173+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T14:42:23.493256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T14:42:23.493256+0000","last_clean_scrub_stamp":"2023-01-24T14:42:23.493256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":14,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-01-24T14:43:21.600271+0000","last_change":"2023-01-24T14:42:24.669889+0000","last_active":"2023-01-24T14:43:21.600271+0000","last_peered":"2023-01-24T14:43:21.600271+0000","last_clean":"2023-01-24T14:43:21.600271+0000","last_became_active":"2023-01-24T14:42:24.667022+0000","last_became_peered":"2023-01-24T14:42:24.667022+0000","last_unstale":"2023-01-24T14:43:21.600271+0000","last_undegraded":"2023-01-24T14:43:21.600271+0000","last_fullsized":"2023-01-24T14:43:21.600271+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T14:42:23.493256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T14:42:23.493256+0000","last_clean_scrub_stamp":"2023-01-24T14:42:23.493256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"14'1","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-01-24T14:43:21.606264+0000","last_change":"2023-01-24T14:42:24.658975+0000","last_active":"2023-01-24T14:43:21.606264+0000","last_peered":"2023-01-24T14:43:21.606264+0000","last_clean":"2023-01-24T14:43:21.606264+0000","last_became_active":"2023-01-24T14:42:24.655994+0000","last_became_peered":"2023-01-24T14:42:24.655994+0000","last_unstale":"2023-01-24T14:43:21.606264+0000","last_undegraded":"2023-01-24T14:43:21.606264+0000","last_fullsized":"2023-01-24T14:43:21.606264+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T14:42:23.493256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T14:42:23.493256+0000","last_clean_scrub_stamp":"2023-01-24T14:42:23.493256+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":13,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-01-24T14:43:28.657981+0000","last_change":"2023-01-24T14:42:24.705611+0000","last_active":"2023-01-24T14:43:28.657981+0000","last_peered":"2023-01-24T14:43:28.657981+0000","last_clean":"2023-01-24T14:43:28.657981+0000","last_became_active":"2023-01-24T14:42:24.697628+0000","last_became_peered":"2023-01-24T14:42:24.697628+0000","last_unstale":"2023-01-24T14:43:28.657981+0000","last_undegraded":"2023-01-24T14:43:28.657981+0000","last_fullsized":"2023-01-24T14:43:28.657981+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T14:42:23.493256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T14:42:23.493256+0000","last_clean_scrub_stamp":"2023-01-24T14:42:23.493256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":112,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-01-24T14:43:21.602078+0000","last_change":"2023-01-24T14:42:23.597018+0000","last_active":"2023-01-24T14:43:21.602078+0000","last_peered":"2023-01-24T14:43:21.602078+0000","last_clean":"2023-01-24T14:43:21.602078+0000","last_became_active":"2023-01-24T14:42:23.584003+0000","last_became_peered":"2023-01-24T14:42:23.584003+0000","last_unstale":"2023-01-24T14:43:21.602078+0000","last_undegraded":"2023-01-24T14:43:21.602078+0000","last_fullsized":"2023-01-24T14:43:21.602078+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T14:42:22.489089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T14:42:22.489089+0000","last_clean_scrub_stamp":"2023-01-24T14:42:22.489089+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":39,"num_read_kb":29,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"17'2514","reported_seq":371,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-01-24T17:43:14.058899+0000","last_change":"2023-01-24T14:42:24.641821+0000","last_active":"2023-01-24T17:43:14.058899+0000","last_peered":"2023-01-24T17:43:14.058899+0000","last_clean":"2023-01-24T17:43:14.058899+0000","last_became_active":"2023-01-24T14:42:24.637489+0000","last_became_peered":"2023-01-24T14:42:24.637489+0000","last_unstale":"2023-01-24T17:43:14.058899+0000","last_undegraded":"2023-01-24T17:43:14.058899+0000","last_fullsized":"2023-01-24T17:43:14.058899+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T14:42:23.493256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T14:42:23.493256+0000","last_clean_scrub_stamp":"2023-01-24T14:42:23.493256+0000","objects_scrubbed":0,"log_size":2514,"log_dups_size":0,"ondisk_log_size":2514,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":355,"num_read_kb":0,"num_write":356,"num_write_kb":355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":13,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-01-24T14:43:28.857091+0000","last_change":"2023-01-24T14:42:24.677952+0000","last_active":"2023-01-24T14:43:28.857091+0000","last_peered":"2023-01-24T14:43:28.857091+0000","last_clean":"2023-01-24T14:43:28.857091+0000","last_became_active":"2023-01-24T14:42:24.673637+0000","last_became_peered":"2023-01-24T14:42:24.673637+0000","last_unstale":"2023-01-24T14:43:28.857091+0000","last_undegraded":"2023-01-24T14:43:28.857091+0000","last_fullsized":"2023-01-24T14:43:28.857091+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T14:42:23.493256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T14:42:23.493256+0000","last_clean_scrub_stamp":"2023-01-24T14:42:23.493256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"17'132110","reported_seq":132125,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-01-24T17:43:11.511058+0000","last_change":"2023-01-24T14:42:24.686669+0000","last_active":"2023-01-24T17:43:11.511058+0000","last_peered":"2023-01-24T17:43:11.511058+0000","last_clean":"2023-01-24T17:43:11.511058+0000","last_became_active":"2023-01-24T14:42:24.683545+0000","last_became_peered":"2023-01-24T14:42:24.683545+0000","last_unstale":"2023-01-24T17:43:11.511058+0000","last_undegraded":"2023-01-24T17:43:11.511058+0000","last_fullsized":"2023-01-24T17:43:11.511058+0000","mapping_epoch":10,"log_start":"17'122100","ondisk_log_start":"17'122100","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T14:42:23.493256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T14:42:23.493256+0000","last_clean_scrub_stamp":"2023-01-24T14:42:23.493256+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4115,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":132111,"num_write_kb":528438,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"14'1","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-01-24T14:43:21.603966+0000","last_change":"2023-01-24T14:42:24.657421+0000","last_active":"2023-01-24T14:43:21.603966+0000","last_peered":"2023-01-24T14:43:21.603966+0000","last_clean":"2023-01-24T14:43:21.603966+0000","last_became_active":"2023-01-24T14:42:24.653175+0000","last_became_peered":"2023-01-24T14:42:24.653175+0000","last_unstale":"2023-01-24T14:43:21.603966+0000","last_undegraded":"2023-01-24T14:43:21.603966+0000","last_fullsized":"2023-01-24T14:43:21.603966+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T14:42:23.493256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T14:42:23.493256+0000","last_clean_scrub_stamp":"2023-01-24T14:42:23.493256+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4131,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":359,"num_read_kb":0,"num_write":132469,"num_write_kb":528795,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":12526,"ondisk_log_size":12526,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":39,"num_read_kb":29,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":2,"up_from":7,"seq":30064781921,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":5520,"kb_used_data":72,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":104852080,"statfs":{"total":107374182400,"available":107368529920,"internally_reserved":0,"allocated":73728,"data_stored":28978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":8,"seq":34359749199,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":138144,"kb_used_data":664,"kb_used_omap":0,"kb_used_meta":137472,"kb_avail":104719456,"statfs":{"total":107374182400,"available":107232722944,"internally_reserved":0,"allocated":679936,"data_stored":623477,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":140771328},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":8,"seq":34359749207,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":214176,"kb_used_data":664,"kb_used_omap":0,"kb_used_meta":213504,"kb_avail":104643424,"statfs":{"total":107374182400,"available":107154866176,"internally_reserved":0,"allocated":679936,"data_stored":623477,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":218628096},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2023-01-24T17:43:17.212 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-01-24T17:43:17.530 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-01-24T17:43:17.532 INFO:teuthology.orchestra.run.smithi179.stderr:dumped all 2023-01-24T17:43:17.545 INFO:teuthology.orchestra.run.smithi179.stdout:{"pg_ready":true,"pg_map":{"version":5438,"stamp":"2023-01-24T17:43:17.358814+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":594499,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":398,"num_read_kb":29,"num_write":132661,"num_write_kb":530909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":12678,"ondisk_log_size":12678,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":314572800,"kb_used":357904,"kb_used_data":1400,"kb_used_omap":0,"kb_used_meta":356480,"kb_avail":314214896,"statfs":{"total":322122547200,"available":321756053504,"internally_reserved":0,"allocated":1433600,"data_stored":1275932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":365035520},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":180,"num_write_kb":714,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001916"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":14,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-01-24T14:43:21.597173+0000","last_change":"2023-01-24T14:42:24.664412+0000","last_active":"2023-01-24T14:43:21.597173+0000","last_peered":"2023-01-24T14:43:21.597173+0000","last_clean":"2023-01-24T14:43:21.597173+0000","last_became_active":"2023-01-24T14:42:24.661922+0000","last_became_peered":"2023-01-24T14:42:24.661922+0000","last_unstale":"2023-01-24T14:43:21.597173+0000","last_undegraded":"2023-01-24T14:43:21.597173+0000","last_fullsized":"2023-01-24T14:43:21.597173+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T14:42:23.493256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T14:42:23.493256+0000","last_clean_scrub_stamp":"2023-01-24T14:42:23.493256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":14,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-01-24T14:43:21.600271+0000","last_change":"2023-01-24T14:42:24.669889+0000","last_active":"2023-01-24T14:43:21.600271+0000","last_peered":"2023-01-24T14:43:21.600271+0000","last_clean":"2023-01-24T14:43:21.600271+0000","last_became_active":"2023-01-24T14:42:24.667022+0000","last_became_peered":"2023-01-24T14:42:24.667022+0000","last_unstale":"2023-01-24T14:43:21.600271+0000","last_undegraded":"2023-01-24T14:43:21.600271+0000","last_fullsized":"2023-01-24T14:43:21.600271+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T14:42:23.493256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T14:42:23.493256+0000","last_clean_scrub_stamp":"2023-01-24T14:42:23.493256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"14'1","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-01-24T14:43:21.606264+0000","last_change":"2023-01-24T14:42:24.658975+0000","last_active":"2023-01-24T14:43:21.606264+0000","last_peered":"2023-01-24T14:43:21.606264+0000","last_clean":"2023-01-24T14:43:21.606264+0000","last_became_active":"2023-01-24T14:42:24.655994+0000","last_became_peered":"2023-01-24T14:42:24.655994+0000","last_unstale":"2023-01-24T14:43:21.606264+0000","last_undegraded":"2023-01-24T14:43:21.606264+0000","last_fullsized":"2023-01-24T14:43:21.606264+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T14:42:23.493256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T14:42:23.493256+0000","last_clean_scrub_stamp":"2023-01-24T14:42:23.493256+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":13,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-01-24T14:43:28.657981+0000","last_change":"2023-01-24T14:42:24.705611+0000","last_active":"2023-01-24T14:43:28.657981+0000","last_peered":"2023-01-24T14:43:28.657981+0000","last_clean":"2023-01-24T14:43:28.657981+0000","last_became_active":"2023-01-24T14:42:24.697628+0000","last_became_peered":"2023-01-24T14:42:24.697628+0000","last_unstale":"2023-01-24T14:43:28.657981+0000","last_undegraded":"2023-01-24T14:43:28.657981+0000","last_fullsized":"2023-01-24T14:43:28.657981+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T14:42:23.493256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T14:42:23.493256+0000","last_clean_scrub_stamp":"2023-01-24T14:42:23.493256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":112,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-01-24T14:43:21.602078+0000","last_change":"2023-01-24T14:42:23.597018+0000","last_active":"2023-01-24T14:43:21.602078+0000","last_peered":"2023-01-24T14:43:21.602078+0000","last_clean":"2023-01-24T14:43:21.602078+0000","last_became_active":"2023-01-24T14:42:23.584003+0000","last_became_peered":"2023-01-24T14:42:23.584003+0000","last_unstale":"2023-01-24T14:43:21.602078+0000","last_undegraded":"2023-01-24T14:43:21.602078+0000","last_fullsized":"2023-01-24T14:43:21.602078+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T14:42:22.489089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T14:42:22.489089+0000","last_clean_scrub_stamp":"2023-01-24T14:42:22.489089+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":39,"num_read_kb":29,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"17'2514","reported_seq":371,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-01-24T17:43:14.058899+0000","last_change":"2023-01-24T14:42:24.641821+0000","last_active":"2023-01-24T17:43:14.058899+0000","last_peered":"2023-01-24T17:43:14.058899+0000","last_clean":"2023-01-24T17:43:14.058899+0000","last_became_active":"2023-01-24T14:42:24.637489+0000","last_became_peered":"2023-01-24T14:42:24.637489+0000","last_unstale":"2023-01-24T17:43:14.058899+0000","last_undegraded":"2023-01-24T17:43:14.058899+0000","last_fullsized":"2023-01-24T17:43:14.058899+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T14:42:23.493256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T14:42:23.493256+0000","last_clean_scrub_stamp":"2023-01-24T14:42:23.493256+0000","objects_scrubbed":0,"log_size":2514,"log_dups_size":0,"ondisk_log_size":2514,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":355,"num_read_kb":0,"num_write":356,"num_write_kb":355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":13,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-01-24T14:43:28.857091+0000","last_change":"2023-01-24T14:42:24.677952+0000","last_active":"2023-01-24T14:43:28.857091+0000","last_peered":"2023-01-24T14:43:28.857091+0000","last_clean":"2023-01-24T14:43:28.857091+0000","last_became_active":"2023-01-24T14:42:24.673637+0000","last_became_peered":"2023-01-24T14:42:24.673637+0000","last_unstale":"2023-01-24T14:43:28.857091+0000","last_undegraded":"2023-01-24T14:43:28.857091+0000","last_fullsized":"2023-01-24T14:43:28.857091+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T14:42:23.493256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T14:42:23.493256+0000","last_clean_scrub_stamp":"2023-01-24T14:42:23.493256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"17'132165","reported_seq":132180,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-01-24T17:43:15.963858+0000","last_change":"2023-01-24T14:42:24.686669+0000","last_active":"2023-01-24T17:43:15.963858+0000","last_peered":"2023-01-24T17:43:15.963858+0000","last_clean":"2023-01-24T17:43:15.963858+0000","last_became_active":"2023-01-24T14:42:24.683545+0000","last_became_peered":"2023-01-24T14:42:24.683545+0000","last_unstale":"2023-01-24T17:43:15.963858+0000","last_undegraded":"2023-01-24T17:43:15.963858+0000","last_fullsized":"2023-01-24T17:43:15.963858+0000","mapping_epoch":10,"log_start":"17'122100","ondisk_log_start":"17'122100","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T14:42:23.493256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T14:42:23.493256+0000","last_clean_scrub_stamp":"2023-01-24T14:42:23.493256+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4115,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":132166,"num_write_kb":528658,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"14'1","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-01-24T14:43:21.603966+0000","last_change":"2023-01-24T14:42:24.657421+0000","last_active":"2023-01-24T14:43:21.603966+0000","last_peered":"2023-01-24T14:43:21.603966+0000","last_clean":"2023-01-24T14:43:21.603966+0000","last_became_active":"2023-01-24T14:42:24.653175+0000","last_became_peered":"2023-01-24T14:42:24.653175+0000","last_unstale":"2023-01-24T14:43:21.603966+0000","last_undegraded":"2023-01-24T14:43:21.603966+0000","last_fullsized":"2023-01-24T14:43:21.603966+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T14:42:23.493256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T14:42:23.493256+0000","last_clean_scrub_stamp":"2023-01-24T14:42:23.493256+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4131,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":359,"num_read_kb":0,"num_write":132524,"num_write_kb":529015,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":12581,"ondisk_log_size":12581,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":39,"num_read_kb":29,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":2,"up_from":7,"seq":30064781926,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":5520,"kb_used_data":72,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":104852080,"statfs":{"total":107374182400,"available":107368529920,"internally_reserved":0,"allocated":73728,"data_stored":28978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":8,"seq":34359749204,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":138208,"kb_used_data":664,"kb_used_omap":0,"kb_used_meta":137536,"kb_avail":104719392,"statfs":{"total":107374182400,"available":107232657408,"internally_reserved":0,"allocated":679936,"data_stored":623477,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":140836864},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":8,"seq":34359749207,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":214176,"kb_used_data":664,"kb_used_omap":0,"kb_used_meta":213504,"kb_avail":104643424,"statfs":{"total":107374182400,"available":107154866176,"internally_reserved":0,"allocated":679936,"data_stored":623477,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":218628096},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2023-01-24T17:43:17.546 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-01-24T17:43:17.546 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-01-24T17:43:17.860 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-01-24T17:43:17.862 INFO:teuthology.orchestra.run.smithi179.stderr:dumped all 2023-01-24T17:43:17.875 INFO:teuthology.orchestra.run.smithi179.stdout:{"pg_ready":true,"pg_map":{"version":5438,"stamp":"2023-01-24T17:43:17.358814+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":594499,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":398,"num_read_kb":29,"num_write":132661,"num_write_kb":530909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":12678,"ondisk_log_size":12678,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":314572800,"kb_used":357904,"kb_used_data":1400,"kb_used_omap":0,"kb_used_meta":356480,"kb_avail":314214896,"statfs":{"total":322122547200,"available":321756053504,"internally_reserved":0,"allocated":1433600,"data_stored":1275932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":365035520},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":180,"num_write_kb":714,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001916"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":14,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-01-24T14:43:21.597173+0000","last_change":"2023-01-24T14:42:24.664412+0000","last_active":"2023-01-24T14:43:21.597173+0000","last_peered":"2023-01-24T14:43:21.597173+0000","last_clean":"2023-01-24T14:43:21.597173+0000","last_became_active":"2023-01-24T14:42:24.661922+0000","last_became_peered":"2023-01-24T14:42:24.661922+0000","last_unstale":"2023-01-24T14:43:21.597173+0000","last_undegraded":"2023-01-24T14:43:21.597173+0000","last_fullsized":"2023-01-24T14:43:21.597173+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T14:42:23.493256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T14:42:23.493256+0000","last_clean_scrub_stamp":"2023-01-24T14:42:23.493256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":14,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-01-24T14:43:21.600271+0000","last_change":"2023-01-24T14:42:24.669889+0000","last_active":"2023-01-24T14:43:21.600271+0000","last_peered":"2023-01-24T14:43:21.600271+0000","last_clean":"2023-01-24T14:43:21.600271+0000","last_became_active":"2023-01-24T14:42:24.667022+0000","last_became_peered":"2023-01-24T14:42:24.667022+0000","last_unstale":"2023-01-24T14:43:21.600271+0000","last_undegraded":"2023-01-24T14:43:21.600271+0000","last_fullsized":"2023-01-24T14:43:21.600271+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T14:42:23.493256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T14:42:23.493256+0000","last_clean_scrub_stamp":"2023-01-24T14:42:23.493256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"14'1","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-01-24T14:43:21.606264+0000","last_change":"2023-01-24T14:42:24.658975+0000","last_active":"2023-01-24T14:43:21.606264+0000","last_peered":"2023-01-24T14:43:21.606264+0000","last_clean":"2023-01-24T14:43:21.606264+0000","last_became_active":"2023-01-24T14:42:24.655994+0000","last_became_peered":"2023-01-24T14:42:24.655994+0000","last_unstale":"2023-01-24T14:43:21.606264+0000","last_undegraded":"2023-01-24T14:43:21.606264+0000","last_fullsized":"2023-01-24T14:43:21.606264+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T14:42:23.493256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T14:42:23.493256+0000","last_clean_scrub_stamp":"2023-01-24T14:42:23.493256+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":13,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-01-24T14:43:28.657981+0000","last_change":"2023-01-24T14:42:24.705611+0000","last_active":"2023-01-24T14:43:28.657981+0000","last_peered":"2023-01-24T14:43:28.657981+0000","last_clean":"2023-01-24T14:43:28.657981+0000","last_became_active":"2023-01-24T14:42:24.697628+0000","last_became_peered":"2023-01-24T14:42:24.697628+0000","last_unstale":"2023-01-24T14:43:28.657981+0000","last_undegraded":"2023-01-24T14:43:28.657981+0000","last_fullsized":"2023-01-24T14:43:28.657981+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T14:42:23.493256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T14:42:23.493256+0000","last_clean_scrub_stamp":"2023-01-24T14:42:23.493256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":112,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-01-24T14:43:21.602078+0000","last_change":"2023-01-24T14:42:23.597018+0000","last_active":"2023-01-24T14:43:21.602078+0000","last_peered":"2023-01-24T14:43:21.602078+0000","last_clean":"2023-01-24T14:43:21.602078+0000","last_became_active":"2023-01-24T14:42:23.584003+0000","last_became_peered":"2023-01-24T14:42:23.584003+0000","last_unstale":"2023-01-24T14:43:21.602078+0000","last_undegraded":"2023-01-24T14:43:21.602078+0000","last_fullsized":"2023-01-24T14:43:21.602078+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T14:42:22.489089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T14:42:22.489089+0000","last_clean_scrub_stamp":"2023-01-24T14:42:22.489089+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":39,"num_read_kb":29,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"17'2514","reported_seq":371,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-01-24T17:43:14.058899+0000","last_change":"2023-01-24T14:42:24.641821+0000","last_active":"2023-01-24T17:43:14.058899+0000","last_peered":"2023-01-24T17:43:14.058899+0000","last_clean":"2023-01-24T17:43:14.058899+0000","last_became_active":"2023-01-24T14:42:24.637489+0000","last_became_peered":"2023-01-24T14:42:24.637489+0000","last_unstale":"2023-01-24T17:43:14.058899+0000","last_undegraded":"2023-01-24T17:43:14.058899+0000","last_fullsized":"2023-01-24T17:43:14.058899+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T14:42:23.493256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T14:42:23.493256+0000","last_clean_scrub_stamp":"2023-01-24T14:42:23.493256+0000","objects_scrubbed":0,"log_size":2514,"log_dups_size":0,"ondisk_log_size":2514,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_